USF Seminar Series: Apache Spark, Machine Learning, Recommendations Feb 05 2016

Power of data. Simplicity of design. Speed of innovation. IBM Spark spark.tc

Spark and Recommendations

Spark, Streaming, Machine Learning, Graph Processing, Approximations, Probabilistic Data Structures, NLP

USF Seminar Series Thanks, USF!!

Feb 5th, 2016

Chris Fregly Principal Data Solutions Engineer

We’re Hiring! (Only Nice People) advancedspark.com!

Power of data. Simplicity of design. Speed of innovation. IBM Spark spark.tc spark.tc Power of data. Simplicity of design. Speed of innovation. IBM Spark

Who Am I?

Streaming Data Engineer Netflix OSS Committer

Data Solutions Engineer

Apache Contributor

Principal Data Solutions Engineer IBM Technology Center

Meetup Organizer Advanced Apache Meetup

Book Author Advanced .

Due 2016

Advanced Apache Spark Meetup http://advancedspark.com Meetup Metrics Top 5 Most-active Spark Meetup! 2400+ Members in just 6 mos!! 2500+ Docker image downloads

Meetup Mission Deep-dive into Spark and related open source projects Surface key patterns and idioms Focus on distributed systems, scale, and performance

Power of data. Simplicity of design. Speed of innovation. IBM Spark spark.tc Power of data. Simplicity of design. Speed of innovation. IBM Spark spark.tc

Live, Interactive Demo!! Audience Participation Required

(cell phone or laptop)

demo.advancedspark.com End User ->

ElasticSearch ->

Spark ML ->

Data Scientist -> 5

<- Kafka <- Spark Streaming <- Cassandra, Redis <- Zeppelin, iPython

Presentation Outline   Scaling with Parallelism and Composability

  Similarity and Recommendations

  When to Approximate

  Common Algorithms and Data Structures   Common Libraries and Tools

Scaling with Parallelism

Peter O(log n)

O(log n)

Scaling with Composability

Max (a max b max c max d) == (a max b) max (c max d)

Set Union (a U b U c U d) == (a U b) U (c U d)

Addition (a + b + c + d) == (a + b) + (c + d)

Multiply (a * b * c * d) == (a * b) * (c * d)

Division??

What about Division? Division (a / b / c / d) != (a / b) / (c / d) (3 / 4 / 7 / 8) != (3 / 4) / (7 / 8) (((3 / 4) / 7) / 8) != ((3 * 8) / (4 * 7)) 0.134 != 0.857

What were the Egyptians thinking?! Not Composable

“Divide like an Egyptian”

What about Average?

Overall AVG ( [3, 1] ((3 + 5) + (5 + 7)) 20 [5, 1] == ----------------------- == --- == 5 [5, 1] ((1 + 2) + 1) 4 [7, 1]

Pairwise AVG (3 + 5) (5 + 7) 8 12 20 ------- + ------- == --- + --- == --- == 10 != 5 2 2 2 2 2

Divide, Add, Divide? Not Composable

Single Divide at the End? Doesn’t need to be Composable!

AVG (3, 5, 5, 7) == 5

Add, Add, Add? Composable!

Similarity

Euclidean Similarity Exists in Euclidean, flat space Based on Euclidean distance Linear measure Bias towards magnitude

Cosine Similarity Angular measure Adjusts for Euclidean magnitude bias

Normalizes to unit vectors

Jaccard Similarity Set similarity measurement Set intersection / set union -> Based on Jaccard distance Bias towards popularity

Log Likelihood Similarity Adjusts for popularity bias Netflix “Shawshank” problem

Word Similarity Edit Distance Calculate char differences between words Deletes, transposes, replaces, inserts

Document Similarity TD/IDF Term Freq / Inverse Document Freq Used by most search engines

Word2Vec Words embedded in vector space nearby similars

Similarity Pathway ie. Closest recommendations between 2 people

Calculating Similarity Exact Brute-Force “All-pairs similarity” aka “Pair-wise similarity”, “Similarity join” Cartesian O(n^2) shuffle and comparison

Approximate Sampling Bucketing (aka “Partitioning”, “Clustering”) Remove data with low probability of similarity

Reduce shuffle and comparisons 20

Bonus: Document Summary Text Rank aka “Sentence Rank” TF/IDF + Similarity Graph + PageRank

Intuition Surface summary sentences (abstract) Most similar to all others (TF/IDF + Similarity Graph) Most influential sentences (PageRank)

Similarity Graph Vertex is movie, tag, actor, plot summary, etc. Edges are relationships and weights

Topic-Sensitive PageRank Graph diffusion algorithm Pre-process graph, add vector of probabilities to each vertex Probability of ending up at this vertex from every other

vertex

Recommendations

Basic Terminology User: User seeking recommendations Item: Item being recommended Explicit User Feedback: like or rating Implicit User Feedback: search, click, hover, view, scroll Instances: Rows of user feedback/input data Overfitting: Training a model too closely to the training data & hyperparameters Hold Out Split: Holding out some of the instances to avoid overfitting Features: Columns of instance rows (of feedback/input data) Cold Start Problem: Not enough data to personalize (new) Hyperparameter: Model-specific config knobs for tuning (tree depth, iterations) Model Evaluation: Compare predictions to actual values of hold out split Feature Engineering: Modify, reduce, combine features

Feature Engineering Dimension Reduction Reduce number of features (aka “feature space”)

Principle Component Analysis (PCA) Find principle features that describe the data in terms of variance Peel the dimensional layers back until you describe the data

Example: One-Hot Encoding Convert categorical feature values to 0’s, 1’s Remove any hint of a relationship between the categories Bears -> 1 Bears -> [1,0,0] 49’ers -> 2 --> 49’ers -> [0,1,0] Steelers-> 3 Steelers-> [0,0,1]

1 binary column per category

Features Binary Features: True or False

Numeric Discrete Features: Integers

Numeric Features: Real values

Ordinal Features: Maintains order (S -> M -> L -> XL -> XXL)

Temporal Features: Time-based (Time of Day, Binge Watching)

Categorical Features: Finite, unique set of categories (NFL teams)

Non-Personalized Recommendations

Cold Start Problem “Cold Start” problem New user, don’t know their pref, must show them something!

Movies with highest-rated actors Top K Aggregations

Most desirable singles PageRank of like activity

Facebook social graph Recommend friend activity

Personalized Recommendations

Clustering (aka. Nearest Neighbors) User-to-User Clustering Similar movies watched or rated Similar wiewing pattern (ie. binge or casual)

Item-to-Item Clustering Similar tags/genres on movies Similar textual description (TF/IDF, Word2Vec, NLP, Image)

31 http://crockpotveggies.com/2015/02/09/automating-tinder-with-eigenfaces.html!My OKCupid Profile! My Hinge Profile!

User-to-Item Collaborative Filtering Matrix Factorization ①  Factor the large matrix (left) into 2 smaller matrices (right) ②  Fill in the missing values with in the large matrix

Item-to-Item Collaborative Filtering Made famous by Amazon Paper ~2003 Problem As # of users grew, Matrix Factorization couldn’t scale

Solution Offline/Batch Generate itemId -> List[customerId] vectors

Online/Real-time For each item in cart, recommend similar items from vector space

When to Approximate? Memory or time constrained queries Relative vs. exact counts are OK (# errors between then and now)

Using machine learning or graph algos Inherently probabilistic and approximate Finding topics in documents (LDA) Finding similar pairs of users, items, words at scale (LSH) Finding top influencers (PageRank)

Streaming aggregations (distinct count or top k) Inherently sloppy means of collecting (at least once delivery)

Approximate as much as you can get away with! Ask for forgiveness later !!

When NOT to Approximate? If you’ve ever heard the term…

“Sarbanes-Oxley”

…in-that-order, at the office, after 2002.

A Few Good Algorithms

You can’t handle the approximate!

Common to These Algos & Data Structs Low, fixed size in memory Known error bounds Store large amount of data Less memory than Java/Scala collections Tunable tradeoff between size and error Rely on multiple hash functions or operations Size of hash range defines error

Bloom Filter Set.contains(key): Boolean

“Hash Multiple Times and Flip the Bits Wherever You Land”

Bloom Filter Approximate set membership for key False positive: expect contains(), actual !contains() True negative: expect !contains(), actual !contains()

Elements only added, never removed

Bloom Filter in Action

set(key) contains(key): Boolean

Images by @avibryant

TRUE -> maybe contains FALSE -> definitely does not contain.

CountMin Sketch Frequency Count and TopK

“Hash Multiple Times and Add 1 Wherever You Land”

CountMin Sketch (CMS) Approximate frequency count and TopK for key ie. “Heavy Hitters” on Twitter

Johnny Hallyday Martin Odersky Donald Trump

CountMin Sketch In Action

Images derived from @avibryant

Find minimum of all rows

… …

Can overestimate, but never underestimate

Multiple hash functions (1 hash function per row)

Binary hash output (1 element per column)

x 2 occurrences of “Top Gun” for slightly additional complexity

Top Gun

Top Gun (x 2)

A FewGood Men

Top Gun (x 2)

add(Top Gun, 2)

getCount(Top Gun): Long

Use Case: TopK movies using total views

add(A Few Good Men, 1)

add(Taps, 1)

A FewGood Men

Overlap Top Gun

Overlap A Few Good Men

HyperLogLog Count Distinct

“Hash Multiple Times and Uniformly Distribute Where You Land”

HyperLogLog (HLL) Approximate count distinct Slight twist Special hash function creates uniform distribution

Error estimate 14 bits for size of range m = 2^14 = 16,384 slots error = 1.04/(sqrt(16,384)) = .81%

Not many of these

HyperLogLog In Action Use Case: Distinct number of views per movie

0 32 Top Gun: Hour 2 user2001

user 4009

user 3002

user 7002

user 1005

user 6001

User 8001

User 8002

user 1001

user 2009

user 3005

user 3003

Top Gun: Hour 1 user 3001

user 7009

Uniform Distribution: Estimate distinct # of users by inspecting just the beginning

Uniform Distribution: Estimate distinct # of users

by inspecting just the beginning

Composable: Hour 1 + 2 (lose a bit of precision)

Locality Sensitive Hashing Set Similarity

“Pre-process Items into Buckets, Compare Within Buckets”

Locality Sensitive Hashing (LSH) Approximate set similarity Hash designed to cluster similar items Avoids cartesian all-pairs comparison Pre-process m rows into b buckets b << m

Hash items multiple times Similar items hash to overlapping buckets Compare just contents of buckets Much smaller cartesian … and parallel !!

DIMSUM Set Similarity

“Pre-process Items into Buckets, Compare Within Buckets”

DIMSUM “Dimension Independent Matrix Square Using MR” Remove vectors with low probability of similarity RowMatrix.columnSimiliarites(threshold)

Twitter DIMSUM Case Study 40% efficiency gain over bruce-force cosine sim

Common Tools to Approximate

Twitter Algebird

Apache Spark

Composable Library

Distributed Cache

Big Data Processing

Twitter Algebird Rooted in Algebraic Fundamentals! Parallel Associative Composable Examples Min, Max, Avg BloomFilter (Set.contains(key)) HyperLogLog (Count Distinct) CountMin Sketch (TopK Count)

Redis Implementation of HyperLogLog (Count Distinct) 12KB per item count 2^64 max # of items 0.81% error (Tunable) Add user views for given movie

PFADD TopGun_HLL user1001 user2009 user3005 PFADD TopGun_HLL user3003 user1001

Get distinct count (cardinality) of set

PFCOUNT TopGun_HLL Returns: 4 (distinct users viewed this movie)

ignore duplicates

Tunable

Union 2 HyperLogLog Data Structures PFMERGE TopGun_HLL Taps_HLL

Spark Approximations Spark Core

RDD.count*Approx() Spark SQL

PartialResult HyperLogLogPlus approxCountDistinct(column)

Spark ML Stratified sampling PairRDD.sampleByKey(fractions: Double[ ]) DIMSUM sampling Probabilistic sampling reduces amount of comparison shuffle RowMatrix.columnSimilarities(threshold)

Spark Streaming A/B testing StreamingTest.setTestMethod(“welch”).registerStream(dstream)

Demos!

Counting Exact Count vs. Approx HyperLogLog, CountMin Sketch

HashSet vs. HyperLogLog

HashSet vs. CountMin Sketch

Set Similarity Exact Jaccard Similarity vs. Approx Locality Sensitive Hashing

Brute Force Cartesian All Pair Similarity

90 mins!

All Pairs & Locality Sensitive Hashing

<< 90 mins!

Many More Demos Available! http://advancedspark.com

Download Docker or Clone Github

Bonus: Netflix Recommendations From Offline DVD Ratings to Real-time Trending Now

$1 Million Netflix Prize (2006-2009) Goal Improve movie predictions by 10% (RMSE)

Dataset (userId, movieId, rating, timestamp) Test data withheld to calculate RMSE upon submission

Winning algorithm 10.06% improvement (RMSE) Ensemble of 500+ ML Combined using GBDT’s Computationally impractical

Secret to the Winning Algorithms Adjust for the following… Human bias “Alice effect”: Alice tends to rate lower than average user “Inception effect”: Inception is rated higher than average “Alice-Inception effect”: Combo of Alice and Inception Time-based bias Number of days since a user’s first rating Number of days since a movie’s first rating Number of people who have rated a movie A movie’s overall mean rating

Current Netflix Recommendations

Throw away loffline-generated user factors (U)

Netflix Common ML Algorithms Logistic Regression Linear Regression Gradient Boosted Decision Trees Random Forest Matrix Factorization SVD Restricted Boltzmann Machines Deep Neural Nets Markov Models LDA Clustering … 70

Bonus: Netflix Search No results? No problem… Show similar results! Used as implicit feedback for future decision making

Netflix and Data Netflix has a lot of data about a lot of users and a lot of movies. Netflix can use this data to buy new movies. Netflix is global. Netflix can use this data to choose original programming. Netflix knows that a lot of people like Politics and Kevin Spacey.

The UK doesn’t have any White Castles. So they renamed my favourite movie, “Harold and Kumar Get the Munchies”

(This broke all of my unit tests.)

My favorite movie, “Harold and Kumar Go to White Castle”

Summary: Buy NFLX Stock!

Thank You!! Chris Fregly @cfregly IBM Spark Tech Center http://spark.tc San Francisco, California, USA

http://advancedspark.com Sign up for the Meetup and Book Contribute to Github Repo Run all Demos using Docker

Find me: LinkedIn, Twitter, Github, Email, Fax 73

Image derived from http://www.duchess-france.org/

Power of data. Simplicity of design. Speed of innovation. IBM Spark spark.tc

Power of data. Simplicity of design. Speed of innovation.

IBM Spark

advancedspark.com @cfregly

USF Seminar Series: Apache Spark, Machine Learning, Recommendations Feb 05 2016

Software

State of Security: Apache Spark & Apache Zeppelin

Apache Ignite and Apache Spark - GridGain Systems · Ignite and Spark Integration Spark Application Spark Worker Spark Job Spark Job Yarn Mesos Docker HDFS Spark Worker Spark Job

Apache Spark - Courses€¦ · Apache Spark Introduction to Data Science DATA11001 Nitinder Mohan CollaborativeNetworking (CoNe) nitinder.mohan@helsinki.fi. What is Apache Spark?

Hortonworks Data Platform - Apache Spark Component …€¦ · · 2018-04-15Hortonworks Data Platform: Apache Spark Component Guide ... Tuning Spark ... and debugging Spark shell

Performance-Analyse von Apache Spark und Apache Hadoop€¦ · Apache Spark, Apache Hadoop, Big Data, Benchmarking, Performance-Analyse Kurzzusammenfassung Diese Bachelorarbeit beschäftigt

Budapest Spark Meetup - Apache Spark @enbrite.ly

Accelerator for Apache Spark Functional Specification · Accelerator for Apache Spark – Functional Specification 12 Table 1: Accelerator for Apache Spark Components Component Software

Developing Apache Spark Applications - Cloudera · Apache Spark Quick Start Apache Spark Overview Apache Spark Programming Guide Using the Spark DataFrame API A DataFrame is a distributed

NEW ARCHITECTURES FOR APACHE SPARK TM AND BIG DATA · NEW ARCHITECTURES FOR APACHE SPARK AND BIG DATA The Apache Spark Platform for Big Data The Apache Spark platform is an open-source

Apache Spark PDF

KNIME Extension for Apache Spark Installation Guide · Apache Livy (recommended) Spark Job Server (deprecated) Supported Spark and Hadoop distributions KNIME Extension for Apache

Introduction to Cassandra • Why Spark - Apache Cassandra | Apache Kafka | Apache Spark · 2017. 12. 20. · • Introduction to Cassandra • Why Spark + Cassandra • Problem background

Apache Spark Streaming

[@NaukriEngineering] Apache Spark

Running Apache Spark & Apache Zeppelin in Production

Introduction to Apache Spark

Integrating Apache Hive with Kafka, Spark, and BI...Community Connection: Integrating Apache Hive with Apache Spark--Hive Warehouse Connector Apache Spark-Apache Hive connection configuration

Apache spark session

Apache Spark - LMU

TeachYourself Apache Spark...HOUR 1 Introducing Apache Spark..... 1 2 Understanding Hadoop ... Part II: Programming with Apache Spark HOUR 6: Learning the Basics of Spark Programming