January 6, 2015
Scala Data Pipelines for Music Recommendations
Chris Johnson@MrChrisJohnson
Who am I??• Chris Johnson
– Machine Learning guy from NYC– Focused on music recommendations– Formerly a PhD student at UT Austin
Spotify in Numbers 3
•Started in 2006, now available in 58 markets•50+ million active users, 15 million paying subscribers•30+ million songs, 20,000 new songs added per day•1.5 billion playlists•1 TB user data logged per day•900 node Hadoop cluster•10,000+ Hadoop jobs run every day
4Music Recommendations at Spotify •Discover•Radio•Related Artists
How can we find good recommendations? 5
•Manual Curation
•Manually Tag Attributes
•Audio Content
•News, Blogs, Text analysis
•Collaborative Filtering
Music Recommendations Data Flow 6
Why ? 7
Why ? 8
Interview Question
The Genre Toplist Problem 9
•Assume we have access to daily log data for all plays on Spotify.
•Goal: Calculate the top 1k artists on for each genre based on total daily plays
{"User": “userA”, "Date": “2015-01-10", "Artist": “Beyonce", "Track": "Halo", "Genres": ["Pop", "R&B", "Soul"]}{"User": “userB”, "Date": “2015-01-10”, "Artist": "Led Zeppelin”, "Track": "Achilles Last Stand", "Genres": ["Rock", "Blues Rock", "Hard Rock"]}……….
Genre Toplists with Python MapReduce 10
11
Scalding is a Scala library that makes it easy to specify Hadoop MapReduce jobs. Scalding is built on top of Cascading, a Java library that abstracts away low-level Hadoop details. Scalding is comparable to Pig, but offers tight integration with Scala, bringing advantages of Scala to your MapReduce jobs.
Genre Toplists with Scalding 12
Why ? 13
•Data pipeline flows naturally follow the functional paradigm
•Productivity without sacrificing performance•Active community and ecosystem -Scalding-Summingbird-Algebird-Spark -Breeze•Many data storage solutions integrate well with JVM-Cassandra-HBase-Voldemort-Datomic
Spotify’s Scalding repository over time 14
Genre Toplists with Scalding 15
sortWithTake doesn’t fully sort 16
•Uses PriorityQueueMonoid from Algebird library
•What is a Monoid??-Definition: A Set S and a binary operation • : S x S —> S such that1. Associativity: For all a, b, and c in S the equation
(a • b) • c = a • (b • c) holds2. Identity Element: There exists an element e in S such that for every
element a in S, the equations e • a = a • e = a hold
•Example: The natural numbers N under the addition operation. (1 + 2) + 3 = 1 + (2 + 3) 0 + 1 = 1 + 0 = 1
class PriorityQueueMonoid[K](max : Int)(implicit ord : Ordering[K]) extends Monoid[PriorityQueue[K]]
sortWithTake 17
•Uses PriorityQueueMonoid from Algebird
•PriorityQueue aggregations form a commutative monoid!1. Associative:
PQ1 = [ (Jay Z, 545), (Miles Davis, 272), …] PQ2 = [ (Beyonce, 731), (Kurt Vile, 372), …] PQ3 = [ (Twin Shadow, 87), … ] PQ1 ++ (PQ2 ++ PQ3) = (PQ1 ++ PQ2) ++ PQ32.Commutative:
PQ1 ++ PQ2 = PQ2 ++ PQ13.Identity:
PQ1 ++ EmptyPQ = PQ1
class PriorityQueueMonoid[K](max : Int)(implicit ord : Ordering[K]) extends Monoid[PriorityQueue[K]]
sortWithTake 18
•Uses PriorityQueueMonoid from Algebird
•Ok, great observation… but what’s the point of all this!??-All monoid aggregations and reduces can begin on the Mapper side
and finish on the Reducer side since the order doesn’t matter!-Scalding implicitly takes care of Mapper side combining and custom
combiner-Reduces network traffic to reducers
class PriorityQueueMonoid[K](max : Int)(implicit ord : Ordering[K]) extends Monoid[PriorityQueue[K]]
reduced traffic
Section name 19
How do we store track metadata? 20
•Lots of metadata associated with tracks (100+ columns!)- artist, album, record label, genres, audio features, …•Options:1. Store each track as one long row with many columns-Sending lots of data over network when you only need 1 or 2 columns
2. Store each column as a separate data source-Jobs require costly joins, especially when requiring many columns
•Can we do better?..
Apache Parquet to the rescue! 21
•Apache Parquet is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.•Efficiently read a subset of columns without scanning the entire dataset
•Row group: A logical horizontal partitioning of the data into rows. There is no physical structure that is guaranteed for a row group. A row group consists of a column chunk for each column in the dataset.•Column chunk: A chunk of the data for a particular column. These live in a particular
row group and is guaranteed to be contiguous in the file.•Predicate push-down: Define predicates (<, >, <=, …) to filter out column chunks or
even full row groups, evaluated at Hadoop InputFormat layer before Avro conversion
Genre Toplists with Scalding + Parquet 22
Driven - job visualization and performance analytics 23
Luigi - data plumbing since 2012 24
•Workflow management framework developed by Spotify •Python luigi configuration takes care of dependency resolution, job
scheduling, fault tolerance, etc.•Support for Hive queries, MapReduce jobs, python snippets, Scalding,
Crunch, Spark, and more!•Like Oozie but without all of the messy XML
https://github.com/spotify/luigi
Luigi 25
Section name 26
So…. back to music recommendations! 27
•Manual Curation
•Manually Tag Attributes
•Audio Content
•News, Blogs, Text analysis
•Collaborative Filtering
Collaborative Filtering 28
Hey,I like tracks P, Q, R, S!
Well,I like tracks Q, R, S, T!
Then you should check out track P!
Nice! Btw try track T!
Image via Erik Bernhardsson
Implicit Matrix Factorization 29
1 0 0 0 1 0 0 10 0 1 0 0 1 0 0 1 0 1 0 0 0 1 10 1 0 0 0 1 0 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 1
•Aggregate all (user, track) streams into a large matrix•Goal: Approximate binary preference matrix by the inner product of 2 smaller matrices by
minimizing the weighted RMSE (root mean squared error) using a function of total plays as weight •Why?: Once learned, the top recommendations for a user are the top inner products between
their latent factor vector in X and the track latent factor vectors in Y.
X YUsers
Songs
• = bias for user• = bias for item• = regularization parameter
• = 1 if user streamed track else 0• • = user latent factor vector• = item latent factor vector
Alternating Least Squares 30
1 0 0 0 1 0 0 10 0 1 0 0 1 0 0 1 0 1 0 0 0 1 10 1 0 0 0 1 0 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 1
X YUsers
Songs
• = bias for user• = bias for item• = regularization parameter
• = 1 if user streamed track else 0• • = user latent factor vector• = item latent factor vector
Fix tracks
•Aggregate all (user, track) streams into a large matrix•Goal: Approximate binary preference matrix by the inner product of 2 smaller matrices by
minimizing the weighted RMSE (root mean squared error) using a function of total plays as weight •Why?: Once learned, the top recommendations for a user are the top inner products between
their latent factor vector in X and the track latent factor vectors in Y.
31
1 0 0 0 1 0 0 10 0 1 0 0 1 0 0 1 0 1 0 0 0 1 10 1 0 0 0 1 0 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 1
X YUsers
Songs
• = bias for user• = bias for item• = regularization parameter
• = 1 if user streamed track else 0• • = user latent factor vector• = item latent factor vector
Fix tracks
Solve for users
•Aggregate all (user, track) streams into a large matrix•Goal: Approximate binary preference matrix by the inner product of 2 smaller matrices by
minimizing the weighted RMSE (root mean squared error) using a function of total plays as weight •Why?: Once learned, the top recommendations for a user are the top inner products between
their latent factor vector in X and the track latent factor vectors in Y.
Alternating Least Squares
32
1 0 0 0 1 0 0 10 0 1 0 0 1 0 0 1 0 1 0 0 0 1 10 1 0 0 0 1 0 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 1
X YUsers
Songs
• = bias for user• = bias for item• = regularization parameter
• = 1 if user streamed track else 0• • = user latent factor vector• = item latent factor vector
Fix users
•Aggregate all (user, track) streams into a large matrix•Goal: Approximate binary preference matrix by the inner product of 2 smaller matrices by
minimizing the weighted RMSE (root mean squared error) using a function of total plays as weight •Why?: Once learned, the top recommendations for a user are the top inner products between
their latent factor vector in X and the track latent factor vectors in Y.
Alternating Least Squares
33
1 0 0 0 1 0 0 10 0 1 0 0 1 0 0 1 0 1 0 0 0 1 10 1 0 0 0 1 0 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 1
X YUsers
Songs
• = bias for user• = bias for item• = regularization parameter
• = 1 if user streamed track else 0• • = user latent factor vector• = item latent factor vector
Fix usersSolve for tracks
•Aggregate all (user, track) streams into a large matrix•Goal: Approximate binary preference matrix by the inner product of 2 smaller matrices by
minimizing the weighted RMSE (root mean squared error) using a function of total plays as weight •Why?: Once learned, the top recommendations for a user are the top inner products between
their latent factor vector in X and the track latent factor vectors in Y.
Alternating Least Squares
34
1 0 0 0 1 0 0 10 0 1 0 0 1 0 0 1 0 1 0 0 0 1 10 1 0 0 0 1 0 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 1
X YUsers
Songs
• = bias for user• = bias for item• = regularization parameter
• = 1 if user streamed track else 0• • = user latent factor vector• = item latent factor vector
Fix usersSolve for tracks
Repeat until convergence…
•Aggregate all (user, track) streams into a large matrix•Goal: Approximate binary preference matrix by the inner product of 2 smaller matrices by
minimizing the weighted RMSE (root mean squared error) using a function of total plays as weight •Why?: Once learned, the top recommendations for a user are the top inner products between
their latent factor vector in X and the track latent factor vectors in Y.
Alternating Least Squares
35
1 0 0 0 1 0 0 10 0 1 0 0 1 0 0 1 0 1 0 0 0 1 10 1 0 0 0 1 0 0 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0 1
X YUsers
Songs
• = bias for user• = bias for item• = regularization parameter
• = 1 if user streamed track else 0• • = user latent factor vector• = item latent factor vector
Fix usersSolve for tracks
Repeat until convergence…
•Aggregate all (user, track) streams into a large matrix•Goal: Approximate binary preference matrix by the inner product of 2 smaller matrices by
minimizing the weighted RMSE (root mean squared error) using a function of total plays as weight •Why?: Once learned, the top recommendations for a user are the top inner products between
their latent factor vector in X and the track latent factor vectors in Y.
Alternating Least Squares
Matrix Factorization with MapReduce 36
Reduce stepMap step
u % K = 0i % L = 0
u % K = 0i % L = 1 ... u % K = 0
i % L = L-1
u % K = 1i % L = 0
u % K = 1i % L = 1 ... ...
... ... ... ...
u % K = K-1i % L = 0 ... ... u % K = K-1
i % L = L-1
item vectorsitem%L=0
item vectorsitem%L=1
item vectorsi % L = L-1
user vectorsu % K = 0
user vectorsu % K = 1
user vectorsu % K = K-1
all log entriesu % K = 1i % L = 1
u % K = 0
u % K = 1
u % K = K-1
Figure via Erik Bernhardsson
Matrix Factorization with MapReduce 37
One map taskDistributed
cache:All user vectors where u % K = x
Distributed cache:
All item vectors where i % L = y
Mapper Emit contributions
Map input:tuples (u, i, count)
where u % K = x
andi % L = y
Reducer New vector!
Figure via Erik Bernhardsson
38
•Fast and general purpose cluster computing system•Provides high-level apis in Java, Scala, and Python•Takes advantage of in-memory caching to reduce I/O bottleneck of
Hadoop MapReduce•MLlib: Scalable Machine Learning library packaged with Spark -Collaborative Filtering and Matrix Factorization-Classification and Regression-Clustering-Optimization Primitives•Spark Streaming: Real time, scalable, fault-tolerant stream processing•Spark SQL: allows relational queries expressed in SQL, HiveQL, or
Scala to be executed using Spark
Matrix Factorization with Spark 39
streams user vectors item vectors
worker 1 worker 2 worker 3 worker 4 worker 5 worker 6
•Partition streams matrix into user (row) and item (column) blocks, partition, and cache-Unlike with the MapReduce implementation, ratings are never shuffled across the network!•For each iteration:1. Compute YtY over item vectors and broadcast2. For each item vector, send a copy to each user rating partition that requires it (potentially all partitions) 3. Each partition aggregates intermediate terms and solves for optimal user vectors
Matrix Factorization with Spark 40
user vectors item vectors
worker 1 worker 2 worker 3 worker 4 worker 5 worker 6
•Partition streams matrix into user (row) and item (column) blocks, partition, and cache-Unlike with the MapReduce implementation, ratings are never shuffled across the network!•For each iteration:1. Compute YtY over item vectors and broadcast2. For each item vector, send a copy to each user rating partition that requires it (potentially all partitions) 3. Each partition aggregates intermediate terms and solves for optimal user vectors
streams
Matrix Factorization with Spark 41
user vectors item vectors
•Partition streams matrix into user (row) and item (column) blocks, partition, and cache-Unlike with the MapReduce implementation, ratings are never shuffled across the network!•For each iteration:1. Compute YtY over item vectors and broadcast2. For each item vector, send a copy to each user rating partition that requires it (potentially all partitions) 3. Each partition aggregates intermediate terms and solves for optimal user vectors
worker 1 worker 2 worker 3 worker 4 worker 5 worker 6
streams
Matrix Factorization with Spark 42
user vectors item vectors
worker 1 worker 2 worker 3 worker 4 worker 5 worker 6
YtY YtY YtY YtY YtY YtY
•Partition streams matrix into user (row) and item (column) blocks, partition, and cache-Unlike with the MapReduce implementation, ratings are never shuffled across the network!•For each iteration:1. Compute YtY over item vectors and broadcast2. For each item vector, send a copy to each user rating partition that requires it (potentially all partitions) 3. Each partition aggregates intermediate terms and solves for optimal user vectors
streams
Matrix Factorization with Spark 43
user vectors item vectors
worker 1 worker 2 worker 3 worker 4 worker 5 worker 6
YtY YtY YtY YtY YtY YtY
•Partition streams matrix into user (row) and item (column) blocks, partition, and cache-Unlike with the MapReduce implementation, ratings are never shuffled across the network!•For each iteration:1. Compute YtY over item vectors and broadcast2. For each item vector, send a copy to each user rating partition that requires it (potentially all partitions) 3. Each partition aggregates intermediate terms and solves for optimal user vectors
streams
Matrix Factorization with Spark 44
user vectors item vectors
worker 1 worker 2 worker 3 worker 4 worker 5 worker 6
YtY YtY YtY YtY YtY YtY
•Partition streams matrix into user (row) and item (column) blocks, partition, and cache-Unlike with the MapReduce implementation, ratings are never shuffled across the network!•For each iteration:1. Compute YtY over item vectors and broadcast2. For each item vector, send a copy to each user rating partition that requires it (potentially all partitions) 3. Each partition aggregates intermediate terms and solves for optimal user vectors
streams
45
Vs
http://www.slideshare.net/Hadoop_Summit/spark-and-shark
Matrix Factorization with MapReduce
Matrix Factorization with Spark
Scala Breeze 46
•Native Scala numerical processing library•Linear Algebra-Matrix operations-Operator overloading and syntactic sugar•Sampling from Probably Distributions•Numerical Optimization•Plotting and Visualizations•Numpy for Scala
Zeppelin + Spark + Parquet for ETL 47
Zeppelin + Spark + Parquet for ETL 48
What should I be worried about? 49
•Multiple “right” ways to do the same thing•Implicits can make code difficult to navigate•Learning curve can be tough•Avoid flattening before a join•Be aware that Scala default collections are immutable (though mutable
versions are also available)•Use monoid reduces and aggregations where possible and avoid folds•Be patient with the compiler
Section name 50
Fin