Embed Size (px)
Citation preview
5/22/18
1
STAT/CSE 416: Intro to Machine Learning
Recommending Products:Matrix Factorization
©2018 Emily Fox
STAT/CSE 416: Intro to Machine LearningEmily FoxUniversity of WashingtonMay 22, 2018
STAT/CSE 416: Intro to Machine Learning
Solution 0: Popularity
©2018 Emily Fox
5/22/18
2
STAT/CSE 416: Intro to Machine Learning3
Simplest approach: Popularity
What are people viewing now?- Rank by global popularity
Limitation: - No personalization
©2018 Emily Fox
STAT/CSE 416: Intro to Machine Learning
Solution 1: Classification model
©2018 Emily Fox
5/22/18
3
STAT/CSE 416: Intro to Machine Learning5
What’s the probability I’ll buy this product?
©2018 Emily Fox
User info
Purchase history
Product info
Other info
Classifier
Yes!
No
Pros:- Personalized:
Considers user info & purchase history- Features can capture context:
Time of the day, what I just saw,…- Even handles limited user history:
Age of user, …
Cons:- Features may not be available- Often doesn’t perform as well
as collaborative filtering methods (next)
STAT/CSE 416: Intro to Machine Learning
Solution 2: People who bought this also bought…
©2018 Emily Fox
5/22/18
4
STAT/CSE 416: Intro to Machine Learning7
Co-occurrence matrix
• People who bought diapers also bought baby wipes
• Matrix C:store # users who bought both items i & j- (# items x # items) matrix
- Symmetric: # purchasing i & j same as # for j & i (Cij = Cji)
©2018 Emily Fox
STAT/CSE 416: Intro to Machine Learning8
(Weighted) Average of purchased items
User bought items {diapers, milk}- Compute user-specific score for each item j in inventory by
combining similarities:
Score( , baby wipes) = ½ (Sbaby wipes, diapers + Sbaby wipes, milk)
- Could also weight recent purchases more
Sort Score( , j ) and find item j with highest similarity
©2018 Emily Fox
5/22/18
5
STAT/CSE 416: Intro to Machine Learning
Solution 3: Discovering hidden structureby matrix factorization
©2018 Emily Fox
STAT/CSE 416: Intro to Machine Learning10 ©2018 Emily Fox
TrainingData
Featureextraction
ML model
Qualitymetric
ML algorithm
ŵy
x ŷ
5/22/18
6
STAT/CSE 416: Intro to Machine Learning11
Users watch movies and rate them
Movie recommendation
©2018 Emily Fox
User Movie Rating
Each user only watches a few of the available movies
STAT/CSE 416: Intro to Machine Learning12
Matrix completion problem
Data: Users score some movies
Goal: Filling missing data?
Xij known for black cellsXij unknown for white cells
Rows index moviesColumns index users
X =
Rating(u,v) known for black cellsRating(u,v) unknown for white cells
Rating
©2018 Emily Fox
Users
Mo
vies
5/22/18
7
STAT/CSE 416: Intro to Machine Learning13 ©2018 Emily Fox
TrainingData
Featureextraction
ML model
Qualitymetric
ML algorithm
ŵy
x ŷ
STAT/CSE 416: Intro to Machine Learning14
Suppose we had d topics for each user & movie• Describe movie v with topics Rv
- How much is it action, romance, drama,…
• Describe user u with topics Lu- How much she likes action, romance, drama,…
• Rating(u,v) is the product of the two vectors
• Recommendations: sort movies user hasn’t watched by Rating(u,v)
©2018 Emily Fox
5/22/18
8
STAT/CSE 416: Intro to Machine Learning16
Predictions in matrix form
©2018 Emily Fox
≈X LR’
=Xij known for black cells
Xij unknown for white cellsRows index movies
Columns index usersX =Rating
But we don’t know topics of users and movies…
STAT/CSE 416: Intro to Machine Learning17
Matrix factorization model:Discovering topics from data
• Only use observed values to estimate “topic” vectors Lu and Rv
• Use estimated Lu and Rv for recommendations
©2018 Emily Fox
≈Xij known for black cells
Xij unknown for white cellsRows index movies
Columns index usersX =Rating
Many efficient algorithms for factorization
Parameters of model
5/22/18
9
STAT/CSE 416: Intro to Machine Learning18
Is the problem well posed?Can we uniquely identify the latent factors?
If ruv is described by Lu , Rv what happens if we redefine the “topics” as
Then,
Other (orthonormal) transformations can have the same effect.
©2018 Emily Fox
≈Xij known for black cells
Xij unknown for white cellsRows index movies
Columns index usersX =
STAT/CSE 416: Intro to Machine Learning19 ©2018 Emily Fox
TrainingData
Featureextraction
ML model
Qualitymetric
ML algorithm
ŵy
x ŷ
5/22/18
10
STAT/CSE 416: Intro to Machine Learning20
Matrix factorization objective
• Minimize mean squared error:- (Other loss functions are possible)
• Non-convex objective
©2018 Emily Fox
≈Xij known for black cells
Xij unknown for white cellsRows index movies
Columns index usersX =Rating
Parameters of model
STAT/CSE 416: Intro to Machine Learning21 ©2018 Emily Fox
TrainingData
Featureextraction
ML model
Qualitymetric
ML algorithm
ŵy
x ŷ
5/22/18
11
STAT/CSE 416: Intro to Machine Learning22
Coordinate descent
Goal: Minimize some function g
Often, hard to find minimum for all coordinates, but easy for each coordinate
Coordinate descent:Initialize ŵ = 0 (or smartly…)
while not convergedpick a coordinate j
ŵj ß
©2017 Emily Fox
STAT/CSE 416: Intro to Machine Learning23
Comments on coordinate descent
How do we pick next coordinate?- At random (“random” or “stochastic” coordinate descent), round robin, …
No stepsize to choose!
Super useful approach for many problems- Converges to optimum in some cases (e.g., “strongly convex”)
©2017 Emily Fox
5/22/18
12
STAT/CSE 416: Intro to Machine Learning24
Coordinate descent for matrix factorization
• Fix movie factors Rv , optimize for user factors Lu
• First key insight:
©2018 Emily Fox
minL,R
X
(u,v):ruv 6=?
(Lu ·Rv � ruv)2
STAT/CSE 416: Intro to Machine Learning25
Minimize objective separately for each user
©2018 Emily Fox
minLu
X
v2Vu
(Lu ·Rv � ruv)2• For each user u:
• Second key insight:
5/22/18
13
STAT/CSE 416: Intro to Machine Learning26
Overall coordinate descent algorithm
©2018 Emily Fox
• Fix movie factors, optimize for user factors- Independent least-squares over users
• Fix user factors, optimize for movie factors- Independent least-squares over movies
• System may be underdetermined: • Converges to• Choices of regularizers and impact on algorithm:
minL,R
X
(u,v):ruv 6=?
(Lu ·Rv � ruv)2
minRv
X
u2Uv
(Lu ·Rv � ruv)2
minLu
X
v2Vu
(Lu ·Rv � ruv)2
STAT/CSE 416: Intro to Machine Learning27 ©2018 Emily Fox
TrainingData
Featureextraction
ML model
Qualitymetric
ML algorithm
ŵy
x ŷ
5/22/18
14
STAT/CSE 416: Intro to Machine Learning28
Using the results of matrix factorization
• Discover “topics” Rv for each movie v
• Discover “topics” Lu for each user u
• Score(u,v) is the product of the two vectors èPredict how much a user will like a movie
• Recommendations: sort movies user hasn’t watched by Score(u,v)
©2018 Emily Fox
STAT/CSE 416: Intro to Machine Learning29
Example topics discovered from Wikipedia
Xij known for black cellsXij unknown for white cells
Rows index moviesColumns index users
X = ≈X LR’
=
Application to text data:
©2018 Emily Fox
5/22/18
15
STAT/CSE 416: Intro to Machine Learning
Bringing it all together: Featurized matrix factorization
©2018 Emily Fox
STAT/CSE 416: Intro to Machine Learning32
Limitations of matrix factorization
• Cold-start problem- This model still cannot handle a new user or movie
©2018 Emily Fox
Xij known for black cellsXij unknown for white cells
Rows index moviesColumns index users
X =Rating
5/22/18
16
STAT/CSE 416: Intro to Machine Learning33
Cold-start problem more formally
Consider a new user u’ and predicting that user’s ratings- No previous observations
- Objective considered so far:
- Optimal user factor:
- Predicted user ratings:
©2018 Emily Fox
minL,R
1
2
X
ruv
(Lu ·Rv � ruv)2 +
�u
2||L||2F +
�v
2||R||2F
STAT/CSE 416: Intro to Machine Learning34
Combining features and discovered topics
• Features capture context- Time of day, what I just saw,
user info, past purchases,…
• Discovered topics from matrix factorization capture groups of users who behave similarly- Women from Seattle who teach and have a baby
• Combine to mitigate cold-start problem- Ratings for a new user from features only- As more information about user is discovered,
matrix factorization topics become more relevant
©2018 Emily Fox
User info
Movie infoMODEL
Yes!
No
5/22/18
17
STAT/CSE 416: Intro to Machine Learning35
Collaborative filtering with specified features
• Create feature vector for each movie (often have this even for new movies):
• Define weights on these features for how much all users like each feature
• Fit linear model:
• Minimize:
©2018 Emily Fox
STAT/CSE 416: Intro to Machine Learning36
Building in personalization• Of course, users do not have identical preferences• Include a user-specific deviation from the global set of user weights:
• If we don’t have any observations about a user, use wisdom of the crowd
• As we gain more information about the user, forget the crowd
• Can add in user-specific features, and cross-features, too
©2018 Emily Fox
5/22/18
18
STAT/CSE 416: Intro to Machine Learning37
Featurized matrix factorization—A combined approach
Feature-based approach:- Feature representation of user and movies fixed- Can address cold-start problem
Matrix factorization approach:- Suffers from cold-start problem- User & movie features are learned from data
A unified model:
©2018 Emily Fox
STAT/CSE 416: Intro to Machine Learning38
Blending models• Squeezing last bit of accuracy by blending models• Netflix Prize 2006-2009- 100M ratings
- 17,770 movies
- 480,189 users
- Predict 3 millionratings to highest accuracy
-Winning team blended over 100 models
©2018 Emily Fox
![E u d u / u l l DK í õ r í & o µ ] v v vDK í õ r í ] & o ... · d u À ] v l vDK í õ r í > ] Ì } o u vDK í õ r í / u l l DK í õ r í & o µ ] v v vDK í õ r í](https://img.pdfslide.us/doc/110x75/5f7bc92517496b0fbb1c67af/e-u-d-u-u-l-l-dk-r-o-v-v-vdk-r-o-d-u.jpg)
![K ( ( ] } ( Z } u u ] ] } v ( } v v Æ u ] v ] } v U d Z ... · W r o r u v } o o P } ( v P ] v ] v P U W u ] X ^ o XE } X Z } o o E } X E u Z v l } µ v v µ o / v } u í í ó õ](https://img.pdfslide.us/doc/110x75/5ec7842a4d2ad0013340341d/k-z-u-u-v-v-v-u-v-v-u-d-z-w-r-o-r-u-v-o-o-p.jpg)
![D Æ Á o o r } o Ì u v v U } r ] v ] v v & u ] r ] ^ ] ] í · Microsoft PowerPoint - Presentation12 Author: gajai Created Date: 4/28/2020 12:51:30 PM](https://img.pdfslide.us/doc/110x75/60327d5267df207a10612940/d-o-o-r-o-oe-u-v-v-u-r-v-v-v-u-r-microsoft-powerpoint.jpg)
![Z Z / v ] µ U E ZK ~E&Z/ · ] v µ Ç r u ] r P } À v u v } } ] } v ] v : v v v ] v v ] } v o v X , ] } Z ] E d E](https://img.pdfslide.us/doc/110x75/5f263ea58108b66f6a378e66/z-z-v-u-e-zk-ez-v-r-u-r-p-v-u-v-v-v-v-v.jpg)
![u } ] } v o } } } ( ] P Z r Á ] v P } o ] ] o } µ o ] u D ... · u } ] } v o } } } ( ] P Z r Á ] v P } o ] ] o } µ o ] u ... u } ] } v o } } } ( ] P Z r Á ] v P } o ] ] o } µ](https://img.pdfslide.us/doc/110x75/5e0ee75edea8723fe9108e76/u-v-o-p-z-r-v-p-o-o-o-u-d-u-v-o-.jpg)
![] P ] o ] ] } v } ( Z } r u U Z ] P Z r } } v µ u ] ' µ ] v } µ À ] } · 2018. 11. 23. · v U o } U } D ] o > Ç U W ' ] o o ] Z v ^ o o Ç Ç r, v } ] µ Æ U ( } u Z K ^ ]](https://img.pdfslide.us/doc/110x75/5fd3576db0ef3b7d4a6d7d1f/-p-o-v-z-r-u-u-z-p-z-r-v-u-v-2018.jpg)
![t } l ] v r Á D v P u v W />K & µ v u v o } v À v ] } v U ... · & } µ ] v P } v ] u o U o } Á r } } o µ ] } v 6DPSOH 'R QRW EXUQ 6DPSOH 6DIH VWRUDJH RI FKHPLFDOVH ZDVWH PDWHULDOV](https://img.pdfslide.us/doc/110x75/602500b0f787816e732ab228/t-l-v-r-d-v-p-u-v-w-k-v-u-v-o-v-v-v-u-.jpg)
![W o v } W ^ ] v P Á } l } } v ( ] r } u U ( ] r À ] X Forms/2019-Rex... · W o v } W ^ ] v P Á } l } } v ( ] r } u U ( ] r À ] X ... x](https://img.pdfslide.us/doc/110x75/5e140061736f0945f51c66ac/w-o-v-w-v-p-l-v-r-u-u-r-x-forms2019-rex-w-o.jpg)
![EW î ì í õ r ì ó ^D v P u v } ( / v ( } u ] } v ^ µ ] Ç Z ] l...WK Z h P v ] v K P v ] ] } v ~ ^ U hZKWK> U ^ U E/^ U Zd r h U hZK KEdZK> U^ ^ Z W µ } v ( v P v Ç W^](https://img.pdfslide.us/doc/110x75/5f03733d7e708231d4093def/ew-r-d-v-p-u-v-v-u-v-z-l-wk-z-h.jpg)
![%XOOHWLQ 6WDWLVWLTXH $JULFROH …...W ] Æ ^ µ µ v u } ] ^ µ µ Æ u } ] ^ µ µ v v W } µ ] u r í õ v } À u r í õ } } r í õ u r í ô dh Z h> ^ ï U í ô í U î ô 9](https://img.pdfslide.us/doc/110x75/5f0f5b7f7e708231d443c24e/xoohwlq-6wdwlvwltxh-julfroh-w-v-u-u-.jpg)
![d Æ Ç ] P v } Z v o d Æ u ] v ] ] } v U '^d } u u ] ] } v > µ …centralexciseludhiana.gov.in/uploads/annexure/Annexure-1...x s r u u y t r s t x t r u u w y y s x u r u v w v](https://img.pdfslide.us/doc/110x75/5f2b2d5bacfa1d1be17e74cc/d-p-v-z-v-o-d-u-v-v-u-d-u-u-v-c-x-s-r.jpg)
![v ] µ ] } v o & u Á } l } ( Z v ] r } µ ] } v } u u ] } ( Z W o ] u v ......W } o ] Ç ] ( ^/ v ] µ ] } v o & u Á } l } ( Z v ] r } µ ] } v } u u ] } ( Z W o ] u v } ( D }](https://img.pdfslide.us/doc/110x75/5f7117e4e9dcaf79a54a145b/-v-v-o-u-l-z-v-r-v-u-u-z-w-o-u-v-.jpg)
![Z ] U v ] ] À v ] ( ] ^ Z^ } } v À ] µ r î ] } v W u µ o ] r v } u ] } v Á v v ... · o ] À o } v P µ v r } µ v r ] u ~d d V î r ð Z } µ X / v Z E Z o v U Z } v P v v](https://img.pdfslide.us/doc/110x75/60091178ad56861cc15cd4be/z-u-v-v-z-v-r-v-w-u-o-r-v-u-v-.jpg)
![v Z v ] v P rDz P ] } v À ] î ì^ } } u ] À ] } v ] v r µ ] v À ] À } v ] r µ u ... · } ] v v } u o o Ç P Ç Z î ò^ r µ ] µ ] ] v r r v v } } u Ç u Á v } ( ( X d }](https://img.pdfslide.us/doc/110x75/602b16183c08e8497561ef9f/v-z-v-v-p-rdz-p-v-u-v-v-r-v-v-.jpg)
![u ] v K · î ó ó d v ( > ] · 2020. 10. 6. · ï W î ì r À r ì ì ñ õ õ rE:Z v & µ v ] v P U >> À X t/ } u u ] o > µ v Ç µ ] u v U / v X o ï W î ì r À r ì ì](https://img.pdfslide.us/doc/110x75/608bf56467f21424837b2ee1/u-v-k-d-v-2020-10-6-w-r-r-rez.jpg)
![W Z } W P ^ o ] } v v · 2020. 9. 24. · ^ ] } v õ W W Z r } ] } v U K µ r } ( r^ µ v U s } ] t / v Æ ( ] v Z } ] À ] } v W } µ r ] v o µ ] v P } µ r } ( r µ v v ] } v](https://img.pdfslide.us/doc/110x75/60501b3e634f3c5bf340ac21/w-z-w-p-o-v-v-2020-9-24-v-w-w-z-r-v-u-k-r-r.jpg)
![^ r } ( r Z r ] v } u ] v ] } v / u u µ v } lZ ] } Z Ç ( } E } v r^ u o o o ......H[WUDFUDQLDO 57 YV QR 57 Q e7UHDWPHQW UHODWHG SXOPRQDU\ WR[LFLW\ DIWHU FKHVW 57 YV JUDGH RQH SHU](https://img.pdfslide.us/doc/110x75/611359c83d38ca50d51869bc/-r-r-z-r-v-u-v-v-u-u-v-lz-z-e-v-r-u-o-o-o-.jpg)
![&>h/ E > dZK>zd > E r r î · 2020. 4. 12. · u ] v v v v o u v d Z Ç X / v W Z u v Z U < o ] P u v Z U : v } v , U ] } X E o } v Æ } } l } ( ] ] X í ó Z ] ] }](https://img.pdfslide.us/doc/110x75/5fc40ddc9f211a7e5e54d61e/h-e-dzkzd-e-r-r-2020-4-12-u-v-v-v-v-o-u-v-d-z.jpg)
![1DWLRQDO6FLHQFH)RXQGDWLRQ o } v ] Z Z u ] v ] ] } v ~ Z & } µ u … · 2020-05-14 · & } } } u^ µ } U o o = í r ô ï ï r õ ò ò r ò ð ò ô ~ = í r ô ï ï r } } u r](https://img.pdfslide.us/doc/110x75/5f4298d035b7c2390f54958a/1dwlrqdo6flhqfhrxqgdwlrq-o-v-z-z-u-v-v-z-u-2020-05-14.jpg)
![& u ] o Ç r } r & u ] o Ç î ì í õ d Z d ] v ] v P o ... · & u ] o Ç r } r & u ] o Ç î ì í õ d Z d ] v ] v P o ] ] } v P u v dZ /E/E' >K d/KE W z l ] u î r z dZ /E/E](https://img.pdfslide.us/doc/110x75/5ec7957f20204234931b8b75/-u-o-r-r-u-o-d-z-d-v-v-p-o-u-.jpg)
![storage.googleapis.com...mv|-m| t b t7 v|ol;u ";u b1; " r;uv|-uv >$_; u;- t v;1u;| bm 0;1olbm] - 1 v|ol;u v;u b1; v r;uv|-u ? bm |_; ; ;v o= |_; 1 v|ol;u 5 )_ bv - ®| 7; ; ;u |_bm]](https://img.pdfslide.us/doc/110x75/5f0eb9ca7e708231d440a2ed/-mv-m-t-b-t7-volu-u-b1-ruv-uv-u-t-v1u-bm-01olbm.jpg)
![h & ] Z ] } o ] U µ v v u v o P o } ] o v µ v } v Æ } v } À i · î î õ í ì ì 9 Z W l l Á Á Á X v µ u u X } u l u P Ì ] v l ï í ì ð ð r o r ( ] Z ] r o µ ] v r](https://img.pdfslide.us/doc/110x75/5f9976c9c8406a51d475ed43/h-z-o-u-v-v-u-v-o-p-o-o-v-v-v-v-i-.jpg)
![u ] o ^ Ç u } ] µ u > v } v ( v v U > v h v ] À ] Ç U h v ... · > v } v ( v v U > v h v ] À ] Ç U h v ] < ] v P } u U í í r í î l î l î ì í õ ^ Ç u](https://img.pdfslide.us/doc/110x75/5e64075b0cf9711b3041267b/u-o-u-u-v-v-v-v-u-v-h-v-u-h-v-v-.jpg)
![^ Z ] d Ç } ( ] ] o ] Ç^ Z v u ' v , ] v P / u ] u v U ...edudel.nic.in/welcome_folder/FacilitiesAvailableCWSN201920.pdf · Zd r ð ì ì l U D ] v D l U Z i v µ o Z ] r ñ ï](https://img.pdfslide.us/doc/110x75/60dafe668a4904792c008d4b/-z-d-o-z-v-u-v-v-p-u-u-v-u-zd-r-l-u-d.jpg)
![Fender GT-40 - How to Make it Sound Great, v2 · ï & l r µ v Z } ] l ] v P v r ] u o } µ X / ( Ç } µ o ] l u U / u } ] ( Ç } u v À Z u v Á U } / } u } ( u Ç } Á v ( } u](https://img.pdfslide.us/doc/110x75/600dae86294c7932ac29464b/fender-gt-40-how-to-make-it-sound-great-v2-l-r-v-z-l-v-p-v.jpg)
![} v µ u D ] o r/ v Z & } u - Pfaff](https://img.pdfslide.us/doc/110x75/6247990a9ce1b1281d1c8867/-v-u-d-o-r-v-z-amp-u-pfaff.jpg)
