Introduction Recommender Systemskonstan/CHI2003RS.pdfSample Vendors

Recommender Systems: Interfaces and Technology

Copyright 2002 John Riedl and Joseph A. Konstan

April 7, 2003

CHI 2003 1

CHI 2003 1

Recommender Systems: Collaborating in Commerce and Communities

Joseph A. KonstanJohn Riedl

University of Minnesota{konstan,riedl}@cs.umn.edu

http://www.cs.umn.edu/Research/GroupLens

CHI 2003 2

Introduction

What are Recommender Systems?

Goals of this Tutorial

Brief History of Recommender Systems

CHI 2003 3

The Problem: Overload

CHI 2003 4

Too much stuff!

Too many books!Too many journal articles!

Too many movies!Too much content!

Too many product choices!Heck, even too many people!

CHI 2003 5

Recommenders

Tools to help identify worthwhile stuffFiltering interfaces

E-mail filters, clipping servicesRecommendation interfaces

Suggestion lists, “top-n,” offers and promotions

Prediction interfacesEvaluate candidates, predicted ratings

CHI 2003 6

Scope of Recommenders

Purely Editorial Recommenders

Content Filtering Recommenders

Collaborative Filtering Recommenders

Hybrid Recommenders



April 7, 2003

CHI 2003 2

CHI 2003 7

Tutorial Goals and Outline

CHI 2003 8

Goals

When you leave, you should …Understand recommender systems and their applicationKnow enough about recommender systems technology to evaluate application ideasBe able to design and critique recommender application designsSee where recommender systems have been, and where they are going

CHI 2003 9

OutlineIntroductionRecommender Systems Application SpaceMovieLens Case StudyRecommender AlgorithmsEight Principles and Case StudiesDesigning Recommender ApplicationsPrivacy IssuesCommercial Tool Survey

CHI 2003 10

Highlights andFocus Areas

E-Commerce RecommendersCommunity RecommendersKnowledge Management RecommendersOff-The-Desktop RecommendersGroup Application Design ExercisePlenty of Discussion

CHI 2003 11

History of Recommender Systems

CHI 2003 12

The Early Years …

Why cave dwellers survived

Critics, critics, everywhere

The dreaded editor



April 7, 2003

CHI 2003 3

CHI 2003 13

Automation

Information retrievalDynamic information needStatic content base

Information filteringStatic information needDynamic content base

CHI 2003 14

Collaborative Filtering

PremiseInformation needs more complex than keywords or topics: quality and taste

Small Community: ManualTapestry – database of content & commentsActive CF – easy mechanisms for forwarding content to relevant readers

CHI 2003 15

Automated CF

The GroupLens Project (CSCW ’94)ACF for Usenet News

users rate itemsusers are correlated with other userspersonal predictions for unrated items

Nearest-Neighbor Approachfind people with history of agreementassume stable tastes

CHI 2003 16

Usenet Interface

CHI 2003 17

ACF Blossomed

1995Ringo (later Firefly)Bellcore Video Recommender

1996 Recommender Systems Workshop

Early commercializationAgents Inc. (later Firefly)Net Perceptions

new issues of scale and performance!CHI 2003 18

Today

Broad commercial applicationWidely used in e-commerceavailable commercial tools

Diverse research communitylive research systemssubstantial integration with machine learning, information filtering, HCI, and more



April 7, 2003

CHI 2003 4

CHI 2003 19

Introductions

CHI 2003 20

Who are We?

John RiedlCollaborative and distributed systems

Joe KonstanHuman-computer interaction

GroupLens ResearchNet PerceptionsWord of Mouse

CHI 2003 21

Who are You?

NameWhat you doWho you work for / where you studyBriefly

Your experience with recommender systemsOne key thing you want to get out of this tutorial

CHI 2003 22

The Recommender Application Space

CHI 2003 23

Dimensions of Analysis

DomainPurposeWhose OpinionPersonalization LevelPrivacy and TrustworthinessInterfaces<Algorithms Inside>

CHI 2003 24

Domains of Recommendation

Content to CommerceNews, information, “text”Products, vendors, bundles



April 7, 2003

CHI 2003 5

CHI 2003 25

Purposes of Recommendation

The recommendations themselvesSalesInformation

Education of user/customer

Build a community of users/customers around products or content

CHI 2003 26

Whose Opinion?

“Experts”

Ordinary “phoaks”

People like you

CHI 2003 27

Personalization Level

GenericEveryone receives same recommendations

DemographicMatches a target group

EphemeralMatches current activity

PersistentMatches long-term interests

CHI 2003 28

Privacy and Trustworthiness

Who knows what about me?Personal information revealedIdentityDeniability of preferences

Is the recommendation honest?Biases built-in by operator

“business rules”Vulnerability to external manipulation

CHI 2003 29

Interfaces

Types of OutputPredictionsRecommendationsFilteringOrganic vs. explicit presentation

Types of InputExplicitImplicit

CHI 2003 30

MovieLens Case Study



April 7, 2003

CHI 2003 6

CHI 2003 31

Key Concepts

UsersItems – MoviesRatings – Explicit, 1 to 5 starsCorrelations – Relationships among usersPredictions – Personal predicted ratingsRecommendations – Suggested Items

CHI 2003 32

How It Works

C.F. Engine

Ratings Correlations

CHI 2003 33

GroupLens Model

C.F. Engine


ratings

CHI 2003 34

GroupLens Model

C.F. Engine


ratings

CHI 2003 35

GroupLens Model

C.F. Engine


ratings

CHI 2003 36

GroupLens Model

C.F. Engine


ratings

request



April 7, 2003

CHI 2003 7

CHI 2003 37

GroupLens Model

C.F. Engine


ratings

request

Neighborhood

CHI 2003 38

GroupLens Model

C.F. Engine

Ratings CorrelationsNeighborhood

ratings

requestpredictions

recommendations

CHI 2003 39

Understanding the Computation

Hoop Dreams

Star Wars

Pretty Woman

Titanic Blimp Rocky XV

Joe D A B D ? ? John A F D F Susan A A A A A A Pat D A C Jean A C A C A Ben F A F Nathan D A A

CHI 2003 40

Hoop Dreams

Star Wars

Pretty Woman




CHI 2003 41

Hoop Dreams

Star Wars

Pretty Woman




CHI 2003 42

Hoop Dreams

Star Wars

Pretty Woman






April 7, 2003

CHI 2003 8

CHI 2003 43

Hoop Dreams

Star Wars

Pretty Woman




CHI 2003 44

Hoop Dreams

Star Wars

Pretty Woman




CHI 2003 45

Hoop Dreams

Star Wars

Pretty Woman




CHI 2003 46

ML-home

CHI 2003 47

ML-comedy

CHI 2003 48

ML-clist



April 7, 2003

CHI 2003 9

CHI 2003 49

ML-rate

CHI 2003 50

ML-search

CHI 2003 51

ML-slist

CHI 2003 52

ML-groups

CHI 2003 53

Recommender Algorithms

CHI 2003 54

Collaborative Filtering

?

TargetTargetCustomerCustomer

Weighted Sum

3



April 7, 2003

CHI 2003 10

CHI 2003 55

E-Commerce Scale

Millions of ProductsMillions of CustomersThousands of Clicks per Second

Scalability!

CHI 2003 56

Sparsity

Many customers have no relationshipMany products have no relationshipSynonymy

Similar products treated differentlyIncreases sparsity, loss of transitivityResults in poor quality

CHI 2003 57

Item-Item Collaborative Filtering

I

II

II I II

I

I

I

I

I

I

I

I

I

CHI 2003 58


I

II

II I II

I

I

I

I

I

I

I

I

I

CHI 2003 59


I

II

II I II

I

I

I

I

I

I

I

I

I

CHI 2003 60

Item-Item Discussion

Good quality, in sparse situationsPromising for incremental model

buildingSmall quality degradationBig performance gain



April 7, 2003

CHI 2003 11

CHI 2003 61

Another Possible Solution:Dimensionality Reduction

Latent Semantic IndexingUsed by the IR communityWorked well with the vector space modelUsed Singular Value Decomposition (SVD)

Main IdeaTerm-document matching in feature spaceCaptures latent associationReduced space is less-noisy

CHI 2003 62

SVD: Mathematical Background

=R

m X n

U

m X r

S

r X r

V’

r X n

Sk

k X k

Uk

m X k

Vk’

k X n

The reconstructed matrix Rk = Uk.Sk.Vk’ is the closest rank-k matrix to the original matrix R.

Rk

CHI 2003 63

SVD for Collaborative Filtering

. 2. DirectPredictionm x n

1. Low dimensional representation O(m+n) storage requirement

m x k

k x n

CHI 2003 64

Singular Value Decomposition

Reduce dimensionality of problemResults in small, fast modelRicher Neighbor Network

Incremental UpdateFolding inModel Update

CHI 2003 65

Collaborative Filtering Algorithms

Non-Personalized Summary StatisticsK-Nearest Neighbor

User-UserItem-Item

Dimensionality ReductionSingular Value DecompositionFactor Analysis

Content + Collaborative FilteringCHI 2003 66

Collaborative Filtering Algorithms

Graph TechniquesHorting: Navigate Similarity Graph

ClusteringClassifier Learning

Rule-Induction LearningBayesian Belief Networks



April 7, 2003

CHI 2003 12

CHI 2003 67

Case Studies in Personalization

CHI 2003 68

Overview

Commercial Agents that Treat You RightLands’ End, Clinique, Launch

We Know (Knew) Who You AreDoubleClick, Angarra, and their apps

Looking for An Opportunity?Product of the Month Clubs, TiVo

Straight from the LabCalendar Apprentice, Moods, and Effective Interrogation

CHI 2003 69

Lands’ End

The ChallengeGet people to buy clothes at a distance

Some Personal ToolsMy Personal Shopper

Learn preferences, recommend, support short-term interests

My Virtual ModelLearn body, use to help customers visualize

CHI 2003 70

My Personal Shopper

CHI 2003 71

Profile Data

CHI 2003 72

Results



April 7, 2003

CHI 2003 13

CHI 2003 73

Today’s Interest

CHI 2003 74

My Virtual Model

CHI 2003 75

Seeing MVM

CHI 2003 76

Using MVM

CHI 2003 77

Clinique

The ChallengeBuy make-up online?Attract people to their brand online?

Some Personal ToolsPersonal Consultation

Interview to determine skin typeLooks Maker (of blessed memory)

Match products you already have

CHI 2003 78

Personal Consultation



April 7, 2003

CHI 2003 14

CHI 2003 79

PC – 2

CHI 2003 80

Used

CHI 2003 81

Launch

Personalized RadioSome neat stuff

Runs with rules of real radioExplains why music is playingAllows ratings of Artists, Albums, SongsRepetition (to a point) is Good!Die! Die! Die! Button

SadlyNotion of “rating” is a bit strange

CHI 2003 82

Launch 1

CHI 2003 83

Launch 2

CHI 2003 84

Launch 3



April 7, 2003

CHI 2003 15

CHI 2003 85

Launch 4

CHI 2003 86

Launch 5

CHI 2003 87

Launch Home

CHI 2003 88

Launch Levels

CHI 2003 89

What About First Timers?

DoubleclickPopular banner ad networkDecided in early 2002 to abandon full personalization

AngarraTracks users and compiles “non-identifying”personalizing informationMakes this data available for first-time personalization (to the segment level)Includes control cases to test effectiveness

CHI 2003 90

Brooks Brothers



April 7, 2003

CHI 2003 16

CHI 2003 91

Could Do Better

Product of the Month ClubsForce customers into genre categoriesPotential of automatic shipment

Do you trust Dean and Deluca?

TiVo – Digital Video RecorderGathers lots and lots of dataStill does a lousy job filling “time”

CHI 2003 92

From The Laboratory

Mitchell’s Calendar ApprenticeLearns rules of how to scheduleTakes over as confidence and user permit

Effective InterrogationValue-of-information analysisWhich items to ask about?

High popularity? High Entropy?Value to individual? Value to Community?System-Driven? User-Driven?

CHI 2003 93

FMRL Movie Recommendations

User submits “theme” queryTheme contains examples of items similar to those desired by the user

Set of potentially similar items identifiedUsing item-to-item correlation in ratings space

Potentially similar items ranked based on traditional ACF predictions

CHI 2003 94

Theme Creation

Meeting Ephemeral Needs

CHI 2003 95

Theme Selection

CHI 2003 96

Theme Query Interface – Query Results



April 7, 2003

CHI 2003 17

CHI 2003 97

Principles LearnedBe a Customer Agent

Listen, Learn, and UseAnticipate PitfallsBring “Inside” Opportunities and InfoMake the Match

Box Products, Not PeopleIndividuals, not DemographicsEvolving PersonalizationReal-Time Updates

CHI 2003 98

Case Studies inKnowledge Management

CHI 2003 99

Overview

Recommending peopleeBay, ReferralWeb

Recommending DocumentsTacit

Privacy Issues

CHI 2003 100

Ebay feedback profile

CHI 2003 101

ReferralWeb

CHI 2003 102

Tacit



April 7, 2003

CHI 2003 18

CHI 2003 103

Privacy Issues

Same as E-commerce, plusExtra sensitivity of profile data

E.g., Tacit’s dual profiles

Honesty/openness vs. edited content

CHI 2003 104

Principles

Recommending documentsEasier to “keep up to date”Automatic routing of content to peopleMore effective researchMore valuable portals and services

Advertising revenue!

Recommending peopleMore efficient organizationsSocial stickiness

CHI 2003 105

Principles

Key issues:Many techniques for analyzing data (keywords, associations, etc.)Control and data privacyGetting honest data when users have ulterior motives

CHI 2003 106

Case Studies in Expertise

CHI 2003 107

Overview

Overcoming ReluctancePriceline Hotels, Ticketmaster

Helping Users Explore PossibilitiesEntrée, Confidence

Wasted OpportunitySee’s Candies

CHI 2003 108

Priceline

The ChallengeGet you to commit money to a hotel they won’t identify

The ApproachesReassure that many have succeeded beforeDemonstrate knowledge of the localeCarefully explain rating systemHelp users with price guidelines



April 7, 2003

CHI 2003 19

CHI 2003 109

PL1

CHI 2003 110

PL2

CHI 2003 111

PL3

CHI 2003 112

PL4

CHI 2003 113

TM Hockey

CHI 2003 114

Entrée



April 7, 2003

CHI 2003 20

CHI 2003 115

ML with Conf

CHI 2003 116

Sees

CHI 2003 117

Principles LearnedDemonstrate Product Expertise

Use expertise and recommenders to build customer trustProvide deep product data, so that customers can make informed decisionsMake it fun!

CHI 2003 118

Case Studies in Community Recommenders

CHI 2003 119

Overview

Use Communities to Create ContentEpinions, Slashdot, Flutter.com, Social Navigation, Filterbots

Turn Communities into ContentGeocities, MatchMaker, LambdaMoo, MSN Games

CHI 2003 120

Epinions Sienna overview



April 7, 2003

CHI 2003 21

CHI 2003 121

Epinions profile

CHI 2003 122

Epinions profile bottom

CHI 2003 123

Epinions earnings

CHI 2003 124

Slashdot Front page

CHI 2003 125

Slashdot Discussion

CHI 2003 126

Flutter.com (betfair.com)



April 7, 2003

CHI 2003 22

CHI 2003 127

Social Navigation

Assumptions:Awareness of others (current or past) helps user find relevant pathPaths/location of others is distinctive enough for user to recognize

ApproachMake history or present visible

CHI 2003 128

Social Nav Example

CHI 2003 129

Filterbots

CHI 2003 130

Inspiration

Need selfless, consistent raters

Humans? No: ratings robots.

CHI 2003 131

Filterbot Integration

GroupLens

CHI 2003 132

Filterbots Alone



April 7, 2003

CHI 2003 23

CHI 2003 133

GroupLens

Filterbots Apart

CHI 2003 134

Advantages of the FilterBot model

Combines best of agents and humansagents rate frequently, quickly, consistentlyhumans add subjective taste and quality

Framework pulls out the best of eachuse only the bots that work; ignore the othersuse only the people who agree; ignore the othersbalance people and bots based on available ratings and agreement

CHI 2003 135

Risks of Filterbots

• What if no humans read certain articles?“voluntary” censorship or quality control?

• What about rogue filterbots?

• What if people “prefer” filterbots to humans?

CHI 2003 136

The Filterbot Rainbow

Pure CF PersonalAgent

CF + Filterbot Agents + CF

MixedCommunity

CHI 2003 137

Yahoo & GeoCities

Yahoo bought GeoCities, which than tried to change rules on ownership of content; customers protested by “blacking out” all their content until site returned to old rules.

CHI 2003 138

Slashdot article about Geocities



April 7, 2003

CHI 2003 24

CHI 2003 139

Geocities Personal Page

CHI 2003 140

Turn Communities into Content

CHI 2003 141

Matchmaker: Seeker Features

CHI 2003 142

Lambda Moo Map

CHI 2003 143

Game site: MSN

CHI 2003 144

Principles

Editorial process is value addedHelp customers find interesting

information from other customersFree is better than paying for it

customers trust what they produceReward creativelyEnable customer control (e.g., social nets)Encourage response (e.g., eBay)



April 7, 2003

CHI 2003 25

CHI 2003 145

Case Studies in Using Implicit Preference Data

CHI 2003 146

Overview

Gather “Work Product” DataGoogle, PHOAKS, OWL

Make Implicit Ratings VisibleAmazon

Don’t Ignore Me!My Yahoo!

CHI 2003 147

Google PageRank

• Ranks pages based on incoming links• Links from higher ranked pages matter

more• Combines text analysis with importance

to decide which pages to show you• Runs on network of thousands of PCs!• Works to be hard to trick (e.g., citation

trading)

CHI 2003 148

GOOGLE

CHI 2003 149

PHOAKS

Read Usenet news to find web sites!Implicit ratingsFilter URLs to find endorsementsCreate top-n lists of web sites for a Usenet newsgroup community

Links to endorsements (with age shown)Tested against hand-maintained FAQ lists

CHI 2003 150

PHOAKS



April 7, 2003

CHI 2003 26

CHI 2003 151

MITRE’s OWL

Recommender for word processorsMonitored word processor command useIdentified common patterns of useRecommended commands to learn

CHI 2003 152

OWL

CHI 2003 153

Amazon.com

Broad collection of recommendersLots of use of purchase data

But what if it was a gift?What if you didn’t like it?What if your account is shared?

Makes this data visibleImprove your recommendationsExplain a recommendation

CHI 2003 154

Amazon Improve Your Recommendations

CHI 2003 155

Amazon Explanation

CHI 2003 156

My Yahoo!

Typical portalUser control over “streams” of contentStreams managed independentlyDoesn’t track usage to update

Already seenWhat I seem to like/dislike

Doesn’t suggest streams I might like



April 7, 2003

CHI 2003 27

CHI 2003 157

My Yahoo

CHI 2003 158

Principles Learned

Watch What I DoActions speak louder than wordsDetermine actions by contextMake implicitly gathered data visibleRespond to customers’ reactions to your recommendations

CHI 2003 159

Case Studies around Touchpoints

CHI 2003 160

Outline

KiosksCall CentersMobileZagats

CHI 2003 161

Kiosks

Alienware PC's Now Offered on Best Buy ``Computer Creation Stations'‘

Blockbustercustomer identityprivacy issues

Music Storesampling versus “listening”

CHI 2003 162

Call Centers

Inbound“screen-pops”Legacy systemsappropriateness

OutboundPredict who will buyPredict what they will buyPredict when to contact themOnline campaign management



April 7, 2003

CHI 2003 28

CHI 2003 163

GUS Call Center Experiment

Consumers call in purchasesImplemented personalization, tooExperiment:

one group of agents with old methodone group of agents with personalization

CHI 2003 164

With traditional With cross-sell realtime methods recommendations

Avg Cross-Sell Value $19.50* 60%higher

Cross Sell Success Rate 9.8% 50%higher

* Converted from British Pounds

Directed Buyers: Increasing Cross-Sell

CHI 2003 165

WMLLens Login

CHI 2003 166

Consumer Desires

You will love Attack of the Giant

Leeches!

I have 20,000Copies

If Only I couldBring thisShopping!

CHI 2003 167

Consumer Desires

Recommendations wherever I amRecommendations that I can trustControl over what preferences I shareControl over who I share with

CHI 2003 168

Recommendations Unplugged

What movieShould I

see?

What good movies

are close by?

What DVDShould I

buy?

Tell me what I should

See!

Wireless PDA

Voice

WML

Experimental questions•How do users interact?•What usage patterns?•What happens as users gain experience?

•How do different modalities compare?

•How does usage compare with web?

Avant Go



April 7, 2003

CHI 2003 29

CHI 2003 169

Peer-to-Peer Recommenders

Builds on model based item-item algorithm

Separate model construction from usageKey Question: How to find neighbors?Incrementally build a model just for me

CHI 2003 170

Zagat

CHI 2003 171

Zagat What it Takes

What happened to my favorite guide?They let you rate the restaurants!

What should be done?Personalized guides, from the people who “know good restaurants!”

CHI 2003 172

Principles Learned

Maintain excellent service across touchpoints

It’s still you however your customers get thereDifferent strokes for different folksEquivalent service does not mean equal service

CHI 2003 173

Designing Recommender Applications

CHI 2003 174

The O-I-P Model

OutputsPresenting recommendations

InputsGathering preference and product data

ProcessProducing recommendations from inputs



April 7, 2003

CHI 2003 30

CHI 2003 175

Outputs

TypesSuggestions, predictions, ratings, reviews

DeliveryPush, pull, organic/passive

PresentationLists, annotations, annotations

CHI 2003 176

Inputs

From the targeted userExplicit vs. implicitRatings, purchase history, keyword/attribute, navigation

From the communityProduct attributes, popularity, ratings, purchase histories, reviews, navigation

CHI 2003 177

Process

Manual recommendation – editorsStatistical summarizationAttribute-based selection

Ephermeral personalizationCurrent selectionMarket Basket

Long-Term personalization

CHI 2003 178

Exercise

Work in groups of 3-4

Identify a recommender applicationE-commerce or community application (broadly defined)Ideally tied to one of your research interests, businesses, or personal interests

Outline desired outputs, available inputsDiscuss appropriate processes

CHI 2003 179

Privacy

The conflict between recommender systems and

privacy

CHI 2003 180

Privacy versus Personalization

Personalization

Privacy



April 7, 2003

CHI 2003 31

CHI 2003 181

Some Stories

ToysmartAmazon Pricing

CDnow email

CHI 2003 182

Privacy or Trust?

Customers are willing to share dataIf they trust the recipientIf they know what it is being used forIf they benefit

Customers are very waryIf they fear their data is being soldIf they fear it is being used to take advantage of them

CHI 2003 183

P3P from W3C

Web Server PersonalizationEngine

UserProfile

Profile Parser

PrivacyPolicy

ProfileData

CHI 2003 184

Where are the Profiles?

Web client

Web server

Smart Card

PDA

Encrypt

CHI 2003 185

Microsoft Hailstorm

P3P approach to PrivacyMicrosoft volunteers to keep the profiles

Consumer response?Great silence

Business response!Not with our customers!

CHI 2003 186

Commercial Tools



April 7, 2003

CHI 2003 32

CHI 2003 187

Tools

Data mining for offline analyticsCampaign management so marketer has

something to doPersonalization for real-time decision-makingDynamic content for Web site Dynamic Content

+ PersonalizationApproach: Goals, Relate to Recommender

Systems, Sample Vendors

CHI 2003 188

Commercial ToolsGoals

• Analyze customer and traffic data to drive business decisions

• Simplify deployment of analytic solutions

• “Close the loop” between the marketer and the customer

DynamicContent

Data Mining

CampaignMgmt Personalize

CHI 2003 189

Commercial ToolsRelate to Personalization

•Similar analytic techniques•Migrating towards real-time

solutions•Lessons learned: human-in-

the-loop stops the loop

DynamicContent

Data Mining


CHI 2003 190

Commercial ToolsSample Vendors

• digiMine• Accrue• SPSS (+NetGenesis)

DynamicContent

Data Mining


CHI 2003 191


• Run inbound and outbound campaigns

• Derive more revenue from customers

• Evaluate campaign performance

DynamicContent

Data Mining


CHI 2003 192


• Choose people to campaign at• Choose products to offer• Work “for” rather than “at”

customers

DynamicContent

Data Mining




April 7, 2003

CHI 2003 33

CHI 2003 193


• E.piphany (www.epiphany.com)• Kana (www.kana.com)• Teradata (NCR, www.teradata.com)

DynamicContent

Data Mining


CHI 2003 194


• Recommend products or content to site visitors

• Help customers find value from site• Create strong relationships with

customers

DynamicContent

Data Mining


CHI 2003 195


• Recommender systems can be used to personalize

• Personalization has very broad meaning: “Hello John” to “I know what you did last summer”

DynamicContent

Data Mining


CHI 2003 196


• Net Perceptions (www.netperceptions.com)

• LikeMinds (www.macromedia.com)• TripleHop (www.triplehop.com)

DynamicContent

Data Mining


CHI 2003 197


• Produce HTML content from database

• Simplify authoring, editing, and management of site

• Support interactivity and personalization

DynamicContent

Data Mining


CHI 2003 198


• Often provide simple rules to decide how to deliver content

• Other vendors must work in dynamic content environment

DynamicContent

Data Mining




April 7, 2003

CHI 2003 34

CHI 2003 199


• Vignette (www.vignette.com)• Broadvision(www.broadvision.com)• Microsoft CMS

DynamicContent

Data Mining


CHI 2003 200

Holy Grail

Combine technologies across entire spectrum

Data mining for offline analyticsCampaign management to drive marketing decisions to customersPersonalization for real-time decision-makingDynamic content for Web site

CHI 2003 201

Conclusions

CHI 2003 202

Trust and Usability

Recommendation is essentially task-basedTricky issues of balance and focus

Correctness vs. ValueDiversity

People notice poor or manipulated recommendations (Amazon, MovieLens)

Interfaces are evolvingMore integrated into task flow

CHI 2003 203

Real-world ExperienceLarge international catalog retailer

17% hit rate, 23% acceptance rate in call centerMedium European outbound call center

17% hit rate, 6.7% acceptance rate from an outbound telemarketing call$350.00 price of average item soldItems were in an electronics over-stocked category and were sold-out within 3 weeks

Medium American online children’s product store (e-mail campaign)

19% click-thru rate vs. 10% industry average14.3% conversion to sale vs. 2.5% industry average

CHI 2003 204

The Golden Rule

“The one who has the gold, makes the rules.”

Price BotsJunglee (Excite), Jango (Amazon), MySimon

Opinion LeadersDeja.com (Half.com), AskIda (Best Buy), Epinions

RecommendersOwned by businesses



April 7, 2003

CHI 2003 35

CHI 2003 205

Ida1

CHI 2003 206

Ida2

CHI 2003 207

Ida3

CHI 2003 208

IdaRecs

CHI 2003 209

IdaZoom

CHI 2003 210

IdaMoreRecs



April 7, 2003

CHI 2003 36

CHI 2003 211

Detail

CHI 2003 212

Compare

CHI 2003 213

Compare2

CHI 2003 214

Recommender System for User

Find what I wantKnow I will like itTrust system to help meTeam up with my friends to defeat evil

marketers

CHI 2003 215

Recommender System for Marketer

Show people what they will buyLearn what people want so you have itLearn how much they want it so you get

as much as possible

CHI 2003 216

Who will Prevail?

Who is deploying recommender systems?Who has the money?

Consumers will react if trickedAlternatives exist, and will be deployed if

necessary



April 7, 2003

CHI 2003 37

CHI 2003 217

Recommender Systems

• Becoming necessary for e-commerce• Create value for businesses• Create value for customers• Many open research problems

TechnologyDeployment and InterfacesEffectiveness

CHI 2003 218

Discussion