Upload
mounia-lalmas
View
2.017
Download
5
Tags:
Embed Size (px)
Citation preview
- 1 -
User Engagement: from Sites to a Network of Sites
or The Network Effect Matters!
Ricardo Baeza-Yates Mounia Lalmas
Yahoo! Labs Barcelona
Joint work with Janette Lehmann and Elad Yom-Tov and many others at Yahoo! Labs
- 2 -
Outline
o Motivation, definition and scope
o Models of user engagement
o Networked user engagement
- 3 -
Motivation, Definition and Scope
o Definition and scope
o Characteristics of user engagement
o Measures of user engagement
o Our research agenda
- 4 -
User Engagement – connecting three sides
Quality of the user experience that emphasizes the positive aspects of the interaction, and in particular the phenomena associated with users wanting to use a web application longer and frequently.
Successful technologies are not just used, they are engaged with
user feelings: happy, sad, excited, bored, …
The emotional, cognitive and/or behavioural connection that exists, at any point in time and over time, between a user and a technological resource
user interactions: click, read comment, recommend, buy, …
user mental states: concentrated, challenged, lost, interested …
S. Attfield, G. Kazai, M. Lalmas and B. Piwowarski. Towards a science of user engagement (Position Paper), WSDM Workshop on User Modelling for Web Applications, 2011.
- 5 -
Would a user engage with this web site?
http://www.nhm.ac.uk/
- 6 -
Would a user engage with this web site?
http://www.amazingthings.org/ (art event calendar)
- 7 -
Would a user engage with this web site?
http://www.lowpriceskates.com/ (e-commerce – skating)
- 8 -
Would a user engage with this web site?
http://chiptune.com/ (music repository)
- 9 -
Would a user engage with this web site?
http://www.theosbrinkagency.com/ (photographer)
- 10 -
Characteristics of user engagement (I)
• Users must be focused to be engaged • Distortions in the subjective perception of time used to
measure it Focused attention
• Emotions experienced by user are intrinsically motivating • Initial affective hook can induce a desire for exploration, active
discovery or participation Positive Affect
• Sensory, visual appeal of interface stimulates, promote focused attention
• Linked to design principles (e.g. symmetry, balance, saliency) Aesthetics
• People remember enjoyable, useful, engaging experiences and want to repeat them
• Reflected in e.g. the propensity of users to recommend an experience/a site/a product
Endurability
- 11 -
Characteristics of user engagement (II)
• Novelty, surprise, unfamiliarity and unexpected • Appeal to user curiosity, encourages inquisitive
behavior and promotes repeated engagement Novelty
• Richness captures the growth potential of an activity • Control captures the extent to which a person is able
to achieve this growth potential Richness and control
• Trust is a necessary condition for user engagement • Implicit contract among people and entities which is
more than technological
Reputation, trust and expectation
• Difficulties in setting up “laboratory” style experiments Motivation, interests,
incentives, and benefits
- 12 -
Forrester Research – The four I’s
• Presence of a user • Measured by e.g. number of visitors, time spent Involvement
• Action of a user • Measured by e.g. CTR, online transaction, uploaded
photos or videos Interaction
• Affection or aversion of a user • Measured by e.g. satisfaction rating, sentiment
analysis in blogs, comments, surveys, questionnaires Intimacy
• Likelihood a user advocates • Measured by e.g. forwarded content, invitation to join Influence
Measuring Engagement, Forrester Research, June 2008
- 13 -
Peterson et al Engagement measure - 8 indices Click Depth Index: page views Duration Index: time spent Recency Index: rate at which users return over time Loyalty Index: level of long-term interaction the user has with the site or product (frequency) Brand Index: apparent awareness of the user of the brand, site, or product (search terms) Feedback Index: qualitative information including propensity to solicit additional information or supply direct feedback Interaction Index: user interaction with site or product (click, upload, transaction)
Peterson etal. Measuring the immeasurable: visitor engagement, WebAnalyticsDemystified, September 2008
- 14 -
Measuring user engagement
Measures Characteristics
Self-reported engagement
Questionnaire, interview, report, product reaction cards
Subjective, user study (lab/online)
Mostly qualitative
Cognitive engagement
Task-based methods (time spent, follow-on task)
Neurological measures (e.g. EEG)
Physiological measures (e.g. eye tracking, mouse-tracking)
Objective, user study (lab/online)
Mostly quantitative
Scalability an issue?
Interaction engagement
Web analytics + “data science” (CTR, bounce rate, dwell time, etc)
Metrics and user models
Objective, data study
Quantitative Large scale
- 15 -
Interaction engagement – Online metrics Proxy of user engagement
- 16 -
Diagnostic and what we can do
Diagnostic: work exists, but fragmented. In particular: o What and how to measure depend on services and goals o Going beyond site engagement
What we have done: 1. Models of user engagement 2. Networked user engagement 3. Complex networks analysis
Future: Economic model for networked UE
- 17 -
Models of User Engagement Online sites differ concerning their engagement!
Games Users spend much time per visit
Search Users come frequently and do not stay long
Social media Users come frequently and stay long
Special Users come on average once
News Users come periodically
Service Users visit site, when needed
is it possible to model these differences and compare different classes of sites?
- 18 -
Data and Metrics
Interaction data, 2M users, July 2011, 80 USA sites Popularity #Users Number of distinct users
#Visits Number of visits
#Clicks Number of clicks
Activity ClickDepth Average number of page views per visit.
DwellTimeA Average time per visit
Loyalty ActiveDays Number of days a user visited the site
ReturnRate Number of times a user visited the site
DwellTimeL Average time a user spend on the site.
- 19 -
Diversity in User Engagement
Users and Loyalty Sites have different user groups
Proportion of user groups is site-dependent
Time and Popularity Site engagement can be periodic or
contains peaks
Engagement of a site depends on users and time
mail, social media
shopping, entertainment
media (special events)
daily activity, navigation
media, entertainment
- 20 -
Methodology
General models User-based models Time-based models Dimensions
8 metrics 5 user groups 8 metrics per user group
weekdays, weekend 8 metrics per time span
#Dimensions 8 40 16
Kernel k-means with Kendall tau rank correlation kernel
Num. of clusters based on eigenvalue distribution of kernel matrix Significant metric values with Kruskal-Wallis/Bonferonni
#Clusters (Models) 6 7 5
Analyzing cluster centroids = models
- 21 -
Models of user engagement (I)
• 6 general models
• Popularity, activity and loyalty are independent from each other
• Popularity and loyalty are influenced by external and internal factors
e.g. frequency of publishing new information, events, personal interests
• Activity depends on the structure of the site
Models based on engagement metrics
interest-specific
e-commerce, configuration
periodic media
- 22 -
Models of user engagement (II)
User-based [7 models] Models based on engagement per user
group
Time-based [5 models] Models based on engagement over
weekdays and weekend
Models based on engagement metrics, user and time
navigation game, sport hobbies, interest-specific
daily news
Sites of the same type (e.g. mainstream media) do not necessarily belong to the same model
The groups of models describe different aspects of engagement, i.e. they are independent from each other
- 23 -
Relationships between models
General User Time General 0.00 3.50 4.23 User 3.50 0.00 4.25 Time 4.23 4.25 0.00
Groups of models are independent from each other
Example: Model mu2
[high popularity and activity in all user groups, increasing loyalty]
50% to model mt2 [high popularity on weekends and high loyalty on weekdays]
50% to model mt3 [high activity and loyalty on weekends]
Variance of Information [0,5.61]
- 24 -
Recap & Next
User engagement is complex and standard metrics capture only a part of it
User engagement depends on users and time First step towards a taxonomy of models of user
engagement … and associated metrics
Next Interaction between models
User demographics, time of the day, geo-location, etc
J. Lehmann, M. Lalmas, E. Yom-Tov and G. Dupret. Models of User Engagement, UMAP 2012.
- 25 -
Understanding the problem: Users on Yahoo! network of sites
- 26 -
Networked user engagement: engagement across a network of sites
Large online providers (AOL, Google, Yahoo!, etc.) offer not one service (site), but a network of services (sites)
Each service is usually optimized individually, with some effort to direct users between them
Success of a service depends on itself, but also on how it is reached from other services (user traffic)
Measuring user engagement across a network of sites should account for user traffic
between sites.
- 27 -
Online multi-tasking
users spend more and more of their online session multi-tasking, e.g. emailing, reading news, searching for information ONLINE MULTI-TASKING navigating between sites, using browser tabs, bookmarks, etc seamless integration of social networks platforms into many services
leaving a site is not a “bad thing!”
- 28 -
Online multi-tasking
Users switch between sites within an online session (several sites are visited and the same site is visited several times)
Navigation
Back button
Link Other1
1995 35.7% 45.7% 18.6%
1997 31.7% 43.4% 24.9%
2006 14.3% 43.5% 42.2% Oberdorf et al
1) Usage of tabs, bookmarks, typing the URL directly,… 2) http://uxmovement.com/navigation/why-external-links-should-open-in-new-tabs/
Browser usage changed Less usage of back button
- 29 -
Online multi-tasking
Users switch between sites within an online session (several sites are visited and the same site is visited several times)
Navigation
Back button
Link Other1
1995 35.7% 45.7% 18.6%
1997 31.7% 43.4% 24.9%
2006 14.3% 43.5% 42.2%
Oberdorf et al]
Dubroy et al
Browser usage changed More and more usage of tabs
1) Usage of tabs, bookmarks, typing the URL directly,… 2) http://uxmovement.com/navigation/why-external-links-should-open-in-new-tabs/
UX Movement2: External links affect your site and users Links that take users to different websites should open in new tabs.
- 30 -
Online multi-tasking
Users switch between sites within an online session (several sites are visited and the same site is visited several times)
A short visit does not mean less engagement
Measuring user engagement across a network of sites should account for multi-tasking
Navigation
Back button
Link Other1
1995 35.7% 45.7% 18.6%
1997 31.7% 43.4% 24.9%
2006 14.3% 43.5% 42.2%
- 31 -
Networked user engagement - Two studies
o Is there a network effect? o study of 50 Yahoo! sites
o downstream engagement as a measure of networked user engagement
o effect of stylistics (layout and structure)
o Can we quantify the network effect? o study of 728 Yahoo! sites and traffic between them
o use metrics from the complex network area together with engagement metrics to characterize networked user engagement
- 32 -
Is there a network effect?
o Can we measure the network effect? o Downstream engagement
o Can we influence downstream engagement? o Session types
o Link types
The success of a web site largely depends on itself, but also on the network effect
This is particularly relevant for the Yahoo! network of properties
- 33 -
Measuring networked user engagement: Downstream engagement
User session
Yah
oo! s
ites
Downstream engagement for site A
(% remaining session time)
Site A
- 34 -
Downstream engagement
Varies significantly across sites
Exhibits different distributions according to site type
Is not highly correlated with other engagement measures such as dwell time
Optimizing downstream engagement will have little effect on user engagement within that site
0% 10% 20% 30% 40% 50% 60% 70%
0
0.02
0.04
0.06
0.08
0.1
0.12
1%
9%
17%
25
%
33%
41
%
49%
58
%
66%
74
%
82%
90
%
98%
0 20 40 60 80
100 120 140
0% 20% 40% 60% 80%
- 35 -
What causes engagement to change?
Web page style?
Web page content?
- 36 -
Methodology
Front pages of 50 popular Y! properties crawled every
hour
Sample user data from
toolbar data (19.4M
sessions, 265K users)
May 2011
Defining downstream engagement
measure
Measuring downstream engagement measure with
web page stylistics and
content
Influencing network
downstream engagement
with front page
stylistics
Studying the effect of links
on front pages on
downstream engagement
- 37 -
Data
Page attributes were defined for 50 popular (by page views) Yahoo! sites: Sampled the front page once every hour during the month of May 2011. Generate two types of attributes for each site at each time and date:
Stylistics (layout and structure) of a page General such as time of day, date, weekday or not
Downstream engagement values were measured using Yahoo! toolbar data: A total of 19.4M sessions recorded from approximately 265,000 users.
User and front page datasets joined by site, date and time, such that for each site and each date and time combination we have: average downstream engagement average dwell time vector of corresponding style attributes collected around the time that user
engagement was measured
- 38 -
Page stylistics provide good information to predict downstream engagement for many Yahoo! sites
Accuracy Precision Average DE site 1 site 2 site 3 site 4 site 5 site 6 site 7 site 8 site 9 site 10
0.80 0.76 0.72 0.71 0.65 0.63 0.63 0.63 0.61 0.60
0.54 0.52 0.43 0.40 0.42 0.34 0.38 0.44 0.34 0.31
0.07 0.11 0.21 0.18 0.20 0.19 0.26 0.14 0.18 0.13
The top-10 sites for which downstream engagement (DE) could be “accurately” predicted based on their stylistics
Downstream engagement of a number of sites of not particular types (models) could not be predicted from their stylistics.
- 39 -
Influential features
o Time of day
o Number of (non-image/non-video) links to Yahoo! sites in HTML body o Average rank of Yahoo! links on page o Number of (non-image/non-video) links to non-Yahoo! sites in HTML body
o Number of span tags (tags that allow adding style to content or manipulating content, e.g. JavaScript)
o Link placements and number of Yahoo! links can influence downstream engagement
o Not new, but here shown to hold also across sites
o Links to non-‐Yahoo! sites have a posi>ve effect on downstream engagement
o Possibly because when users are faced with abundance of outside links they decide to focus their aBen>on on a central content provider, rather than visi>ng mul>tude of external sites
- 40 -
Number of unique Yahoo! links (-) Number of (non-image/non-video) links
to Yahoo sites in the body of the HTML (+)
Number of (non-image/non-video) links to non-Yahoo! sites in the body of the HTML (+)
Number of video links within the page (+) Number of Java scripts on the page (-)
Three case studies
Here we look at three different Yahoo! sites, and the effect of their stylistics for downstream engagement
Time of day (+) Weekend (-) Number of unique Yahoo! links (-) Average rank of Yahoo! links on page (-) Number of paragraph tags (+)
Number of image links to non-Yahoo! sites in the body of the HTML (-)
Number of table elements (-) Average rank of Yahoo! links on page (-) Number of (non-image/non-video) links
to non-Yahoo! sites in the body of the HTML (+)
Time of day (+)
Sites Average downstream engagement
Accuracy Precision
e-commerce news women-interests
0.26 (+/- 0.31) 0.15 (+/- 0.02) 0.21 (+/- 0.06)
0.63 0.65 0.72
0.38 0.37 0.53
- 41 -
Influencing engagement through links
e-commerce news women- interests
Downstream engagement Same site Other Y! site Non Y! site
0.03 -0.09 -0.10
-0.31 0.20 0.04
-0.27 0.22 -0.25
Dwell time Same site Other Y! site Non Y! site
0.51 -0.61 -0.51
0.78 0.38 0.04
0.82 -0.68 0.80
The correlations between the number of various links and the values of downstream engagement and dwell time
• e-commerce: • links have little effect on downstream engagement, but have on dwell time
• news: • more news stories lead to more time spent on news • external links do not affect downstream engagement, but affect dwell time
• women-interests: • links to other Yahoo! sites can help increase engagement, but they may decrease dwell time
- 42 -
Users are more amenable to enhancing downstream engagement during certain sessions
Goal-specific sessions are those sessions where users have a specific goal in mind: do email, read news, check FB
Sessions when at least 50% of visited sites belonged to the five most common sites (for that user) were classified as goal-specific Goal-specific sessions accounted for 38% of sessions.
Approximately 92% of users had sessions of both kinds.
Average downstream engagement in goal-specific sessions was 0.16 vs. 0.2 for other sessions.
Accuracy of predicting downstream engagement was 0.76 for goal-specific sessions vs. 0.81 for other sessions.
When users do not have specific goals in mind, they may be more ready to accept suggestions (e.g. more links) for additional browsing
- 43 -
Further Work: Quantifying the Network Effect
Previously using one metric (downstream engagement), we showed that there is a network effect, and that the network effect can be influenced.
We go one step further and propose a methodology to account for the traffic between sites when measuring user engagement on a network of sites.
o Engagement networks
o Metrics from complex networks area (network-level and node-level)
o Application on 728 Yahoo! sites
- 44 -
Thank you
Thanks to many people at Yahoo! Labs
Questions?