Capturing Mobile Experience in the Wild: A Tale of Two Apps
Ashish Patro*
Shravan Rayanchu, Michael Griepentrog
Yadi Ma, Suman Banerjee University of Wisconsin Madison
Deploying a mobile applicaDon…
Ashish Patro / Insight / CoNext 2013 2
Mobile app stores Developers App Users
Internet
ApplicaDon Server
What is the “ApplicaDon Experience”?
Ashish Patro / Insight / CoNext 2013 3
What factors are impacDng the users’ experience?
What factors are impacDng my applicaDon revenues?
What is the baWery drain of my applicaDon across different devices?
..........
Developers
Challenge for developers…
Ashish Patro / Insight / CoNext 2013 4
Developers App Users
Example 1: Diverse Devices
Ashish Patro / Insight / CoNext 2013 5
Developers App Users
ApplicaDon Server
Type (tablet, phone) and pla_orm
Screen type/area, OS, features (keyboard), baWery
capacity, device age
Example 2: User diversity
Ashish Patro / Insight / CoNext 2013 6
Developers App Users
ApplicaDon Server
LocaDon, Dme of day, New vs. old users
Network quality (signal, congesDon), session duraDons, revenues
Example 3: Diverse Networks
Ashish Patro / Insight / CoNext 2013 7
Developers App Users
ApplicaDon Server
802.11, HSDPA, EVDO, LTE Latency, throughput
Goal: Capture “applicaDon experience”
Ashish Patro / Insight / CoNext 2013
Networks diversity
8
Device diversity
User diversity
How can we enable developers to capture the “applicaDon experience”?
Understanding “applicaDon experience”
Ashish Patro / Insight / CoNext 2013
Device + UI Design (e.g., screen size)
Network performance (e.g., latency)
ApplicaDon Overhead (e.g., BaWery, CPU
etc.)
ApplicaDon Experience
User Engagement (e.g., Session length,
interacDvity, retenDon)
Developer Revenues (e.g., virtual currency
usage)
9
User types (e.g., old vs.
new)
Developers
Understanding “applicaDon experience”
Ashish Patro / Insight / CoNext 2013
Device + UI Design (e.g., screen size)
Network performance (e.g., latency)
ApplicaDon Overhead (e.g., BaWery, CPU
etc.)
ApplicaDon Experience
User Engagement (e.g., Session length,
interacDvity, retenDon)
Developer Revenues (e.g., virtual currency
usage)
10
User types (e.g., old vs.
new)
How can we capture these metrics across all users?
Developers
SoluDon: Embedded measurements • Developed a measurement toolkit: “Insight” • Using the applicaDon as a vantage point
Ashish Patro / Insight / CoNext 2013 11
Insight
In this talk…
• Study with 2 popular applicaDons – MMORPG game: > 1 million users (over 3 years)
– Study applicaDon: > 160,000 users (over 1 year)
Ashish Patro / Insight / CoNext 2013 12
• Insight: Our mobile applicaDon analyDcs toolkit
Outline
• Insight overview and deployment
• Understanding applicaDon usage
• Impact of network performance
• Related work and summary
Ashish Patro / Insight / CoNext 2013 13
Insight measurement toolkit
Mobile App
App Code
Insight threads
App Servers
Ashish Patro / Insight / CoNext 2013 14
Insight servers
Insight toolkit advantages
Mobile App
App Code
Insight threads
Ashish Patro / Insight / CoNext 2013 15
Light-‐weight library code with easy to use API
Contextual measurements: When desired app is running
No addiDonal overhead for deployment of framework
Insight measurements
Ashish Patro / Insight / CoNext 2013
Mobile App + Insight
16
Insight measurements
Ashish Patro / Insight / CoNext 2013
Mobile App + Insight
In-‐app acDviDes
User acDvity
17
Session duraDons
ApplicaDon specific events (e.g. in-‐app purchases, fighDng monsters)
Insight measurements
Ashish Patro / Insight / CoNext 2013
Mobile App + Insight Device
Device info
18
Vendor, screen size, Pla_orm, OS version
Insight measurements
Ashish Patro / Insight / CoNext 2013
Mobile App + Insight
19
Status Info (LocaDon, BaWery, CPU, Network)
Status updates
LocaDon: Country, State
ApplicaDon Overhead: BaWery + CPU + Memory usage
Network status: Signal, type (WiFi, HSDPA,
EVDO etc.)
Insight measurements
Ashish Patro / Insight / CoNext 2013
Mobile App + Insight Insight server
20
AcDve network Measurements
ApplicaDon level latency: Round Trip Time (RTT) measurements to our server
Insight measurements
Ashish Patro / Insight / CoNext 2013
Mobile App + Insight Insight server
In-‐app acDviDes (e.g., in app-‐purchases)
User acDvity
Device InformaDon
Device Info
21
Status Info (LocaDon, BaWery, CPU, Network)
AcDve network measurements
Status updates
In-‐situ applicaDon vantage point captures a rich set of metrics
Deployed applicaDons (1)
MMORPG Game: Parallel Kingdom (PK) • Analyzing both iOS and Android users • AcDons: Spend food, trade items, aWack players/monsters
• Deployed more than 3 years • > 1 million players • > 61 million sessions
Ashish Patro / Insight / CoNext 2013 22
Deployed applicaDons (2)
Ashish Patro / Insight / CoNext 2013
Study Tool: StudyBlue (SB) • Analyzed Android users only • AcDons: Study, Create flashcards,
quizzes • Deployed more than 1 year • > 160,000 users tracked • > 1.1 million sessions
23
0
30000
60000
90000
120000
150000
0 200 400 600 800 1000 1200 1400
Tota
l sess
ions
per
day
Days Elapsed (PK)
Age 2Age 3
Age 4
0
3000
6000
9000
12000
0 50 100 150 200 250
Tota
l sess
ions
per
day
Days elapsed (SB)
SchoolStart Break
Unique sessions per day
ApplicaDon usage for PK spikes with new updates while it is highly correlated with Dme of week and year for SB
Ashish Patro / Insight / CoNext 2013 24
Outline
• Insight overview and deployment
• Understanding applicaKon usage
• Impact of network performance
• Related work and summary
Ashish Patro / Insight / CoNext 2013 25
Understanding “ApplicaDon Experience”
Ashish Patro / Insight / CoNext 2013
Device + UI Design (e.g., screen size)
Resource ConsumpDon (e.g., BaWery, CPU etc.)
26
User types (e.g., old vs.
new)
0
0.2
0.4
0.6
0.8
1
iPad iPhone DroidXtreme
MyTouch3G
Droid2
HTCVision
HTCEVO
SGHCaptivate
HTCHero
SGHMoment
No
rm.
Act
ion
s/se
c
PK: Device Model
Screen9.7" 3.5" 4.1" 3.4" 3.7" 3.7" 4.3" 4" 3.2" 3.2"
Slide out keyboard
size (in.):
Impact of device on user interacDvity
48% lower
Ashish Patro / Insight / CoNext 2013
0
0.2
0.4
0.6
0.8
1
KindleFire
Nexus7
GalaxyS2
HTCEVO
GalaxyNexus
GalaxyS
HTCThunderbolt
DesireHD
No
rm.
Act
ion
s/se
c
SB: Device Model
Tablets
56% lower
27
User interacDvity = User acDons / Dme User interacDvity varied by 2x based on device types and
hardware features (screen size and slide out keyboards).
Understanding “ApplicaDon Experience”
Ashish Patro / Insight / CoNext 2013
Device + UI Design (e.g., screen size)
Resource ConsumpDon (e.g., BaWery, CPU etc.)
28
User types (e.g., old vs.
new)
Device form factor and pla_orm impacted interacDvity (upto 2x).
BaWery consumpDon in the wild (PK)
3x variaDon
Ashish Patro / Insight / CoNext 2013 29
High variaDon in applicaDon footprint (baWery usage) due to diversity of devices
BaWery drain = % usage/ Dme
BaWery drain vs. session duraDon (Evo 4G)
Ashish Patro / Insight / CoNext 2013 30
Sessions with high baWery drain exhibited upto 2x Dmes lower duraDons
Increase in baWery drain
50% lower
Conserving baWery consumpDon…
Ashish Patro / Insight / CoNext 2013
0
0.02
0.04
0.06
0.08
0.1
0.12
0 0.2 0.4 0.6 0.8 1
Pro
babili
ty D
istr
ibutio
n
HTC Evo 4G - Normalized battery drain rate (levels/min)
High (255)Moderate (102)
0
0.1
0.2
0.3
0.4
0.5
0 0.2 0.4 0.6 0.8 1
Pro
babili
ty D
istr
ibutio
nKindle Fire - Normalized
battery drain rate (levels/min)
High (255)Moderate (175)
Mode 36% lower
31
Device specific opDmizaDons (e.g., screen brightness, GPS frequency) can help, but variable gains per device
Low Impact!
Understanding “ApplicaDon Experience”
Ashish Patro / Insight / CoNext 2013
Device + UI Design (e.g., screen size)
Resource ConsumpDon (e.g., BaWery, CPU etc.)
32
User types (e.g., old vs.
new)
Device form factor and pla_orm impacted interacDvity (upto 2x).
High variability in baWery overhead (3x) Gains from device opDmizaDons can vary
Impact of retaining users on revenue (PK)
4x variaDon
Ashish Patro / Insight / CoNext 2013 33
Over a period of 7 months (Age 2 of
the game)
Old users tend to spend more money daily (4x more than new users)
New users Old users
Understanding “ApplicaDon Experience”
Ashish Patro / Insight / CoNext 2013
Device + UI Design (e.g., screen size)
Resource ConsumpDon (e.g., BaWery, CPU etc.)
34
User types (e.g., old vs.
new)
Device form factor and pla_orm impacted interacDvity (upto 2x)
High variability in baWery overhead (3x) Gains from device opDmizaDons can vary
User retenDon is important: Upto (4x) more daily revenues per user
Outline
• Insight overview and deployment
• Understanding applicaDon usage
• Impact of network performance
• Related work and summary
Ashish Patro / Insight / CoNext 2013 35
Impact of network performance
Ashish Patro / Insight / CoNext 2013
Mobile App + Insight
36
Network latency measurements
User InteracDvity
Insight server
Network type usage
Developer revenues
Impact of network performance
Ashish Patro / Insight / CoNext 2013
Mobile App + Insight
37
Network latency measurements
User InteracDvity
Insight server
Network type usage
Developer revenues
Impact of latencies on user interacDvity
Ashish Patro / Insight / CoNext 2013
0
0.2
0.4
0.6
<=0.3 0.4 0.6 0.8 >=1
Cel
lula
r usa
ge ra
tio(m
ax. o
f 1)
Avg. cellular RTT (in seconds)
PKSB
0 0.2 0.4 0.6 0.8
1
<= 0.3 0.6 0.9 1.2 >= 1.4
Nor
m. a
ctio
ns/ti
me
Avg. RTT of session (in seconds)
PKSB
40% lower
38
40% drop in user interacDvity for MMORPG at high latencies (900ms). Lower impact on study applicaDon.
Higher network latencies
Impact of network performance
Ashish Patro / Insight / CoNext 2013
Mobile App + Insight
39
Network latency measurements
User InteracDvity
Insight server
Higher drop in interacDvity for MMORPG (40%) at poor latencies
Network type usage
Developer revenues
Impact of latencies on cellular usage
Ashish Patro / Insight / CoNext 2013
0
0.2
0.4
0.6
<=0.3 0.4 0.6 0.8 >=1
Cel
lula
r usa
ge ra
tio(m
ax. o
f 1)
Avg. cellular RTT (in seconds)
PKSB
0 0.2 0.4 0.6 0.8
1
<= 0.3 0.6 0.9 1.2 >= 1.4
Nor
m. a
ctio
ns/ti
me
Avg. RTT of session (in seconds)
PKSB
42% lower cellular usage
40
High cellular latencies correlated with higher preference for WiFi networks for both apps
Higher average cellular latencies
Impact of network performance
Ashish Patro / Insight / CoNext 2013
Mobile App + Insight
41
Network latency measurements
User InteracDvity
Insight server
Higher drop in interacDvity for MMORPG (40%) at poor latencies
Greater user preference for WiFi networks at high cellular latencies across both apps
Network type usage
Developer revenues
PK: Impact of latencies on revenues
Ashish Patro / Insight / CoNext 2013
52% lower
revenue
42
High network latencies caused upto a 50% drop in developer revenues
Latency vs. Revenue / Dme
Impact of network performance
Ashish Patro / Insight / CoNext 2013
Mobile App + Insight
43
Network latency measurements
User InteracDvity
Insight server
Higher drop in interacDvity for MMORPG (40%) at poor latencies
Greater user preference for WiFi networks at high cellular latencies.
Upto 50% drop in developer revenues (MMORPG) due to poor network performance
Network type usage
Developer revenues
Outline
• Insight overview and deployment
• Understanding applicaDon usage
• Impact of network performance
• Related work and summary
Ashish Patro / Insight / CoNext 2013 44
Related work
• Commercial Tools: Android/iOS, Flurry • Understanding network performance (MobiSys’10) – 3G Test: Standalone mobile app. to measure cellular and WiFi performance
• Smartphone usage studies (MobiSys’11) – Falaki et al. : ApplicaDon, network and device usage study across 255 users
• Mobile applicaDon usage characterisDcs (IMC’11)
Ashish Patro / Insight / CoNext 2013 45
Summary • Presented “Insight”, a measurement framework for mobile applicaDons – >3 year deployment on 2 popular apps
• High variance (3x) in resource usage across devices – Device specific opDmizaDons can help (gains vary)
• User retenDon is criDcal for applicaDons revenue generaDon and engagement: 4x variability
• High latencies lowered both revenues (52%) and user-‐engagement (40%) (more for MMORPG)
Ashish Patro / Insight / CoNext 2013 46
Ashish Patro / Insight / CoNext 2013 47
Thanks!