Embed Size (px)
Citation preview
Test%be'er%%using%mobile%analy2cs
So#test'2016!JULIAN HARTY 13th Oct 2016
Creative Commons License This presentation by Julian Harty is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License. Materials, logs, etc. remain the copyright of their respective owners.
http://creativecommons.org/licenses/by-sa/3.0/deed.en_US
Rev: 15 Jun 2016
Our%Tes2ng%of%Mobile%Apps?
2
https://mobiletestingblog.com/ http://devicelab.fi/
vs.%Real%Use
3
http://www.aim.org/wp-content/uploads/2013/09/china-cell-phone-users.jpg http://cdn2.business2community.com/wp-content/uploads/2014/01/hellaphones1.png http://www.business2community.com/mobile-apps/25-facts-show-mobile-become-biggest-media-advertising-channel-world-0744628#s1I8LIDzm0lKC47k.97 http://www.theatlantic.com/technology/archive/2010/11/cell-phone-users-through-history/66363/ https://newsatjama.jama.com/2013/03/15/mobile-device-users-keep-your-eyes-on-the-road-your-hands-upon-the-wheel/
http://devicelab.fi/ https://mobiletestingblog.com/ http://www.top-news.top/news-12315647.html
Opera2onal%Profiles
• Simplified!representa/ons!of!how!the!so4ware!is!used!
4
Visible%+%Invisible
5 https://www.flickr.com/photos/shashachu/443215138/in/photostream/ https://commons.wikimedia.org/wiki/File:01_16_X-ray_of_Hand.jpg
Known%+%Unknowns%in%SoHware
Visible' Invisible' Of…'
Changes!
Complexity!
Crashes!
Devices!
Emo/ons!
Feelings!
Flaws!
Se?ngs!
Timings!
Usage!
Users!
Heuris2cs%Test%Oracles 7
R+1!
R+0!
RF1!
RF2!
Our!!Releases!C1!C2! C3! C4!
Similar apps
• Visual changes
• Timings
• Comparisons
Effects%of%differences
• Ba8ery'drain'varied'by'3x'for!similar!hardware!specifica/ons!• Custom!code!added!for!Kindle'Fire'to!reduce!brightness!!
• 40%'less'ba8ery'drain'
• Higher!network!latencies!reduced!interac/vity!by!40%!• Users!preferred!WiFFi!
• 69%!for!Parallel!Kingdom,!58%!for!StudyBlue!
• Tablets!2x!usage!!• PullFout!keyboard!also!increased!usage!
4.4%%N>%%4.3%%=%?
9
Understand%the%effects
Cause%&%Effects
10
Tippen Sie auf die Farbe, die Sie bevorzugen, das Spiel fortzusetzen.
Appuyez sur la couleur que vous préférez pour continuer le jeu.
ඔබ #$ඩාව (ගටම කැම. ව/ණය මත ත3ටු කර6න.
Tap on the colour you prefer to continue the game.
Digital!• So4ware!generated!• Consistent!• Ubiquitous!• Dry!
• Machine!oriented!
Analogue!• People!generated!• Variable!• Sparse!• Rich!• Emo/onal!• Human!oriented!
11
Feedback
An%example%of%App%Store%Ra2ngs
https://play.google.com/store/apps/details?id=air.com.inxpo.xpocast&hl=en https://itunes.apple.com/us/app/xpocast/id538003615?mt=8
An%example%of%App%Store%Reviews
https://play.google.com/store/apps/details?id=air.com.inxpo.xpocast&hl=en https://itunes.apple.com/us/app/xpocast/id538003615?mt=8
Worth%knowing%about%reviews
14
Reviews affect the app ranking and the app store conversion. They are also critical for Search Engine Optimization.
Influence
Turn users’ input into innovation
Improve
Reviews have multiple dimensions which can be used to help measure the user experience.
Measure and Engage
Monitoring
Testing
Reviews
Know'your'users'
Parallel!Kingdom[2]!
Regular!users!generate!2.5x'daily'revenues'
Custom!drink!feature!removed[1]!!=>!1'star'feedback'ra/ngs!
Logos!©!respec/ve!owners![1]!example!from!App!Quality!book![2]!example!from!Tale!of!Two!Apps!
A%banking%example
16
https://itunes.apple.com/us/app/m-t-mobile-banking/id397761931?mt=8
Kindle%App%for%iOS
17
Kindle%App%for%iOS
18
06 Apr 2016 03 May 2016
Google%Play%Developer%Console
Use%Analy2cs%&%Heatmaps
20
Layers%of%an%App
Mobile%Analy2cs HPE!AppPulse!Mobile!
23 http://www.openstreetmap.org/#map=5/50.261/0.483
Where the testers were
Where the users were
Crash%analy2cs 24
PreNlaunch%automated%tes2ng 25
Heatmaps
Problema/c!UI! Heatmap!
Source:!heps://www.hotjar.com/heatmaps!
“8!Heatmap!Tests!that!reveal!visitor!behavior.”!
Images source: Appsee.com
Feedback%Cycles
TOPOLOGY!
• No!longer!limited!to!what!“we”!think!“they”!need/do/etc.!Discover!how!the!app!behaves!across!virtually!all!the!popula/on!
• Lower!cost!of!opera/ons!
• Real!usage!can!help!drive!our!tes/ng!and!analysis;!brings!realism!to!our!tes/ng!
Some%benefits%of%analy2cs%data
Tes2ng%based%on%analy2cs%data
TesIng'based'on'informaIon'• Popularity!&!volumes!
• Locales!• UserFflows,!ac/vi/es,!etc.!• Crashes!Aims!include!
• Fast!reproduc/on!• 1stFhand!learning!in!controlled!env.!
TesIng'based'on'insights'• Delta’s!(rate!of!change)!• Crossing!thresholds!• Anomalies!
Aims!include:!
• Maximising!insights,!agility,!and!ability!to!adapt!&!respond!
Analy/cs!for!So4ware!Development!Engineering!Ac/vity,!Benchmarking,!Tes/ng!
Trends,!Defect!Reports!! Extrapola/on!
So4ware!quality!models,!!
boeleneck!analysis!
Specifica/on!refinement,!
asset!realloca/on!
Failure!predic/on!models!
What’s'happened?'(Reporting)'
What’s'happening'now?'(Alerts)'
What!will!happen?!(Forecas/ng)!
How!and!why!did!it!happen?!!
(Factor!analysis)!
What!is!the!next!best!ac/on?!
(Recommenda/on)!
What’s'the'best/worst'that'can'happen?'
(Modeling'/'Simulation)'
InformaIon'
Insight'
Past' Present' Future'
hep://research.microso4.com/pubs/136974/foserF2010Fbuse.pdf!
How%many%devices%are%enough? Perennial!Ques/on!
Devices%Matrices
Android Images used with permission from OpenSignal.com
https://iossupportmatrix.com/
Enough%for%80%%of%reviews “Priori/zing!the!devices!to!test!your!app!on!:!A!case!study!of!android!game!apps”!
<=!Free!!Paid=>!
Enough%for%80%%of%…
• Bugs?!
• Usage?!
• Locales!&!Languages?!
• Network!Condi/ons?!
• …?!
35
Gaining%confidence Fools!rush!in…!
36
Precision!&!accuracy!
Precision:!repeatability! Accuracy:!on!target!!heps://en.wikipedia.org/wiki/Accuracy_and_precision!
Precision!&!accuracy!
• Add!images!here:!precision!
• And!here:!accuracy!
A%system%at%rest?
� A!tale!of!two!three!mobile!analy/cs!libraries!� (and!what!happens!when!bonuses!are!on!the!line…)!
� Who!is!Responsible!(part!1)?!
Divergent!answers!increase!doubt!
Necessary%but%not%sufficient Listening!is!a!means!to!an!end!
41
CorrelaIon?'CausaIon?'Appropriate?'
From:!User!Interac/onFbased!Profiling!System!for!Android!Applica/on!Tuning!Figure!©!ACM!!
Test%Recommenda2on%Engine
The%Dark%Side Of!using!Mobile!Analy/cs!
44
Do!no!harm!
Who is Responsible (Part 2)? hep://www.vam.ac.uk/users/node/1777!A!Study!of!ThirdFParty!Tracking!by!Mobile!Apps!in!the!Wild!4p://4p.cs.washington.edu/tr/2012/03/UWFCSEF12F03F01.PDF!https://sourcedna.com/blog/20151018/ios-apps-using-private-apis.html
Palo Alto Networks WildFire has captured over 18,000 Android apps that contain this library. (October 21, 2015)
http://researchcenter.paloaltonetworks.com/2015/10/chinese-taomike-monetization-library-steals-sms-messages/
Beware'the'automaIon'bias'
Automa/on!Bias!in!Intelligent!Time!Cri/cal!Decision!Support!Systems!hep://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.91.2634&rep=rep1&type=pdf!
“When!presented!with!an!automated!solu/on!40%!of!pilots!reasoned!less!or!none!at!all”!
Things%that%might%help Us!to!improve!UX!
47
Assess%&%Respond%to%Feedback
48
2 feedback
96% of companies don’t respond to feedback on App Stores Two-Star feedback is the most serious and actionable.
Khalid, H., Shihab, E., Nagappan, M., & Hassan, A. E. (2015). What do mobile app users complain about?. Software, IEEE, 32(3), 70-77.
49
http://uk.businessinsider.com/facebook-2g-tuesdays-to-slow-employee-internet-speeds-down-2015-10
Use%similar%devices%to%your%users
50
wired.com/2015/10/facebook-workers-ditch-iphones-in-push-for-world-conquest/
READ THIS BOOK !
Questions now? Questions later… [email protected]
http://themobileanalyticsplaybook.com/
Further%reading%and%research
The!opensource!project!hep://code.google.com/p/webFaccessibilityFtes/ng!heps://github.com/julianharty/webFaccessibilityFtes/ng/!!
Finding!Usability!Bugs!with!Automated!Tests!hep://queue.acm.org/detail.cfm?id=1925091!!Figh/ng!Layout!Bugs!!hep://code.google.com/p/figh/ngFlayoutFbugs/!Experiences!Using!Sta/c!Analysis!to!Find!Bugs!hep://www.google.com/research/pubs/pub34339.html!!My!blog!hep://blog.beeerso4waretes/ng.com/!!“Beware!of!Automa/on!Bias”!by!M.L.!Cummings!hep://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.91.2634&rep=rep1&type=pdf!!!Designing!and!Engineering!Time!by!Steven!Stow!ISBN!978F0F321F50918F5!!!
52
