Embed Size (px)
DESCRIPTION
Lessons learned about QA can manage the rising Android device fragmentation
Citation preview
www.immobilienscout24.dewww.immobilienscout24.de
Android Meetup Berlin | 29.08.2012 | Jürgen Schmidt
Android FragmentationQA lessons learned from the fragmentation hell
www.immobilienscout24.de
About me
Jürgen Schmidt
Test Engineer
Development Lifecycle Expert
Database addicted
| Android Fragmentation | Jürgen Schmidt Page 3
Fragmentation
QA must consider different levels of fragmentation
• OEMs (>27)• OS levels (16)• Display size/resolution (>100)• Further hardware features
Devices (> 400)
| Android Fragmentation | Jürgen Schmidt Page 4
Strategy
• With an effort of 20% cover 80% of all devices to test
• Source of usage statistics:• App statistics on publisher
site on Google Play Store• Reporting Frameworks like
Google Analytics
• Don‘t forget potential new devices or features
20% effort
80% coverage
Follow the famous 80/20 rule
ImmoScout App Facts…
Top30 ~80%
49%
17%8%
3%
2%1%
20%
OEM
Samsung SonyEricsson HTCLG Motorola GoogleSonstige
| Android Fragmentation | Jürgen Schmidt Page 5
Which devices are currentlyused by our users?
Find a minimum set among top devices which cover all top 80% device features ~ 92%
56%
12%
9%
8%4%
3%
3% 3% 2%2%
Display Resolution
480x800 720x1280 320x480 480x854540x960 720x1184 800x1232 800x1280240x320 Other
| Android Fragmentation | Jürgen Schmidt Page 6
Selected top devices for manual test
Wildfire S Galaxy S 2 Galaxy S 3 Nexus 7
OEM HTC Samsung Samsung Asus Google
OS level 2.2 (8) 2.3.5 (10) 4.0.3 (15) 4.1 (16)
Display Res. 320x480 480x800 720x1280 800x1280
Density ldpi mdpi hdpi mdpi
| Android Fragmentation | Jürgen Schmidt Page 7
Test automation
We consider OS level fragmentation for running unit tests and robotium system tests on virtual devices
Jenkins (CI server) with Android plugin offers multi-configuration jobs for executing tests on different virtual devices at the same time
| Android Fragmentation | Jürgen Schmidt Page 8
Further tests to help finding bugs caused by fragmentation
Session Based Tests• Unstructured exploratory test
driven by test and domain experts• Use complete device pool
Crowd-sourced Testing• Bug hunting by real users
under real conditions• Many testers with a lot of
various devices
| Android Fragmentation | Jürgen Schmidt Page 9
Monitoring a (pre-)published app
Many real-time bug reporting frameworks can be integrated into the app
Help in evaluation of reported exceptions or crashes by grouping and annotating of device features
Quality risks caused by fragmentation are managable
Experience show: Most found critical bugs are not specific for certain devices, OS level etc.
Focus only on top and new features instead reaching full coverage
Conclusion
| Android Fragmentation | Jürgen Schmidt Page 10
Limit your efforts in testing fragmentation!
Android fragmentation visualized by OpenSignalMaps:http://opensignalmaps.com/reports/fragmentation.php
TechCrunch article to show how several game developers manage fragmentationhttp://techcrunch.com/2012/06/02/android-qa-testing-quality-assurance/
About Session Based Testing by Satisfice Inc.:http://www.satisfice.com/sbtm/
Blog post about Crowd-sourced testing by Mithun Sridharanhttp://it.toolbox.com/blogs/mithuns-memoirs/crowd-sourced-software-testing-52763
Want to read more about fragmentation?
| Android Fragmentation | Jürgen Schmidt Page 11
www.immobilienscout24.de
Contact:ImmobilienScout24 Andreasstraße 1010243 Berlin
Thanks for your attention!
Fon: 030/24301-11 00Email: [email protected]: www.immobilienscout24.de
