PowerPoint Presentation
[B2G | Gaia | Firefox OS] Front-end Automation in Python (Marionette)
Stephen Donner
Web QA
March 20, 2013
Overview
Genesis / Introduction
Infrastructure (hardware)
Software stack
Builds in Jenkins / Continuous Integration
Test coverage / Capabilities
By the numbers
Will it blen^H^H run? (demo!)
Contribute!
Questions?
Genesis / Introduction
http://www.flickr.com/photos/daniel_gies/5041121388/sizes/z/in/photostream/
Initially, just a B2G QA thing
Genesis / Introduction
http://memegenerator.net/instance/27635541
Hard to keep good secrets for too long!
...so we expanded coverage + infra
Infrastructure (hardware)
7 Unagis, tethered via USB to 7 Mac Minis, running Ubuntu 12.04 LTS
Fun facts: Must have earbuds plugged in for radio tests to work
For the photo/video tests to work, lens/aperture must not be blocked or in the dark
Software Stack
Marionette (https://developer.mozilla.org/en-US/docs/Marionette) - testing server that sits in B2G Selenium-backed WebDriver
marionette_client Python abstraction layer for Marionette
Gaiatest (https://pypi.python.org/pypi/gaiatest) - wrapper around Python unnitests that runs the tests
Gaia UI Tests test framework comprising setUp, tearDown, API wrappers, and the tests themselves
Builds in Jenkins/Continuous Integration
Web QA helps set up, scale, and provide upkeep for more than just our UI tests
http://qa-selenium.mv.mozilla.com:8080/view/B2G/
By the Numbers
Started on November 11, 2012
25 individual contributors [1] From Taipei, Romania, UK, Canada, USA, Beijing
89 [2] tests, consisting of: ~ 19 unit tests
80 UI tests
~1hour per build, including initial flash
[1] https://github.com/mozilla/gaia-ui-tests/graphs[2] http://qa-selenium.mv.mozilla.com:8080/view/B2G/job/b2g.unagi.gaia.nightly.ui/lastBuild/
Test coverage / Capabilities
Sampling of coverage: Dialer (make/receive calls, test IMEI #)
Contacts (call, delete, create, add photo to, send SMS to)
Send/receive SMS
Camera (add, delete photo/videos)
Clock (add/delete alarms)
Calendar (add event, assert today's date)
Browser (load pages over EDGE, Wi-Fi, bookmark, navigate back)
Play a YouTube video
Marketplace (search, install, load, delete apps)
Everything.me (browse to and load/install an app)
Changing wallpaper
Radio (adding/remove station, dial frequency up/down)
Keyboard
Notification bar
and more...
Test coverage / Capabilities
Capabilities:
we push to device (media files photos, contacts, etc.)
we also capture the app HTML, take screenshots on failure for post-test debugging, and store the latest run's logcat (full, debug)
we test the default/empty statesin-between tests, we use the devicemanager API to iterate through and delete media files, delete all contacts, clear Wi-Fi networks, restore modified settings to default, set volume to 0
manifest.inigives us the ability to tell which tests to run, where
attributes:
antenna, b2g, wifi, lan, camera, xfail, panda, sdcard, qemu, bluetooth
Challenges
Two moving targets: B2G (Gaia/Gonk) + Marionette
Marionette-enabled builds are not identical to non-engineering buildsdifferent online/offline handling (patch pending)
changes to how/when adb / remote debugging work in Gaia
memory leak(s)
feature work
performance refactoring
Event-model changes:.click() .tap()
async JavaScript team's expertise is (still) largely Python
very dynamic, heavy HTML5 apps, with CSS3 usage rotates, transforms, SVG hard to find and test location and state of the app
infrastructure set upacquiring Unagi phones (and keeping them healthy)
SIMs (unlimited plans)
figuring out and maintaining a distributed, consistent format to store each phone's info
Will it blen^H^H^ (demo!)
https://www.youtube.com/watch?v=CC2ywYeP-4s
Repository: https://github.com/mozilla/gaia-ui-tests
Contribute!
How to contribute: https://github.com/mozilla/gaia-ui-tests/blob/master/CONTRIBUTING.md
Open issues up for grabs! https://github.com/mozilla/gaia-ui-tests/issues?direction=desc&labels=difficulty+beginner&sort=created&state=open
IRC: #mozwebqa (core Gaia UI Tests)
#ateam (perf tests)
Questions?
B2G
(Boot to Gecko)
Mozilla Marketplace