Agile and the automated testing of accessibility

Agile and the automated testing of accessibility

© 2015 - All Rights Reserved 2

© 2015 - All Rights Reserved 3

Agile Tools

5© 2015 - All Rights Reserved

Shift Left Testing

Catching a defect in the design and requirements phase can reduce cost to fix by up

to 100:1

Software Defect Reduction Top 10 List, Barry Boehm and Victor R. Basili, January 2001

https://www.cs.umd.edu/projects/SoftEng/ESEG/papers/82.78.pdf

© 2015 - All Rights Reserved 6

Rise of Continuous

• Lag between committing code and getting feedback allows damage to magnify– Costs rise– Velocity decreases

• Continuous Integration• Continuous Deployment

http://www.startuplessonslearned.com/2009/06/why-continuous-deployment.html

© 2015 - All Rights Reserved 7

Automation

© 2015 - All Rights Reserved 8

© 2015 - All Rights Reserved 9

Accessibility Testing’s Image

Aim of Software Testing

• Reduce the risk of the existence of a serious problem on a supported configuration to an acceptable level– Define “serious problem”– Define “supported configuration”– Define “acceptable level”

© 2015 - All Rights Reserved 10

© 2015 - All Rights Reserved 11

Android Fragmentation

12© 2015 - All Rights Reserved

Android Fragmentation

We have seen 24,093 distinct devices download our app in the past few months. In our report

last year we saw 18,796. In 2013 we saw 11,868

OpenSignal 2015

© 2015 - All Rights Reserved 13

Combinations

4 operating systems, 2 supported versions5 Assistive technologies4 Browsers, 2 supported versions3 Responsive breakpoints

216 combinations excluding device combinations

© 2015 - All Rights Reserved 14

© 2015 - All Rights Reserved 15

Sparse Testing Matrix

Browser/AT

Chrome KB IE KB

Firefox KB Safari KB Dragon IE NVDA FF Jaws IE VO Safari TB FF Orca FF

Operating

Sytem

Windows

OS X

iOS

Android

Linux

© 2015 - All Rights Reserved 16

© 2015 - All Rights Reserved 17

Accessibility “Acceptance Automation”

18© 2015 - All Rights Reserved

Acceptance Automation

• Comes from the “requirements”– Designers mark up their wireframes and comps– Stories must contain specific accessibility

acceptance criteria• Can go to any level of detail– Test the specific alt attributes on images– Test the DOM order specified in the wireframe– Etc.

© 2015 - All Rights Reserved 19

Open Source Rules Engine

© 2015 - All Rights Reserved 20

aXe Manifesto

• Must have a zero false positive rate• Must be lightweight and fast• Must work in all modern browsers on all

platforms• Must be tested automatically

© 2015 - All Rights Reserved 21

aXe-Core

• Small ~100KB• Fast

– No network connection, no new browser start• Intelligent

– Supports cross domain iframes– Turns document level rules on/off automatically

• Unlimited– No throttling, no server limits

• Reliable– No double-execution or missing CSS problems– No network connectivity problems– No server capacity problems

• Secure– No server access across the wire– No sending data to a server

• Mozilla Public License 2.0– Available via GitHub, npm and Bower

© 2015 - All Rights Reserved 22

aXe Chrome extension

© 2015 - All Rights Reserved 23

aXe Firefox extension

© 2015 - All Rights Reserved 24

aXe extension

• Developers– Use when coding

• Testers– Use during functional testing

• Accessibility Experts– Use for validation

© 2015 - All Rights Reserved 25

aXe API

• Two major calls– axe.getRules() – returns a list of the rules loaded– axe.a11yCheck() – analyze a page or portion of a page

• Getting started– Install package

• bower install –-save-dev axe-core

– Include library• <script src=“(path_to)/axe.js”></script>

– Call function• a11yCheck(target, options, callback);

© 2015 - All Rights Reserved 26

aXe API demo

© 2015 - All Rights Reserved 27

Integrations

• Java Selenium• JavaScript Selenium (axe-webdriverjs)• Grunt (grunt-axe-webdriver)• Cucumber and R-Spec (axe-matchers)• Firefox Dev Tools• Chrome Dev Tools• Ember-aXe (ember-axe)

more…

© 2015 - All Rights Reserved 28

Summary

• Manage your matrix– Optimize for resources and risk

• Leverage automation– Generic automation– Specific automation– Optimize for up-front vs. ongoing effort

© 2015 - All Rights Reserved 29

Questions

• https://addons.mozilla.org/firefox/addon/axe-devtools/

• Chrome: search Google for “chrome axe extension”

© 2015 - All Rights Reserved 30