Using Acceptance Tests to Validate
Accessibility Requirements in RIA
Willian Massami WatanabeRenata Pontin de Mattos Fortes
Ana Luiza Dias
9th International Cross-Disciplinary Conference on Web Accessibility – 16/17th April 2012 – Lyon, France.
Agenda
•Research context
•State of art
•Hypothesis
•Acceptance Tests
•Preliminary results
•Advantages and limitations
•Next steps
2/35
Agenda
•Research context
•State of art
•Hypothesis
•Acceptance Tests
•Preliminary results
•Advantages and limitations
•Next steps
2/35
Research Context
3/35
Research Context
4/35
Web 2.0
Perpetual Perpetual betabeta
Perpetual Perpetual betabeta
Research Context
4/35
Users
Research Context
Perpetual Perpetual betabeta
Perpetual Perpetual betabeta
4/35
Users
• New features• Web applications
improvements
• New features• Web applications
improvements
Research Context
Perpetual Perpetual betabeta
Perpetual Perpetual betabeta
4/35
Less than an hourLess than an hour
Research Context
Perpetual Perpetual betabeta
Perpetual Perpetual betabeta
5/35
RequirementsRequirements
DesignDesign
ImplementationImplementation
TestsTests
DeploymentDeployment
Less than an hourLess than an hour
Research Context
Perpetual Perpetual betabeta
Perpetual Perpetual betabeta
5/35
Continous Integration
Continuouslly compile and test the software to ensure quality
[Kim et al. 2008]
Research Context
5/35
Continuous Integration
Developers
Research Context
6/35
Continuous Integration
Developers
Source code repository
Research Context
6/35
Developers
Source code repository
• Unit testsUnit tests
• Integration Integration
teststests
• CompileCompile
• Acceptance Acceptance
teststests
• Unit testsUnit tests
• Integration Integration
teststests
• CompileCompile
• Acceptance Acceptance
teststests
Report
Research Context
6/35
Continuous Integration
Developers
Source code repository
• Unit testsUnit tests
• Integration Integration
teststests
• CompileCompile
• Acceptance Acceptance
teststests
• Unit testsUnit tests
• Integration Integration
teststests
• CompileCompile
• Acceptance Acceptance
teststests
Report
Research Context
6/35
Continuous Integration
Developers
Source code repository
• Unit testsUnit tests
• Integration Integration
teststests
• CompileCompile
• Acceptance Acceptance
teststests
• Unit testsUnit tests
• Integration Integration
teststests
• CompileCompile
• Acceptance Acceptance
teststests
Report
Research Context
6/35
Continuous Integration
RequirementsRequirements
DesignDesign
ImplementationImplementation
TestsTests
DeploymentDeploymentQuality
Perpetual Perpetual betabeta
Perpetual Perpetual betabeta
Less than an hourLess than an hour
Research Context
7/35
How to evaluate accessibility?
Less than an hourLess than an hour
TestsTests
Research Context
8/35
•User testing [Romen e Svanaes, 2008]
•WCAG conformance [W3C, 2008]
•Barrier walkthrough [Brajnik, 2006]
•Automatic accessibility tests
Research Context
9/35
•User tests
•WCAG conformance
•Barrier walkthrough
•Automatic accessibility tests
Less than an hourLess than an hour
TestsTests
Research Context
9/35
Agenda•Research context
•State of art
•Hypothesis
•Acceptance Tests
•Preliminary results
•Advantages and limitations
•Next steps
10/35
State of art
JavaScript
CSS
HTML
11/35
RIARIA
State of art
HTML
HTML ParserHTML Parser
Static HTML content analysis
Static HTML content analysis
[Velasco et al. 2008]
11/35
State of art
JavaScript CSS
HTML
Not evaluatedNot evaluated
11/35
Agenda
•Research context
•State of art
•Hypothesis
•Acceptance Tests
•Preliminary results
•Advantages and limitations
•Next steps
12/35
Hypothesis
Users
13/35
RIARIA
Hypothesis
Users
RIARIAA
13/35
HypothesisContinuous Integration
Developers
Source code repository
• Unit testsUnit tests
• Integration Integration
teststests
• CompileCompile
• Acceptance Acceptance
teststests
• Unit testsUnit tests
• Integration Integration
teststests
• CompileCompile
• Acceptance Acceptance
teststests
Report
14/35
Agenda
•Research context
•State of art
•Hipothesis
•Acceptance Tests
•Preliminary results
•Advantages and limitations
•Next steps
15/35
Acceptance Test
Definition
“A formal test that determines if a system satisfies its acceptance criteria and allow the user to identify when a
system will be accepted or not.”
16/35
Behaviour Acceptance tests
•Assertions made against the external interface
•Test cases reflects User Stories
17/35
Acceptance Test
18/35
Acceptance Test
19/35
Acceptance Test
Focus management [W3C, 2011]
20/35
Acceptance Test
TAB key navigation to the element
containing the text “Sports”
TAB key navigation to the element
containing the text “Sports”
20/35
Acceptance Test
TAB key navigation to the element
containing the text “Tennis”
TAB key navigation to the element
containing the text “Tennis”
20/35
Acceptance Test
Focus management [W3C, 2011]
20/35
Acceptance Test
Focus management [W3C, 2011]
20/35
Acceptance Test
Focus management [W3C, 2011]
20/35
Acceptance Test
Agenda
•Research context
•State of art
•Hipothesis
•Acceptance Tests
•Preliminary results
•Advantages and limitations
•Next steps
21/35
Focus management [W3C, 2011]
flyout menu
accordion menuoverlay menu
tabbed menu
flyout menu
accordion menuoverlay menu
tabbed menu
[Welie, 2008]
Preliminary results
Acceptance tests
Acceptance tests
HTML validators
HTML validators
22/35
Focus management [W3C, 2011]
All web tools that test navigation accessibility requirementshttp://www.w3.org/WAI/ER/tools/
complete
Cynthia
DaSilva
EvalAccess
WAVE
Accessibility check
Achecker
FAE report
Hera
Preliminary results
HTML validators
HTML validators
23/35
Focus management [W3C, 2011]
http://www.w3.org/WAI/ER/tools/complete
Cynthia
DaSilva
EvalAccess
WAVE
Accessibility check
Achecker
FAE report
Hera
Preliminary results
All web tools that test navigation accessibility requirementsHTML
validatorsHTML
validators
23/35
Focus management [W3C, 2011]
HTML validators
HTML validators
Acceptance tests
Acceptance tests
Preliminary results
24/35
Our hypothesis
DaSilva
EvalAccess
WAVE
FAE report
Focus management [W3C, 2011]
HTML ValidatorsHTML ValidatorsAcceptance
testsAcceptance
tests
Preliminary results
25/35
Focus management [W3C, 2011]
Preliminary results
HTML ValidatorsHTML ValidatorsAcceptance
testsAcceptance
tests
26/35
Focus management [W3C, 2011]
HTML ValidatorsHTML ValidatorsAcceptance
testsAcceptance
tests
Projects that present defects that were not identified by
the Tool
Projects that present defects that were not identified by
the Tool
Preliminary results
26/35
Continuous Integration
Developers
Source code repository
• Unit testsUnit tests
• Integration Integration
teststests
• CompileCompile
• Acceptance Acceptance
teststests
• Unit testsUnit tests
• Integration Integration
teststests
• CompileCompile
• Acceptance Acceptance
teststests
Report
Preliminary results
27/35
Integração Contínua
Developers
Source code repository
• Unit testsUnit tests
• Integraiton Integraiton
teststests
• CompileCompile
• Acceptance Acceptance
teststests
• Unit testsUnit tests
• Integraiton Integraiton
teststests
• CompileCompile
• Acceptance Acceptance
teststests
Report
Projects that might be put in production event though they
contain accessibility errors
Projects that might be put in production event though they
contain accessibility errors
Preliminary results
27/35
Focus management [W3C, 2011]
Preliminary results
HTML ValidatorsHTML ValidatorsAcceptance
testsAcceptance
tests
28/35
Focus management [W3C, 2011]
HTML ValidatorsHTML ValidatorsAcceptance
testsAcceptance
tests
Projects that present no accessibility errors
however are not put into production
Projects that present no accessibility errors
however are not put into production
Preliminary results
28/35
Continuous Integration
Developers
Source code repository
• Unit testsUnit tests
• Integration Integration
teststests
• CompileCompile
• Acceptance Acceptance
teststests
• Unit testsUnit tests
• Integration Integration
teststests
• CompileCompile
• Acceptance Acceptance
teststests
Report
Preliminary results
29/35
Integração Contínua
Developers
Source code repository
• Unit tetsUnit tets
•
TestesIntegration TestesIntegration
teststests
• CompileCompile
• Acceptance Acceptance
teststests
• Unit tetsUnit tets
•
TestesIntegration TestesIntegration
teststests
• CompileCompile
• Acceptance Acceptance
teststests
Report
Prevent the deployment of a stable
release of the software
Prevent the deployment of a stable
release of the software
Preliminary results
29/35
Focus management [W3C, 2011]
Preliminary results
HTML ValidatorsHTML ValidatorsAcceptance
testsAcceptance
tests
30/35
Focus management [W3C, 2011]
HTML Validators HTMLHTML Validators HTMLAcceptance
testsAcceptance
testsWarnings onlyWarnings only
Preliminary results
30/35
Focus management [W3C, 2011]
Preliminary results
Validadores HTMLValidadores HTMLAcceptance
testsAcceptance
tests
Do not make distinction between interface components that were
accessible and the ones that were not
Do not make distinction between interface components that were
accessible and the ones that were not
30/35
Agenda
•Research context
•State of art
•Hipothesis
•Acceptance Tests
•Preliminary results
•Advantages and limitations
•Next steps
31/35
Advantages
•In the experiment, the use of acceptance tests presented a higher number of correct assertions
•Higher effort required in order to execute the tests
•Assertions made agains the DOM structure inside of an use context makes it possible to evaluate ARIA requirements
32/35
Limitation
•The limitation of this hyphotesis is the higher cost to describe all contexts of the users
33/35
Agenda
•Research context
•State of art
•Hipothesis
•Acceptance Tests
•Preliminary results
•Advantages and limitations
•Next steps
34/35
•Implement new use scenarios for the acceptance tests
•Evaluate ARIA requirements
Next steps
35/35
REFERENCES
• Seojin Kim, Sungjin Park, Jeonghyun Yun, and Younghoo Lee. 2008. Automated Continuous Integration of Component-Based Software: An Industrial Experience. In Proceedings of the 2008 23rd IEEE/ACM International Conference on Automated Software Engineering (ASE '08). IEEE Computer Society, Washington, DC, USA, 423-426. DOI=10.1109/ASE.2008.64 http://dx.doi.org/10.1109/ASE.2008.64
• ROMEN, D.; SVANAES, D. Evaluating web site accessibility: validating the wai guidelines through usability testing with disabled users. In: NordiCHI ’08: Proceedings of the 5th Nordic conference on Human-computer interaction, New York, NY, USA: ACM, 2008, p. 535–538.
• W3C Conformance evaluation of web sites for accessibility. 2008a.Available at: http://www.w3.org/WAI/eval/conformance.html
REFERENCES
• BRAJNIK, G. Web accessibility testing: When the method is the culprit. In: Computers Helping People with Special Needs, Springer Berlin / Heidelberg, 2006, p. 156–163 (Lecture Notes in Computer Science, v.4061).Available at: http://www.springerlink.com/content/ h2573t61gu3n8533/
• C. A. Velasco, D. Denev, D. Stegemann, and Y. Mohamad. A web compliance engineering framework to support the development of accessible rich internet applications. In Proceedings of the 2008 international cross-disciplinary conference on Web accessibility (W4A), W4A ’08, pages 45–49, New York, NY, USA, 2008. ACM.
• W3C Accessible Rich Internet Applications (WAI-ARIA) 1.0. 2011. Available at: http://www.w3.org/TR/wai-aria/.
• Welie. 2008. Available at: http://www.welie.com/patterns/.
http://www.pyccuracy.orghttps://github.com/watinha/Pyccuracy-Accessibility-
Actions
Thank you very much!
URLs:
Contacts:watanabe
9th International Cross-Disciplinary Conference on Web Accessibility – 16/17th April 2012 – Lyon, France.
http://watinha.com/pyccuracy_test_1/templates/