Using Acceptance Tests to Validate Accessibility Requirements in RIA

Using Acceptance Tests to Validate

Accessibility Requirements in RIA

Willian Massami WatanabeRenata Pontin de Mattos Fortes

Ana Luiza Dias

9th International Cross-Disciplinary Conference on Web Accessibility – 16/17th April 2012 – Lyon, France.

Agenda

•Research context

•State of art

•Hypothesis

•Acceptance Tests

•Preliminary results

•Advantages and limitations

•Next steps

2/35

Agenda

•Research context

•State of art

•Hypothesis

•Acceptance Tests



•Next steps

2/35

Research Context

3/35

Research Context

4/35

Web 2.0

Perpetual Perpetual betabeta


Research Context

4/35

Users

Research Context



4/35

Users

• New features• Web applications

improvements

• New features• Web applications

improvements

Research Context



4/35

Less than an hourLess than an hour

Research Context



5/35

RequirementsRequirements

DesignDesign

ImplementationImplementation

TestsTests

DeploymentDeployment


Research Context



5/35

Continous Integration

Continuouslly compile and test the software to ensure quality

[Kim et al. 2008]

Research Context

5/35

Continuous Integration

Developers

Research Context

6/35


Developers

Source code repository

Research Context

6/35

Developers


• Unit testsUnit tests

• Integration Integration

teststests

• CompileCompile

• Acceptance Acceptance

teststests



teststests

• CompileCompile


teststests

Report

Research Context

6/35


Developers




teststests

• CompileCompile


teststests



teststests

• CompileCompile


teststests

Report

Research Context

6/35


Developers




teststests

• CompileCompile


teststests



teststests

• CompileCompile


teststests

Report

Research Context

6/35


RequirementsRequirements

DesignDesign

ImplementationImplementation

TestsTests

DeploymentDeploymentQuality




Research Context

7/35

How to evaluate accessibility?


TestsTests

Research Context

8/35

•User testing [Romen e Svanaes, 2008]

•WCAG conformance [W3C, 2008]

•Barrier walkthrough [Brajnik, 2006]

•Automatic accessibility tests

Research Context

9/35

•User tests

•WCAG conformance

•Barrier walkthrough

•Automatic accessibility tests


TestsTests

Research Context

9/35

Agenda•Research context

•State of art

•Hypothesis

•Acceptance Tests



•Next steps

10/35

State of art

JavaScript

CSS

HTML

11/35

RIARIA

State of art

HTML

HTML ParserHTML Parser

Static HTML content analysis

Static HTML content analysis

[Velasco et al. 2008]

11/35

State of art

JavaScript CSS

HTML

Not evaluatedNot evaluated

11/35

Agenda

•Research context

•State of art

•Hypothesis

•Acceptance Tests



•Next steps

12/35

Hypothesis

Users

13/35

RIARIA

Hypothesis

Users

RIARIAA

13/35

HypothesisContinuous Integration

Developers




teststests

• CompileCompile


teststests



teststests

• CompileCompile


teststests

Report

14/35

Agenda

•Research context

•State of art

•Hipothesis

•Acceptance Tests



•Next steps

15/35

Acceptance Test

Definition

“A formal test that determines if a system satisfies its acceptance criteria and allow the user to identify when a

system will be accepted or not.”

16/35

Behaviour Acceptance tests

•Assertions made against the external interface

•Test cases reflects User Stories

17/35

Acceptance Test

18/35

Acceptance Test

19/35

Acceptance Test

Focus management [W3C, 2011]

20/35

Acceptance Test

TAB key navigation to the element

containing the text “Sports”


containing the text “Sports”

20/35

Acceptance Test


containing the text “Tennis”


containing the text “Tennis”

20/35

Acceptance Test


20/35

Acceptance Test


20/35

Acceptance Test


20/35

Acceptance Test

Agenda

•Research context

•State of art

•Hipothesis

•Acceptance Tests



•Next steps

21/35


flyout menu

accordion menuoverlay menu

tabbed menu

flyout menu

accordion menuoverlay menu

tabbed menu

[Welie, 2008]

Preliminary results

Acceptance tests

Acceptance tests

HTML validators

HTML validators

22/35


All web tools that test navigation accessibility requirementshttp://www.w3.org/WAI/ER/tools/

complete

Cynthia

DaSilva

EvalAccess

WAVE

Accessibility check

Achecker

FAE report

Hera

Preliminary results

HTML validators

HTML validators

23/35


http://www.w3.org/WAI/ER/tools/complete

Cynthia

DaSilva

EvalAccess

WAVE

Accessibility check

Achecker

FAE report

Hera

Preliminary results

All web tools that test navigation accessibility requirementsHTML

validatorsHTML

validators

23/35


HTML validators

HTML validators

Acceptance tests

Acceptance tests

Preliminary results

24/35

Our hypothesis

DaSilva

EvalAccess

WAVE

FAE report


HTML ValidatorsHTML ValidatorsAcceptance

testsAcceptance

tests

Preliminary results

25/35


Preliminary results


testsAcceptance

tests

26/35



testsAcceptance

tests

Projects that present defects that were not identified by

the Tool

Projects that present defects that were not identified by

the Tool

Preliminary results

26/35


Developers




teststests

• CompileCompile


teststests



teststests

• CompileCompile


teststests

Report

Preliminary results

27/35

Integração Contínua

Developers



• Integraiton Integraiton

teststests

• CompileCompile


teststests


• Integraiton Integraiton

teststests

• CompileCompile


teststests

Report

Projects that might be put in production event though they

contain accessibility errors

Projects that might be put in production event though they

contain accessibility errors

Preliminary results

27/35


Preliminary results


testsAcceptance

tests

28/35



testsAcceptance

tests

Projects that present no accessibility errors

however are not put into production

Projects that present no accessibility errors

however are not put into production

Preliminary results

28/35


Developers




teststests

• CompileCompile


teststests



teststests

• CompileCompile


teststests

Report

Preliminary results

29/35

Integração Contínua

Developers


• Unit tetsUnit tets

•

TestesIntegration TestesIntegration

teststests

• CompileCompile


teststests

• Unit tetsUnit tets

•

TestesIntegration TestesIntegration

teststests

• CompileCompile


teststests

Report

Prevent the deployment of a stable

release of the software

Prevent the deployment of a stable

release of the software

Preliminary results

29/35


Preliminary results


testsAcceptance

tests

30/35


HTML Validators HTMLHTML Validators HTMLAcceptance

testsAcceptance

testsWarnings onlyWarnings only

Preliminary results

30/35


Preliminary results

Validadores HTMLValidadores HTMLAcceptance

testsAcceptance

tests

Do not make distinction between interface components that were

accessible and the ones that were not

Do not make distinction between interface components that were

accessible and the ones that were not

30/35

Agenda

•Research context

•State of art

•Hipothesis

•Acceptance Tests



•Next steps

31/35

Advantages

•In the experiment, the use of acceptance tests presented a higher number of correct assertions

•Higher effort required in order to execute the tests

•Assertions made agains the DOM structure inside of an use context makes it possible to evaluate ARIA requirements

32/35

Limitation

•The limitation of this hyphotesis is the higher cost to describe all contexts of the users

33/35

Agenda

•Research context

•State of art

•Hipothesis

•Acceptance Tests



•Next steps

34/35

•Implement new use scenarios for the acceptance tests

•Evaluate ARIA requirements

Next steps

35/35

REFERENCES

• Seojin Kim, Sungjin Park, Jeonghyun Yun, and Younghoo Lee. 2008. Automated Continuous Integration of Component-Based Software: An Industrial Experience. In Proceedings of the 2008 23rd IEEE/ACM International Conference on Automated Software Engineering (ASE '08). IEEE Computer Society, Washington, DC, USA, 423-426. DOI=10.1109/ASE.2008.64 http://dx.doi.org/10.1109/ASE.2008.64

• ROMEN, D.; SVANAES, D. Evaluating web site accessibility: validating the wai guidelines through usability testing with disabled users. In: NordiCHI ’08: Proceedings of the 5th Nordic conference on Human-computer interaction, New York, NY, USA: ACM, 2008, p. 535–538.

• W3C Conformance evaluation of web sites for accessibility. 2008a.Available at: http://www.w3.org/WAI/eval/conformance.html

REFERENCES

• BRAJNIK, G. Web accessibility testing: When the method is the culprit. In: Computers Helping People with Special Needs, Springer Berlin / Heidelberg, 2006, p. 156–163 (Lecture Notes in Computer Science, v.4061).Available at: http://www.springerlink.com/content/ h2573t61gu3n8533/

• C. A. Velasco, D. Denev, D. Stegemann, and Y. Mohamad. A web compliance engineering framework to support the development of accessible rich internet applications. In Proceedings of the 2008 international cross-disciplinary conference on Web accessibility (W4A), W4A ’08, pages 45–49, New York, NY, USA, 2008. ACM.

• W3C Accessible Rich Internet Applications (WAI-ARIA) 1.0. 2011. Available at: http://www.w3.org/TR/wai-aria/.

• Welie. 2008. Available at: http://www.welie.com/patterns/.

http://www.pyccuracy.orghttps://github.com/watinha/Pyccuracy-Accessibility-

Actions

Thank you very much!

URLs:

Contacts:watanabe

[email protected]

[email protected]

[email protected]

9th International Cross-Disciplinary Conference on Web Accessibility – 16/17th April 2012 – Lyon, France.

http://watinha.com/pyccuracy_test_1/templates/

Technology

Using Acceptance Tests to Validate Accessibility Requirements in RIA