Evaluation of the Android Accessibility API

Recognition Rate towards a Better User Experience

Mauro C. Pichiliani (pichilia@ita.br)Celso M. Hirata (hirata@ita.br)

Instituto Tecnológico de Aeronáutica - Department of Computer Science

Evaluate the UI recognition accuracy rate of mobile applications by

measuring how many UI elements

are correctly identified by

an accessibility API

Goal

Introduction Online stores with thousands of applications

Interaction model: touches and gestures

Special users (visually impaired) and eyes-free scenarios

Accurate target indentification by accesibility APIs benefits:

• Accessibility services and applications• Automatic extraction of task sequences• UI automation testing• Collaboration frameworks

What is the recognition accuracy rate of UI elements provided by Accessibility APIs

on popular applications?

Related work

HCI literature: focus on size, location and visual properties of UI elements

Hurst et al. [5]: 74% of correct target identification on desktop (location and size)

Content information: used by accessibility services (i.e. screen readers)

Other approaches to increase the recognition of UI elements

Recent efforts for Web accessibility Strategies Guidelines Resources

[5] Hurst, A., Hudson, S. E., Mankoff, J. Automatically identifying targets users interact with during real world tasks. In: Proceedings of the 15th international conference on Intelligent user interfaces, p. 11-20 (2010)

Mobile accessibility API Get low-level information about targets. E.g.: MSAA API

Operating systems have accessibility applications (screen readers, magnification glass)

Android platform provives the most complete accessibility API:

Low-level hooks that capture events Complete identification of the element Reconstruction of the UI View hierarchy Default textual description New accessibility service creation Integration with external devices (e.g. braile

keyboards)

Evaluation methodology (1) Capture of application’s screenshots Accessibility service developed to read the contentDescription and

capture events raised Top 10 most popular applications (February 16th, 2015)

Evaluation methodology (2)

Dynamic content

Dialog messages

Web pages

Common Android activities

Reused elements on distinct activities

Results

Discussion (1) Overall, apps have a high event trigger ratio (99.48%)

Dynamic elements and popup’s unreachable by the accessibility API. Examples:

Discussion (2) Lack of content description for messages, images and buttons

(94.07%) . Examples:

Conclusion & Future work Mobile apps guided by visual access to on screen targets

Impaired users and eyes-free scenarios

97% recognition rate (99.48% event trigger rate, 94.07% content rate)

There is room to improve mobile accessibility APIs

Better APIs impact other contexts

Future work: Evaluation and comparison on other mobile OSs Validation with users Comparison of content descripton effectiveness Test techniques to augment accesibility APIs