40 peter korn light weight ui toolkit

AEGIS Workshop and International Conference, Brussels

Mobile Accessibility with the Java Lightweight UI Toolkit

Peter Korn, Accessibility Principal


LWUIT INTRO


What is LWUIT?

● Advanced, lightweight UI library● Compelling UI

● Consistent across different devices● For today's handsets (and more ...)● Portable

● MIDP, Blackberry, Android, CDC, SE, TV, ...● Inspired by Swing● Tools support● Open Source!

● GPLv2 + Classpath Exception, free for commercial use

● Active community

Lightweight User Interface Toolkit


LWUIT: Key Benefits

● Rapid development● Familiar API● Clean & simple

● Easy deployment● One jar, many devices

● Consistent & flexible● Customizable, extensible

● “Filthy Rich” UI● Brand-able● Designed for mass market devices

● Tested on broad range of hardware

Easy Development and Deployment


LWUIT: Key Features

● Swing-like MVC● Layouts● Fonts● Rich widgets● HTML widget● 3D & SVG integration● Touch screen support● Animations & transitions● Pluggable Look & Feel, theming● I18N/L10N support

Rich Functionality


ACCESSIBILITY: THE BASICS


Access Requirements for LWUIT

● Needs to be operable from the keyboard (by someone who can’t use a touchscreen)● LWUIT already has full phone keyboard

operability● Needs to support a variety of themes for folks

with poor vision – e.g. “high contrast”, “large print”● LWUIT already themeable

● Needs to support an accessibility API for interoperability with assistive technologies● The subject of the remainder of this presentation

What needs to be there to support users with disabilities


Accessibility API requirements

● Expose detailed information about UI components● Role, state(s), boundary, name & desc.● Min/max/current value of anything that takes a

value (e.g. slider)● Text contents, attributes, boundary of all text on

the display● Relationships of UI components with each other;

including tabular● Fire events when things happen to UI

components● Focus, text caret, selection, other UI component

changes● Allow programmatic modification of UI

components● Text insertion, selecting items in list, checking

buttons, etc.

Industry consensus on a “sufficient accessibility API”


Goals and Constraints

● Minimal impact on phones that aren't running AT

● Minimal impact on LWUIT API (only add something if we really need to)

● Run with existing LWUIT apps ● Minimal runtime impact for accessibility

metadata (e.g. name of icons, associating labels with text fields)

● Support native AT, support Java AT


Roles

● LWUIT base components mapped to ARIA roles:● Roles are determined by:

– If a “role” client property is assigned, then it overrides everything else

– instanceof checks: e.g. if component is TextArea: its role is TEXTBOX

– UIID checks: e.g. if the component is a Dialog, then if its UIID is “Menu” its role is MENU (otherwise its role is DIALOG)

– Hierarchy checks: e.g. if grandparent of a component is a Tabs class then it is either a TAB or a TABPANEL (according to class and UIID)

● Added roles:● LWUITCOMPONENT (Generic custom

component)● CONTAINER, LABEL


Attributes or States

● LWUIT base components get some attributes inherently● The user can specify attributes in the

component's client properties. These will override everything.

● Attributes are determined by base component states. For example:– aria-disabled – by isEnabled()– aria-visible – by isVisible()– aria-checked – by isSelected() – aria-expanded, aria-setsize, aria-posinset, aria-

level – for tree● Some attributes will be empty by default, since no

base component supports them (aria-owns)


ARCHITECTURE


Architecture – Event Bus

● The Event Bus listens on 2 TCP/IP ports, one for ATs and one for applications● The Bus opens an AppHandler/ATHandler

instance for each app/AT● ATHandler listens to AT queries, and forward

relevant App events to the AT● AppHandler listens to App events, and forwards

relevant AT queries to the App● All 3 runtimes share a common package that

is used to serialize event objects over the network (And also includes some constants such as roles and attributes)

● While current implementation is done with TCP/IP, we plan to use also OJWC's Pipe protocol.

LWUITapp

LWUITBroker

AT Handler

App Handler

Event Bus

ATBroker

ATapp

ApplicationApplication Event Bus AT


Architecture – AT

● ATBroker provides for AT developers event listeners and query methods, and on the other hand communicates with the Bus via TCP/IP (Or IMC in the future)

● The AT can signal (via the ATBroker) to which events it would like to listen

● The ATBroker receives App events, and notifies all listeners for that event

● The AT can query components states, children, bounds etc, all via simple methods which are translated to objects that are serialized and sent to the Bus

LWUITapp

LWUITBroker

AT Handler

App Handler

Event Bus

ATBroker

ATapp


LWUITapp

LWUITBroker

AT Handler

App Handler

Event Bus

ATBroker

ATapp



Architecture – Application

● The broker once started with Broker.init(), starts polling the application UI.

● Once a new Form comes up, it adds itself (and helper objects) as listeners to all components in the UI.

● In addition it polls other changes to the UI for which we do not have events (Such as hierarchy events, caret events etc.) and sends them as events to the Bus

● Events are then sent to the Bus as serialized AEvent objects (The Bus then decides to which AT it should be forwarded, according to what the ATs specified)

● The broker also does the job of assigning LWUIT base components with roles and attributes.

● The Broker works in a transparent way to the LWUIT App developer, and aside from calling init(), the developer doesn't use it (Perhaps aside of constants)

LWUITapp

LWUITBroker

AT Handler

App Handler

Event Bus

ATBroker

ATapp



Packages

● accessibility.common – a common package used by all 3 runtimes (AT/App/Bus) for a unified way of serializing event objects. Includes:● AEvent – the base event class. Basically

serializes the event type and a queryId. AEvent can also be a query or response (Considering elaborating the class structure)

● Other events: RegisterEvent (App/AT identification), FocusEvent, ComponentQuery (queries component states/children etc.), StatesResponses (Returns component queries)

● Role – constants representing the ARIA roles and their names

● Attribute – constants representing the ARIA att. and their names

● accessibility.bus – contains the bus classes (Bus, ATHandler, AppHandler)

● accessibility.at – Used by AT developers (Contains ATBroker, listeners)

● accessibility.app – Used by the LWUIT App, contains mainly the Broker


Code Samples: Enabling Access

● Making a LWUIT app accessible is very easy, simply add to your midlet: Broker.init(“App Name”);

● Notes: ● This should be added after the

Display.init(midlet) call.● Your app should use the ATBroker project● This applies to the prototype only, when the

accessibility technology will be integrated into OJWC, There would be no need to explicitly initialize the broker, it would be done by LWUIT.


Code Samples: Custom Properties

● Most ARIA properties are determined automatically by the Broker.

● However, some custom components may require setting various properties

● Example: Let's say that we have a custom component that needs to relay the information that it has 6 items:

putClientProperty(Attribute.getAttName(Attribute.ARIA_SETSIZE), 6);

● Notes: ● Only properties listed in the Attribute class are

relevant.● Property changes are scanned by the Broker via

polling.


Code Samples: Setting names

● Another way of adding accessibility info is by using:component.setLabelForComponent(Label componentLabel);● This was created in LWUIT for other uses.● It is especially useful for components that don’t

have a defined name (For example TextArea/Field vs. Label/Button)

● Example:Label nameLabel = new Label(“Name:”);TextField nameField = new TextField();container.addComponent(nameLabel);container.addComponent(nameField);nameField.setLabelForComponent(nameLabel);


Code Samples: ListCellRenderers● Work is needed for custom list cells

● Set the appropriate Role for the cell● Set the appropriate Name, Description● Ensure correct Selection, item count

● Example from ScrollDemo.java:public Component getListCellRendererComponent(List list, Object value, int index, boolean isSelected) { Contact person = (Contact) value; name.setText(person.getName()); email.setText(person.getEmail()); pic.setIcon(person.getPic()); return this;}


Code Samples: ListCellRenderers

public Component getListCellRendererComponent(List list, Object value, int index boolean isSelected) { Contact person = (Contact) value; name.setText(person.getName()); email.setText(person.getEmail()); pic.setIcon(person.getPic()); putClientProperty(Attribute.ATT_NAMES[Attribute.ROLE], new Integer(Role.LISTITEM)); putClientProperty(Attribute.ATT_NAMES[Attribute.ALT], person.getName() + " " + person.getEmail()); if (isSelected) { putClientProperty(Attribute.ATT_NAMES[Attribute.ARIA_SELECTED], Boolean.TRUE); } putClientProperty(Attribute.ATT_NAMES[Attribute.ARIA_POSINSET], Integer.toString(index)); putClientProperty(Attribute.ATT_NAMES[Attribute.ARIA_SETSIZE], Integer.toString(5*CONTACTS_INFO.length)); return this;}


Code Samples: AT

● Writing an assistive technology is also quite easy.

● First, initialize the ATBroker:ATBroker.getInstance().init(midlet,"AT Name");

● Implement the relevant Listener● For focus for example, implement

AFocusListener that gets called on its focusGained method.

● Other listeners: – AHierarchyListener, ADataChangeListener,

ACaretListener, APropertyChangeListener● Then, register on events:

ATBroker.getInstance().addFocusListener(this);

● Notes: ● This should be added after the

Display.init(midlet) call.● Your app should use the ATBroker project


Code Samples: AT

import accessibility.at.AFocusListener;import accessibility.at.ATBroker;import accessibility.common.FocusEvent;import javax.microedition.midlet.MIDlet;import javax.microedition.midlet.MIDletStateChangeException;public class SimpleATMIDlet extends MIDlet implements AFocusListener { protected void startApp() throws MIDletStateChangeException { ATBroker.getInstance().init(this, "Simple AT"); ATBroker.getInstance().addFocusListener(this); } public void focusGained(FocusEvent event) { System.out.println("Gained focus on " + event.getName() + " (Role: " + event.getRole()); } protected void pauseApp() { } protected void destroyApp(boolean unconditional) throws MIDletStateChangeException { }}


Monkey ApplicationScans all UI components and provides details about them

● Demonstrates the abilityfor AT to query entire UIhierarchy

● Also a great accessibilitydebugging tool


DEMOS

Technology

40 peter korn light weight ui toolkit