Upload
sandra-greene
View
253
Download
0
Tags:
Embed Size (px)
Citation preview
The New COM API for Accessibility and Automation in Windows 7
Michael BernsteinPrincipal Software Design EngineerMicrosoft Corporation
Programmatic UI Access: A Key Element of Accessibility
Enables developers to write code that:
Navigates between UI elements
Gathers information about the UI
Interacts with UI elements
Receives notifications when the UI changes
Accessibility
Programmatic Access
2
UI Automation (UIA):Programmatic Access in Windows
Successor to Microsoft Active Accessibility (MSAA) Creates a universal object model for accessing UI Designed based on experience with MSAA usage Supported on Windows Vista, XP and Server 2003 UIA COM Client API introduced in Windows 7
3
UI Automation Core MSAA
UIA Managed Client API UIA Provider API UIA COM Client
API
What’s new in Windows 7?
UIA Clients can now use COM instead of managed code
New Accessibility properties are available Native clients can register custom ‘proxies’
to make existing UI more accessible Developers can define custom Accessibility
properties and events beyond the official UIA list
Inspecting UIA Properties
demo
Context: Clients and Providers
Client-side API (C# or C++)
Client Provider
UIA Infrastructure
UIA
Provider-side interface definitions (C# or C++)
Test Automation and Assistive Technology Manufacturers
Application and Control Developers
TAKING ADVANTAGE OF THE NEW COM CLIENT API
Client ProviderUIA
SAMPLE: Querying a window’s password statusCComPtr<IUIAutomation> spAutomation;CComPtr<IUIAutomationElement> spElement;HRESULT hr;hr = spAutomation.CoCreateInstance(CLSID_CUIAutomation);if (SUCCEEDED(hr)){
hr = spAutomation->GetFocusedElement(&spElement);if (SUCCEEDED(hr)){
BOOL isPassword;spElement-
>get_CurrentIsPassword(&isPassword);}
}
Automation Elements
Just as in the managed API, an automation element represents a piece of UI, regardless of underlying implementation First, create an instance of
CUIAutomation, the factory for the object model
Next, get an element: From an HWND: ElementFromHandle From a Point: ElementFromPoint From current focus: GetFocusedElement
Each method returns an IUIAutomationElement
Basic Properties
IUIAutomationElement exposes properties that make sense for elements in general
Properties are exposed in COM manner and are consistent across all underlying UI
UIA Control Type
UI Framework Framework Specific
Property
UI Automati
onProperty
Button Windows Presentation Foundation
Content NameProperty
Button Win32 Caption NameProperty
Image Trident/HTML ALT NameProperty
Three Key Properties
1. NameProperty String that a user would use to explain which control
was being referred to (Customer: Accessibility Tools)
2. ControlTypeProperty Primary identifier for what type of control the
automation element represents. (Customers: Both)
3. AutomationIdProperty String that uniquely identifies an element among its
siblings in the tree (Customer: Test Automation)
Control Patterns
Control Patterns allow access to a control’s particular abilities InvokePattern:
‘Push Me’ TransformPattern:
‘Move Me’ ScrollPattern:
‘Scroll Me’ SelectionItemPattern:
‘Select Me’ Patterns can contain
properties, methods and events
ComboBox
Simple
ExpandCollaps
e
Value
Find Requests
Find requests allow you to locate elements based on conditions Conditions are based on property matching They can include Boolean operators as well
Requests can be restricted by scope Find can return the first match or all
matches Faster alternative to manual searching
Visiting the Relatives: TreeWalker
All UIA elements are organized into a tree
TreeWalkers allow navigation around the tree GetParentElement,
GetFirstChildElement,GetNextSiblingElement, etc.
Can filter by condition
List View
ListItem1 ListItem2 Scroll Bar
Up button
Down button
A sample control tree:
Consuming Events
Enable clients to be notified when changes occur in the UI Client implements a listener interface Register your listener with CUIAutomation
Clients can scope what parts of automation tree to listen for events on
Applicable events are always associated with a reference to automation element where change occurred
http://msdn.microsoft.com/en-us/library/aa359583(VS.85).aspx
15
DumpUIATree
sample
Accelerate your access: CacheRequests
Define •Create a request and add entries for properties and patterns you want cached•Define filter and scope for caching
Query •Call the element’s BuildUpdatedCache method or “…BuildCache” methods•A new copy with cached data is returned in one chunk
Use •Use ‘Cached’ flavor of property accessors to read cached data or patterns•Children can be cached as well
Revisiting DumpUIATree
sample
NEW PROPERTIES AND PATTERNS
Client ProviderUIA
Patterns for Control Virtualization
Allows controls to have more children than are exposed in Automation tree at one time Parent container permits searching for children
by property Children can be de-virtualized upon request
Useful for: Containers with massive numbers of children
Other new properties and patterns
Name PurposeARIA Properties Expose ARIA properties for
browser elements
Provider Description Query for the name of the provider(s) for an element
Synchronized Input Pattern Wait for a UI element to receive mouse/kbd input
Legacy MSAA Pattern Access original MSAA properties for a provider
Menu mode start/end events Find out when an application enters/exits menu mode
BUILDING GREAT UIA PROVIDERS
Client ProviderUIA
When do you need a provider?
You don’t always need to write a provider. Customizing beyond your framework
Changing perceived properties Annotation API may be sufficient
Adding new abilities Full custom controls
Creating new frameworks Silverlight had to implement Accessibility
Implementing the simple interface
Create a COM object that implements IRawElementProviderSimple Retrieve VARIANT properties
Implement the most important ones for your control, especially Name and ControlType
Retrieve a pattern implementation Connect your COM object to the outside
world through WM_GETOBJECT
Implementing patterns
Control patterns are much like interfacesCan reside on same
object or different object from Simple interface
Implement patterns that reflect your control’s abilities
ComboBoxSimpleExpandCollapseInvokeTableTextToggleValueVirtualized
A basic custom-control UIA provider
sample
http://msdn.microsoft.com/en-us/accessibility/cc307849.aspx
Beyond Simple Elements
Representing children Free support for mirroring HWND tree When you have non-window children …
Implement Fragment for all nodes in your tree Implement FragmentRoot for the root of your tree
Firing Events If a user would notice a change in your control,
you need to fire an event You may check for listeners first
Custom patterns and properties
Allows client and provider to exchange data beyond original Accessibility design
Custom work must be done on both sides Properties and events: business as usual Methods: extra marshalling work required
Client ProviderUIACustom dataCustom data
Custom ‘proxy’ support
Enables a client to register a substitute provider for a UI that lacks a real provider Proxy implements UIA interfaces on behalf of
UI Useful for legacy code that cannot be
modified but must be made accessible
Client ProviderUIA
Custom Proxy Non-UIA channel
WRAPPING IT UP
Is this all of Accessibility?
Is your software usable with … Programmatic access? Keyboard-only access? High contrast / high DPI? Slow reaction times?
Accessibility is not a yes/no question: it is usability for a specific audience
http://msdn.microsoft.com/accessibility
What about managed clients?
The new COM API was designed with .NET interoperability in mind Pro: Fastest performance for programmatic access Con: Imported COM interfaces are not identical with
older interfaces Con: Some Win32 controls have slight behavior
differences
UI Automation Core
UIA Managed Client API UIA Provider API UIA COM Client API
Your Client Code;TlbImport Wrapper
Managed client option #2:Use the existing managed library
We continue to support the existing System.Windows.Automation namespace Pro: No change to existing code Con: May be slower when accessing MSAA applications Con: Cannot access new properties and patterns with
current version
UI Automation Core
UIA Managed Client API UIA Provider API UIA COM Client API
Your Client Code
Call to Action
Use the new UI Automation Client API to interact with existing user interfaces
Create UI Automation providers for new custom controls and UI frameworks
Consider the Accessibility needs of your customers when designing applications
© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market
conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.