Reliability of ADA-CAT for ADA

Compliance

Reliability of ADA-CAT for ADA

ComplianceDenis Anson

Director of Research and Development

Assistive Technology Research Institute

Denis AnsonDirector of Research and

DevelopmentAssistive Technology Research

Institute

What is ADA-CAT?What is ADA-CAT?

ADA-CAT is a combination of a website and set of physical tools

ADA-CAT is a screening tool for accessibility of the environment, based on ADA standards

ADA-CAT was designed to allow non-engineers and non-architects to perform reliable and accurate assessments of compliance with ADA requirements

The WebsiteThe Website

The ADA-CAT website provides assessment checklists (Audits) for features of the environment

The WebsiteThe WebsiteEach item in an audit includes:

The ADA (or other) requirement, in understandable language

A link to the ADA-ABA requirement on the web

An explanation of why this feature is important

Just-In-Time Training

Just-in-Time TrainingJust-in-Time TrainingTraining you get today will be forgotten when you need it

ADA-CAT allows you to review how to assess an item as you are assessing

This includes both text and pictures of the process

The training is based on the ADA-CAT toolkit

The ADA-CAT ToolkitThe ADA-CAT Toolkit

The toolkit includes eleven tools in a custom bag

The tools include three off-the-shelf tools

25-foot tape measure

24 inch spirit level

Stopwatch

Three Off-The-Shelf Tools

Three Off-The-Shelf Tools

The tools are easily found in sports and hardware stores

The assumption might be that everyone knows how to use them

Watching novices attempt to use these common indicates that this isn’t the case

The ADA-CAT ToolkitThe ADA-CAT Toolkit

The toolkit contains three hard to find, but off-the-shelf tools

Light meter reading in Lux

A sound meter reading decibels

A force gauge

Three Hard-To-Find Tools

Three Hard-To-Find Tools

It’s unlikely that many have ever used tools like this

Most people have little understanding of the concepts being measured

Inverse Squared Rule?

Although the tools are “simple” to use, if not used correctly, they won’t give good results

The ADA-CAT ToolkitThe ADA-CAT ToolkitThe toolkit contains five custom tools not available anywhere else

Magic Slope Block

MultiTool

Key Torque Tool

The Font Guide

The StoryStick

Five Custom ToolsFive Custom Tools

These tools were created specifically for use in ADA-CAT assessments

Although the principles are hidden from the user, they can be complex

No-one who has not used ADA-CAT will have any experience with these tools.

Little or No Training Required?

Little or No Training Required?

The assertion is that ADA-CAT, the combination of the website and the tools, will produce reliable results when used by non-architects

This is a bold claim, and requires support

It is also testable

What is Reliability?What is Reliability?

There are two types of reliability that are commonly recognized in tests and measurements

Test-Retest Reliability

Inter-Rater Reliability

Test-Retest Reliability

Test-Retest Reliability

Test-Retest reliability means that when a measurement is taken repeatedly, the outcome is the same

When you measure the width of a door, you should always get the same width for the same door

Tools that have large subjective components will not show this reliability

Inter-Rater ReliabilityInter-Rater Reliability

Formally, this means that the results of a measurement are not dependent on the person doing the measurement

If I measure the weight of a glass of water, then you do, we should get the same result

In accessibility assessment, the “I was/wasn’t able to do it” test depends on who “I” am

Testing ADA-CAT Reliability

Testing ADA-CAT Reliability

A group of four OT research students undertook this task

The students received an introduction to the tools of the toolkit, but no observed trial in the real world

The students used the Elevator Audit as their sample test

Why Elevators?Why Elevators?

The Elevator Audit uses almost all of the tools of the toolkit

We didn’t use the Font Guide, but did use all the other tools

Elevators are commonly available

There is a wide range of accessibility in elevators in daily use

MethodMethod

Identify eight elevators to be assessed

Each researcher measures each elevator

After at least three weeks, each researcher measures each elevator again

Test-Retest Reliability

Test-Retest Reliability

Each researcher’s measurements can be compared with their measurements three weeks later.

Assuming no damage or repair to the elevators, there should be good agreement between the results

Inter-Rater ReliablityInter-Rater ReliablityThe measurements of each researcher can be compared to show the inter-rater reliability

When the scores for all four researchers were compared, the inter-rater reliability was .6

Statistically significant, but not exciting

Why?

Inter-Rater ReliabilityInter-Rater Reliability

Inter-Rater ReliabilityInter-Rater Reliability

Examining the data shows that, in most cases, three of the researchers had closely related data, and one was different.

Removing this individual increased the inter-rater reliability score to above .9

So why was this researcher’s scores different?

We’ll look at that in a minute.

Inter-Rater Reliability (Filtered)

Inter-Rater Reliability (Filtered)

Test-Retest Reliability

Test-Retest Reliability

If the scores of the researchers on first measurement are compared with the measurements 3 weeks later, the correlation is .97

Test-Retest Reliability

Test-Retest Reliability

So, What’s Wrong with Inter-Rater?So, What’s Wrong with Inter-Rater?

Individual items were compared between researchers where there was not good agreement

The differences clustered around two features

Sound levels

Braille

Sound LevelsSound LevelsThe Elevator Audit includes standards for the relationship between elevator signals and background noise, and for background noise levels

The outlier researcher did her evaluations during times when there was a lot of surrounding activities

During a basketball game for the athletic center elevator

Sound LevelsSound Levels

The background sound levels at busy times represent the people in the area, not the elevator, and cannot be controlled by the manufacturer

The sound measurements should be taken at quiet times, as the other three researchers did

Braille?Braille?Visually dependent people have a very difficult time determining if Braille is appropriate

In some of the older, and more abused elevators used in this study, the “raised” component of Braille was lost due to damage and wear

This was interpreted differently by different researchers, resulting in some variability

ConclusionsConclusions

Some additional instructions are needed on when to take sound measurements

Additional examples and counter-examples of Braille need to be added to the assessments

ConclusionsConclusions

Overall, the assertion that ADA-CAT reliably measures compliance with accessibility standards is well supported by this project