Preliminary Evaluation Results: A Discussion with the Advisory Committee on Citizen-Friendly Reporting

August 30, 2005

Human Services Research Institute

Madeleine KimmichMike Nikolenko

Evaluation Background

All three state agencies selected a website as their preferred method to communicate performance information

HSRI developed the survey tool to help evaluate the three sites

Survey questions were based on the GASB criteria for the effective communication of performance measures

Our Core Evaluation Question: How well each project’s presentation of performance measure information is likely to engage and help interested citizens?1. Is the interface user-friendly?

(Understandability)2. Is the information they choose to

present easy to understand and meaningful? (Comparability/Timeliness/Consistency/ Reliability)

3. How can users find out if the information is available? (Accessibility)

The Survey Tool One set of questions for each agency Questions address each of the 6

GASB criteria: Accessibility (#1-2) Understandability (#3-5) Comparability (#6) Timeliness (#7) Consistency (#8a) Reliability (#8b)

The Survey Tool, ContinuedThe survey tool evolved over time:

Turning one survey into three

Eliminating some questions to reduce length

Adding in demographic information

Methodological Issues

Lack of focus groups Very low response rate Change in the method of survey

distribution Targeting respondents

See Table 1 and Figure 1

Results: Accessibility

Most respondents felt it was easy to open the web pages and navigate the sites.

See Tables 2 and 3

Respondent Comment (DHS):“Adobe is a poor format for web pages as the Acrobat reader

had to load and this takes time.”

Results: UnderstandabilityMost respondents felt the measures

presented on the sites were important.

Most respondents also said that the text was clear and helpful.

See Tables 4 and 5 and Figure 2

Respondent Comment (DHS):“The website helps tremendously to provide the public

with a snapshot of the organization.”

Results: ComparabilityMany respondents felt there was a

clear frame of reference for the data and that historical data was offered for comparison (DHS and DCBS).

DMV respondents differed on these questions.

See Tables 6 and 7

Respondent Comment (DMV): “The only thing missing was the

history of previous customer surveys.”

Results: Timeliness

Respondents reported knowing how recently data was collected and most felt this was recent enough to be useful.

See Tables 8 and 9

Respondent Comment (DCBS):“Very timely and clear…I really liked the few sentences

in the “on track” box.”

Results: Consistency and ReliabilityRespondents differed when asked if

they knew the origin of website data.

A majority of respondents trusted the accuracy of the data presented.

See Tables 10 and 11

Results: Other FindingsA majority of respondents said:-There is a convenient way to provide

feedback on the website-They learned something about the

agency from the site-The site did not make them feel

differently about the agency

See Tables 12-15 and Figure 3

Respondent Comment (DMV):“The survey shows that the agency is willing to receive input from customers so they can improve in areas needing work.”

Conclusions

All agency websites performed well Just say “no” to PDF files!

Text descriptions on the sites were helpful It is preferable to layer information

People are not used to critical examination of data This is part of educating interested

citizens

Conclusions, Continued

Educate citizens about the frame of reference What is the agency’s purpose?

It is viable to use websites to engage the public