Evaluation of NIH ACSI Website Project: Highlights of the Final Report

NIH ACSI Meeting Oct. 4, 2006

1

Evaluation of NIH ACSI Website Project: Highlights of the Final Report

Jennifer Crafts, Ph.D.Westat


2

Today’s Topics

Review objectives of the evaluation

Summarize methodology Present highlights of results Present recommendations Q & A


3

Objectives of the Evaluation: Individual Teams

Objective 1: Through the offer of an ACSI license, were teams encouraged to use an online customer satisfaction survey?

Objective 2: What was the value of using the ACSI?


4

Objectives of the Evaluation: Trans-NIH

Objective 3: Did broad ACSI use provide additional enterprise-wide benefits?

Objective 4: Did the evaluation provide any additional understanding about:

How NIH sites are used? How NIH sites are meeting NIH

communication goals?


5

Methods: All Teams

Surveys Observations Review/analysis of secondary data


6

Methods: Subset of Teams

Interviews

Observations


7

Results -- Objective 1:Use of New Evaluation Method

18 ICs, 13 ODOs participated 60 initial licenses 55 licenses active into 2006


8

Results -- Objective 2:Value of ACSI Use for Teams

ACSI activities teams conducted Overall satisfaction with ACSI Use

Factors that impacted satisfaction Usefulness of custom questions and ACSI

scores Use of ACSI data

Planned site improvements Plans for use in redesign Barriers to making planned changes

ACSI score changes Use of ACSI if IC/ODO Had to Pay for License

NIH ACSI Meeting Oct. 4, 2006 9

14% (2)

71% (10)

93% (13)

86% (12)

93% (13)

93% (13)

100% (14)

100% (14)

93% (13)

79% (11)

0% (0)

22% (4)

94% (17)

89% (16)

94% (17)

89% (16)

89% (16)

78% (14)

33% (6)

6% (1)

17% (3)

0% (0)

63% (12)

42% (8)

32% (6)

21% (4)

21% (4)

47% (9)

37% (7)

5% (1)

5% (1)

5% (1)

16% (3)

0% 20% 40% 60% 80% 100% 120%

Reviewed report of ACSI results beforereaching 300 completes

Received initial results report (300completes)

Participated in feedback meeting with SRA toreview initial results

Used ACSI results to monitor siteperformance

Used ACSI segmentation to learn moreabout site visitors

Modified/tailored custom questions for in-depth analysis

Used ACSI results to plan site improvements

Implemented planned changes to site basedon ACSI results

Used ACSI results to evaluate effects of sitechanges (pre-launch vs. post-launch)

Used ACSI feedback in continuousimprovement process

None of the above activities

Percentage of sites

Term start date on or before Dec 2004 (n=14)

Term start date and feedback meeting during 2005 (n=18)

Late start date or slow collecting site (n=19) Method - User survey (n=51)

XXXXXXX

Reported Activities (broken out by license term start date)


Overall Satisfaction With Use of ACSI to Evaluate Site

43% (n=22) 23% (n=12)6%

(n=3) 12% (n=6)6%

(n=3) 10% (n=5)

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree Not applicable

Method - User survey (n=51)

Percentage of sites


11

Team’s Perception of Value of ACSI

Factors that had greatest impact Longevity of license Timing of license


Usefulness of Custom Questions and ACSI Scores


16% (n=8)

18% (n=9)

49% (n=25)

31% (n=16)

27% (n=14)

41% (n=21)

29% (n=15)

29% (n=15)

27% (n=14)

29% (n=15)

6%(n=3)

13% (n=7)

4% (n=2)

6%(n=3)

10% (n=5)

10% (n=5)

10% (n=5)

8%(n=4)

8%(n=4)

4% (n=2)

6%(n=3)

2% (n=1)

2% (n=1)

22% (n=11)

24% (n=12)

24% (n=12)

18% (n=9)

24% (n=12)

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Custom questions useful forevaluating site

Overall satisfaction score usefulfor evaluating site

Element scores useful forevaluating site

Future behavior scores useful forevaluating site

Confident that scores reflectsite's strengths and weaknesses

Percentage of sites

Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree Not applicable


Site Teams’ Use of ACSI Data

1 (2%)

2 (4%)

2 (4%)

5 (10%)

10 (20%)

11 (22%)

13 (25%)

14 (27%)

21 (41%)

22 (43%)

29 (57%)

0 5 10 15 20 25 30 35

Provide feedback to the IC

Participate in customer satisfactionbenchmarking

Establish program priorities

Share the results with a contractorthat manages the web site

Make plans for use of other web siteevaluation methods

Promote the IC and/or the web site

Other

Establish budget priorities

Publish/present a paper about thesite's experience with the ACSI

Don't know

Evaluate contractor performance

Number of sites

Method - Final user survey (n=51)


2 (4%)

4 (8%)

6 (12%)

10 (20%)

17 (33%)

17 (33%)

18 (35%)

19 (37%)

20 (39%)

23 (45%)

23 (45%)

0 5 10 15 20 25

Functionality

Navigation

Addition of new contentareas/topic types

Search

Overall look and feel

Home page redesign

Subpage redesign

Not applicable

Have not used ACSI to planchanges

Site performance

Other


Number of sites

Types of Site Improvements Planned Using ACSI Data


55% (n=28)6%

(n=3) 25% (n=13) 14% (n=7)

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Yes No Not sure Not applicable


Percentage of sites

Teams’ Plans to Use ACSI Data for Next Redesign


0.86 (n=15)

0.43 (n=14)

0.79 (n=21)

-0.12 (n=21)

0.20 (n=22)

-0.41 (n=15)

-0.60

-0.40

-0.20

0.00

0.20

0.40

0.60

0.80

1.00

Mea

n s

core

ch

ang

e

Sites that used ACSI in the activity Sites that did not do the activity

Method - ACSI score sata (n=42) & User survey (n=51)

Implemented site changes Evaluated effects of pre- vs. post-launch site changes

Continuous improvement

Mean Satisfaction Score Changes Based on Use of ACSI


6 (12%)

8 (16%)

9 (18%)

13 (25%)

14 (27%)

24 (47%)

0 5 10 15 20 25 30

Staff time constraints

Financial resource constraints

Not applicable

None

Insufficient [calendar] time

Other

Number of sites


Barriers to Making Changes to Site


3 (10%)

18 (57%)

4 (13%)

7 (22%)

10 (53%)

9 (48%)

0 (0%)

0 (0%)

0 2 4 6 8 10 12 14 16 18 20

Yes, on a continuous annualbasis

Yes, on a periodic/as-needed basis

Not sure

No, ACSI is not of sufficientvalue to continue use


Initial 32 sites Next 19 sites

Number of sites

Use of ACSI if IC/ODO Had to Pay License Fee


19

Results -- Objective 3:Benefits of Trans-NIH ACSI Use

Focus on customer satisfaction measurement

Focus on user groups and user-centered design

Identification and resolution of common issues

Network of NIH web site professionals Informal mentoring Identification of NIH contexts that are best

suited for ACSI


20

NIH Site Characteristics and the ACSI

Associated with successful use: Timing license period with redesign cycle Committed resources Supportive IC management

Associated with issues/difficulties: Intranet Low traffic volume Manual page coding required Skeptical attitude within IC Niche sites


21

Results -- Objective 4: Contribution of Web Sites to NIH Goals

NIH sites serve diverse information needs of multiple audiences

NIH Customer Satisfaction Index consistently high (above e-Government Index)


79

77.1

75.1

73.5

75.2

76.1 76.1

73.5

72.172.6

73.9

71.9

70

72

74

76

78

80

Q4 2004(NIH n=8)

Q1 2005(NIH n=23)

Q2 2005(NIH n=23)

Q3 2005(NIH n=34)

Q4 2005(NIH n=38)

Q1 2006(NIH n=42)

Quarter

Mea

n s

atis

fact

ion

sc

ore

NIH satisfaction E-Gov satisfaction

Comparison of NIH and e-Government Satisfaction Indexes


23

Recommendations: Trans-NIH Level

Optimize support for the network of NIH ACSI users Information sharing Resolution of common web site issues Pooling of evaluation resources Development of best practices guide

Establish NIH ACSI User’s Group Conduct case studies to test new ACSI

products Continue persistent cookie policy


24

Recommendations: Web Site Level

For continuing teams Network with other ACSI users Set realistic schedule and goals for use

For teams considering ACSI use Evaluate potential for successful use of ACSI in terms

of: Commitment to evaluation Resources (staff, budget) Coordination of license with site maintenance/revision

cycle Site characteristics


25

Questions?

Documents

Evaluation of NIH ACSI Website Project: Highlights of the Final Report