Development of a Self-Assessment Method

for Patients to Evaluate Internet-based Health

Information

AMIA Fall 1999 Josette Jones

Josette Jones, RNc, Licentiate MIS, Licentiate Nursing, Doctoral Student School of Nursing

University of Wisconsin - Madison

Academic Advisor: Patricia Flatley Brennan, RN, Ph.D., FAAN, FACMI

AMIA Fall 1999 Josette Jones

Purpose of the study• Development of a Self-Assessment

Method for patients to evaluate quality and relevance of health care oriented websites

• Validation of the Stability and Consistency the Self-Assessment Method for website evaluation

• the use of the tool • the conceptualization

AMIA Fall 1999 Josette Jones

Background and Significance

• The Internet hosts a large number of accurate health-oriented websites with endless opportunities to inform, teach and connect patients.

• Health care consumers need a way to judge the quality and relevance of the information provided on the Internet.

AMIA Fall 1999 Josette Jones

Background and Significance

• Quality as perceived by the “general” users relates not only to accuracy of content but rather to presentation, perceived trust, clear credentials, and other markers that tend to give information 'weight'.

• Patients will accept or use a WWW page when the information is perceived as relevant to his/her situation.

AMIA Fall 1999 Josette Jones

Helping patients determine the Helping patients determine the quality and relevance of health quality and relevance of health information found on the Internet is information found on the Internet is a key responsibility for clinicians a key responsibility for clinicians who want to use network who want to use network technologies to promote the health technologies to promote the health of patients and provide them with of patients and provide them with clinical service.clinical service.

Development of a Self-Assessment Method for Evaluation of Websites

AMIA Fall 1999 Josette Jones

Study Design

• Summary of criteria/guidelines for evaluating WebPages, published in journal articles and on-line publications

• Categorization of the criteria through lexical and contextual analysis

AMIA Fall 1999 Josette Jones

Study Design

• Comparison to the criteria to what the “general” user of the WWW perceives as quality and/or relevance.

• Conceptualization of four criteria considered as indicative of quality and relevance as perceived by the “general” user of information on the Internet.

AMIA Fall 1999 Josette Jones

“Quality and Relevance” Criteria

Content• Topicality • Depth • Accuracy • Quantity• Clarity

Design• Attractiveness• Audience appeal• Writing style• Layout• Graphics &

multimedia

Communication• Readability • Understandable• Easily navigated

Credibility• Assesses author’s

authority and qualifications

• Information currency & functional feedback

AMIA Fall 1999 Josette Jones

The proposed Self-Assessment The proposed Self-Assessment Method Method

for Evaluating Websitesfor Evaluating Websites consists consists of nine unrelated questionsof nine unrelated questions

AMIA Fall 1999 Josette Jones

1. Match of interest

2. Clear and sufficient information

3. Novel info

4. Compelling case

5. Support from graphics and pictures

6. Ease of reading and understanding

7. Up-to-date information

8. Familiarity with publisher

9. Facilitate behavior change

Prompts to Patients

AMIA Fall 1999 Josette Jones

AMIA Fall 1999 Josette Jones

Validation of the Stability and

Consistency the Self-Assessment Method for

Website Evaluation

AMIA Fall 1999 Josette Jones

Sample

• 16 students from a small Midwestern university participated in the testing of the method– Female, Caucasian– Average age: 22 year– Medical/clinical knowledge: none to basic– Internet skills varying from none to good

AMIA Fall 1999 Josette Jones

Procedure

• Participants were asked to list 3 topics related to health and to search the web for information on these topics

• The test was repeated after 3 weeks

AMIA Fall 1999 Josette Jones

• Identified a web site

• Answered each item on the questionnaire with “yes” or “no”.

Testing the Use of the Self-Assessment Method

Summative scores for each answerper item on the questionnaire

 Time 1 (N=48)

Time 2 (N=48)**

Yes No Und Yes

No Und

Match of interest 41 6 1 40 8 0

Clear and sufficient info 33 12 3 37 11 0

Novel info 38 9 1 42 5 1

Compelling case 40 6 1 36 9 2

Support from graphics & pictures

14 13 19 20 14 11

Ease of reading & understanding

39 6 3 44 3 1

Up-to-date info 47 1 0 43 4 1

Familiarity with publisher 11 36 1 12 36 0

Facilitate behavior change 25 14 9 24 18 6

AMIA Fall 1999 Josette Jones

Validating the Conceptualized Criteria

Each of the four criteria were validated for “quality and relevance” using a 4-point Likert scale.

1=not relevant2=somewhat relevant3=quite relevant4=very relevant

AMIA Fall 1999 Josette Jones

“Quality and Relevance” Criteria

Content• Topicality • Depth • Accuracy • Quantity• Clarity

Design• Attractiveness• Audience appeal• Writing style• Layout• Graphics &

multimedia

Communication• Readability • Understandable• Easily navigated

Credibility• Assesses author’s

authority and qualifications

• Information currency & functional feedback

AMIA Fall 1999 Josette Jones

Validation of the criteria indicative of quality and relevance: modal score

(frequency)Time 1N=16

Time 2 N=16

Overall N=32

Credibility 4 (63%) 4 (63%) 4 (63%)

Communication

4 (38%) 4 (50%) 4 (44%)

Content 3 (50%) 3 (50%) 3 (50%)

Design 2 (50%) 3 (44%) 2 (41%)

AMIA Fall 1999 Josette Jones

Matching Perceptions of “Quality and Relevance” to

Questionnaire

Each questionnaire item was validated as useful to determine the “quality and relevance” using a 4-point Likert scale.

1=not relevant2=somewhat relevant3=quite relevant4=very relevant

Perception of Quality and Relevance: Ranked by Modal Score - Frequency

N=32 Modal Score

Frequency %

1. Match of interest 4 63

6. Ease of reading &understanding

4 63

2. Clear and sufficient info 4 53

7. Up-to-date info 4 44

3. Novel info 3 63

4. Compelling case 3 50

8. Familiarity with publisher 3 34

9. Facilitate behavior change 2 41

5. Support from graphics & pictures

2 38

AMIA Fall 1999 Josette Jones

DiscussionThe study suggests that:

• Consumers evaluate quality and information against a simple set of indirect criteria

• Design (Q5) and facilitation of behavior change (Q9) are rated less helpful in evaluating the quality and relevance of health related websites than the other questions.

AMIA Fall 1999 Josette Jones

Limitations of the study

• Limited size of the convenience sample • Demographics of participants• No data available from patient sample.

• No elicitation of unrecognized needs• No measurement of the number of steps

it may take to locate the information

AMIA Fall 1999 Josette Jones

Acknowledgments

• Dr. Patricia Flatley Brennan• The HeartCare team in Madison

and Cleveland• University of Wisconsin, Eau Claire• Dr. M. Oleson - and -

Gyda Bjornsdottir RN, MSN

AMIA Fall 1999 Josette Jones

Josette Jones, RNC, Licentiate Nursing, Licentiate MISwouterjf@uwec.edu

Academic Advisor: Patricia Flatley Brennan, RN, Ph.D., FAAN, FACMI

This study was supported by the HeartCare ProjectThis study was supported by the HeartCare Project

funded through NIH grant R01-LM06249funded through NIH grant R01-LM06249