This article was downloaded by: [Ohio State University Libraries]On: 06 June 2013, At: 12:00Publisher: Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Journal of Health Communication:International PerspectivesPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/uhcm20

Measuring Health Literacy: A Pilot Studyof a New Skills-Based InstrumentLauren McCormack a , Carla Bann a , Linda Squiers a , Nancy D.Berkman a , Claudia Squire a , Dean Schillinger b , Janet Ohene-Frempong c & Judith Hibbard da RTI International, Research Triangle Park, North Carolina, USAb University of California at San Francisco Center for VulnerablePopulations, San Francisco General Hospital, San Francisco,California, USAc J O Frempong & Associates, Inc., Elkins Park, Pennsylvania, USAd Institute for Policy Research and Innovation, University of Oregon,Eugene, Oregon, USAPublished online: 15 Sep 2010.

To cite this article: Lauren McCormack , Carla Bann , Linda Squiers , Nancy D. Berkman , ClaudiaSquire , Dean Schillinger , Janet Ohene-Frempong & Judith Hibbard (2010): Measuring HealthLiteracy: A Pilot Study of a New Skills-Based Instrument, Journal of Health Communication:International Perspectives, 15:S2, 51-71

To link to this article: http://dx.doi.org/10.1080/10810730.2010.499987

PLEASE SCROLL DOWN FOR ARTICLE

Full terms and conditions of use: http://www.tandfonline.com/page/terms-and-conditions

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden.

The publisher does not give any warranty express or implied or make any representationthat the contents will be complete or accurate or up to date. The accuracy of anyinstructions, formulae, and drug doses should be independently verified with primarysources. The publisher shall not be liable for any loss, actions, claims, proceedings,demand, or costs or damages whatsoever or howsoever caused arising directly orindirectly in connection with or arising out of the use of this material.

Articles

Measuring Health Literacy: A Pilot Studyof a New Skills-Based Instrument

LAUREN MCCORMACK, CARLA BANN,LINDA SQUIERS, NANCY D. BERKMAN, ANDCLAUDIA SQUIRE

RTI International, Research Triangle Park, North Carolina, USA

DEAN SCHILLINGER

University of California at San Francisco Center for VulnerablePopulations, San Francisco General Hospital, San Francisco,California, USA

JANET OHENE-FREMPONG

J O Frempong & Associates, Inc., Elkins Park, Pennsylvania, USA

JUDITH HIBBARD

Institute for Policy Research and Innovation, University of Oregon,Eugene, Oregon, USA

Although a number of instruments have been used to measure health literacy, a keylimitation of the leading instruments is that they only measure reading ability orprint literacy and, to a limited extent, numeracy. Consequently, the present studyaimed to develop a new instrument to measure an individual’s health literacy usinga more comprehensive and skills-based approach. First, we identified a set of skillsto demonstrate and tasks to perform. Next, we selected real-world health-relatedstimuli to enable measurement of these skills, and then we developed survey items.After a series of cognitive interviews, the survey items were revised, developed intoa 38-item instrument, and pilot tested using a Web-based panel. Based on thepsychometric properties, we removed items that did not perform as well, resulting

The authors wish to thank our expert panel members for their contribution to this project.We would also like to thank Rebecca Moultrie and Tania Fitzgerald for their assistance withthe project and Shelton Jones for statistical support. This study was supported by NationalCancer Institute grant R01 CA115861-01A2. The views expressed herein are solely those ofthe authors and do not necessarily represent the views of the National Cancer Institute.

Address correspondence to Lauren McCormack, RTI International, 3040 CornwallisRoad, P.O. Box 12194, Research Triangle Park, NC 27709, USA. E-mail: Lmac@rti.org

Journal of Health Communication, 15:51–71, 2010Copyright # Taylor & Francis Group, LLCISSN: 1081-0730 print=1087-0415 onlineDOI: 10.1080/10810730.2010.499987

51

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

in a 25-item instrument named the Health Literacy Skills Instrument. Based on con-firmatory factor analysis, the items were grouped into five subscales representingprose, document, quantitative, oral, and Internet-based information seeking skills.Construct validity was supported by correlations with the short form of the Test ofFunctional Health Literacy in Adults and self-reported skills. The overall instrumentdemonstrated good internal consistency, with a Cronbach’s alpha of 0.86. Additionalanalyses are planned, with the goal of creating a short form of the instrument.

A number of instruments have been used to measure health literacy. Two of the mostcommonly used instruments include the Rapid Estimate of Adult Literacy inMedicine (REALM) (Davis et al., 1991) and the Test of Functional Health Literacyin Adults (TOFHLA) (Parker, Baker, Williams, & Nurss, 1995). However, animportant limitation of these instruments is that they measure reading ability orprint literacy, and in the case of the TOFHLA, numeracy; they do not reflect acomprehensive assessment of health literacy (Berkman et al., 2004; IOM, 2009).Instruments also exist that attempt to screen patient health-literacy level in clinicalsettings (e.g., the Newest Vital Sign; Weiss et al., 2005), measure provider-levelfacilitation of health literacy (e.g., Consumer Assessment of Health Providers andSystems [CAHPS] Item Set for Addressing Health Literacy; Agency for HealthcareResearch and Quality, n.d.), and assess health literacy using sociodemographic andgeographic data elements (Paasche-Orlow, Schillinger, Greene, & Wagner, 2006).

The Department of Education’s 2003 National Assessment of Adult Literacy(NAAL) Survey is the only national assessment of literacy that includes some healthliteracy tasks (Kutner, Greenberg, Jin, & Paulsen, 2006). Of the 28 health literacytasks on the NAAL, 3 represented a clinical domain, 14 represented a preventiondomain, and 11 items represented navigation of the health care system. The NAALyields estimates of the distribution of levels of health literacy for various populationgroups. Though it overcomes some of the limitations of other measures, including afocus on assessing skills other than reading, the NAAL has been criticized for its lackof availability, lack of transparency, and challenges in using it (Weiss, 2009).

In a landmark report, Health Literacy: A Prescription to End Confusion, theInstitute of Medicine (IOM) recommended the development of new health literacymeasures (Nielson-Bohlman, Panzer, & Kindig, 2004). The IOM’s Recommendation2–2 calls for the development, testing, and use of culturally appropriate measures ofhealth literacy and to field them as part of large ongoing population surveys. Parker& Kindig (2006) echoed the call for new measures by stating that ‘‘new measuresof health literacy must be developed and evaluated’’ (p. 891), and the authorsconcluded that ‘‘while progress [in the area of health literacy] is being made, thescope is not broad enough and the pace is not fast enough’’ (p. 891). Baker (2006)also supported the need for more comprehensive measures.

This study aimed to advance the field by developing a more comprehensive measureof health literacy that is publicly available. Similar to other measures, this instrument,titled the Health Literacy Skills Instrument (HLSI), measures print literacy. However,it is innovative in that it also uses non-print stimuli and examines oral and Internet-basedinformation seeking skills. In terms of content domains, it reflects health related issuesacross the life course for health promotion and disease prevention, health care mainte-nance and treatment, and health system navigation. The HLSI can be self-administeredvia a computer, which can reduce data collection costs and minimize potential dis-comfort or embarrassment among participants. TheHLSI is designed to be used in inter-vention research studies as well as for large scale surveillance.

52 L. McCormack et al.

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

This article describes the development of this novel instrument to measure indi-vidual level health literacy using a skills-based approach. We explain the instrumentdevelopment and testing process and provide the results from a pilot test of theinstrument, including how individual items performed as well as validation data.We also discuss policy implications, make recommendations for next steps, andaddress measurement considerations.

Instrument Development

Our core instrument development team included individuals with backgrounds inhealth communication, health services research, psychometrics, literacy, health liter-acy, plain language, cultural competency, and clinical medicine. We used Ratzan andParker’s (2000) definition of health literacy, with minor modifications to the word-ing: The degree to which individuals can obtain, process, understand, and communicateabout health-related information needed to make informed health decisions. A group ofeight health literacy expert panel members provided oversight and guidance,especially in the early developmental phases.

To develop the instrument, we adopted a hierarchical process. First, we identifiedskills to demonstrate and tasks to perform (see Table 1). Tasks were categorized intothree skills set areas: print, oral, and Internet-based information seeking. We classi-fied tasks as follows: identifying and understanding health-related text; interpretinginformation and=or data in the form of tables, charts, pictures, symbols, maps, andvideos; completing computations; making inferences based on information presentedor applying given information to a specific scenario; and information seeking andinteractions on Internet websites.

Second, we selected health-related stimuli to enable an assessment of those skillsand tasks. We sought diversity in the stimuli across three major health domainsreflecting various points in the life course, during periods of health and illness: healthpromotion and disease prevention, heath care maintenance and treatment, andhealth system navigation (see Table 1). To the extent possible, we used the followinginclusion criteria for stimuli selection: (a) relevant to the health of a large segment ofthe public (i.e., gender neutral; not specific to subgroups); (b) culturally sensitive; (c)clinically relevant and less controversial health topics; and (d) both public andprivate-sector materials in a variety of formats=channels, including print documents(e.g., brochure, newspaper article, fact sheet), telephone recording, video clip, andWeb site. We chose stimuli that would take most participants about 1 minute to viewor read. We used real-world stimuli, with a limited number of plain-language materials.Copyright approval or user permission was obtained for private-sector materials.

Third, we developed survey items. We attempted to map each survey item to askill set area as well as a task. In some cases, a survey item could arguably fit intomore than one skill and=or task. The initial version of the instrument included 38items, with 1 to 2 survey items for each stimulus (example stimuli and survey itemsare shown in the Appendix). The survey items did not require prior or outside knowl-edge, that is, each question could be answered based only on the information inthe stimulus. We sought to include items with varying levels of difficulty that werenot contingent upon each other. To score an instrument, it is necessary to have asingle correct response option and multiple incorrect response options or plausible‘‘distracters.’’ We included three to four response options for most questions, witha ‘‘don’t know’’ option.

Measuring Health Literacy 53

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

Table

1.Healthliteracy

skillareabytask

andbyhealthdomain

(25-item

instrument)

Skillsandstim

uliform

at

Task

Print(prose,document,

orquantitative)

Oral(listeningoraudiovisual)�

Internet-based

inform

ationseeking

1)Identifyingand

understanding

health-relatedtext

Prose

1)Whichofthefollowingisprobably

notasecond-degreeburn?(b)

2)Whichofthefollowingisasymp-

tom

oflactose

intolerance?(b)

3)Whichofthefollowingisnotasign

ofastroke?

(b)

2)Interpretinginform

ation

and=ordata

intheform

oftables,charts,

pictures,

symbols,maps,

andvideos

Prose

4)Whichsetoflow

density

lipoprotein

(LDL)andhigh

density

lipoprotein

(HDL)

levelsisbest?

(a)

Document

5)Whichofthefollowing

problemscould

becaused

bythismedicine?

(b)

6)Whichofthefollowingentrancesis

closest

totheelevator?

(c)

7)In

theexample

listed

inthefirst

row

ofthetable,when

should

the

medicinebetaken?(b)

18)Whatdothemusclesin

the

throattypicallydowhen

apersonissleeping?(b)

19)Whatpart

ofthebodydolunge

exerciseswork?(a)

54

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

Document

8)How

much

willtheinsurance

companypayforthephysical

therapyreceived

on7=22=09?(c)

9)Howmuch

does

thepatienthaveto

payforthelaboratory

services

received

on7=15=09?(c)

3)Completing

computations

Quantitative

10)More

men

diefrom

prostate

cancer

thanfrom

other

causes.Would

you

saythisistrue,false,orare

younot

sure?(a)

11)Whoismore

likelyto

die

ofpros-

tate

cancer?

(a)

12)How

manygramsoffiber

are

intw

oservings?

(a)

13)If

apersonisona2,500calorie

diet,whatpercentofthedaily

valueofsaturatedfatwould

heget

from

oneserving?(a)

4)Makinginferencesbased

ontheinform

ation

presentedorapplying

inform

ationto

aspecific

scenario

Prose

14)If

apersonisathighrisk

forheart

disease,whichofthefollowing

levelsoflow

density

lipoprotein

(LDL)cholesterolisbest?

(a)

20)If

apersonwasworriedabout

hiscough,whatnumber

should

hepress?(c)

21)If

apersonwantedto

checkon

thedate

andtimeofan

appointm

entshealreadymade,

(Continued

)

55

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

Table

1.Continued

Skillsandstim

uliform

at

Task

Print(prose,document,

orquantitative)

Oral(listeningoraudiovisual)�

Internet-based

inform

ationseeking

Document

15)If

Joewasvisitingsomeonein

Room

130andwantedto

goto

the

cafeteria,whichoftheseplaces

would

hepass

ifhetookthe

shortestroute?(c)

16)A

personismakingasaladand

wants

toaddoneservingof

chopped,uncooked

carrots.

How

much

should

sheuse?(a)

17)A

personiscookingdinner

for

him

selfandwants

toincludeone

servingfrom

themeatand

beansgroup.Whatshould

he

choose?(a)

whatnumber

should

she

press?(c)

5)UtilizingtheInternet=

computerto

obtain

healthinform

ation

22)Johnweighs200

poundsandhe

walked

atamedium

pace

onafirm

surface

for30minutes.How

manycalories

did

he

burn?(a)

56

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

23)Kate

weighs150

pounds.Which

activitywould

burn

themost

calories?(a)

24)Whatdoes

this

(heart

attack

risk)

tooldo?(a)

25)Johnis39years

old

andsm

okes.His

bloodpressure

is130=90andhe’son

bloodpressure

medicine.

HisHDL

cholesterolis50and

histotalcholesterol

is230.Whatishis

estimated10-year

risk

ofheart

attack?(a)

� Wewerenotable

tomeasure

speakingskillsin

thisinstrument.

The2003versionoftheNAALputhealthtasksinto

prose,document,orquantitativescalesbasedonthefollowingdefinitions,whichweattem

ptedto

adhereto:Theprose

literacy

scalemeasuredtheknowledgeandskillsneeded

tosearch,comprehend,anduse

inform

ationfrom

textsthatwereorganized

insentencesorparagraphs.Thedocumentliteracy

scale

measuredtheknowledgeandskills

needed

tosearch,comprehend,anduse

inform

ationfrom

non-continuoustexts

invariousform

ats.Thequantitative

scale

measuredtheknowledgeandskillsneeded

toidentify

andperform

computationsusing

numbersem

bedded

inprintedmaterials(K

utner

etal.,2006).

Key:(a)¼Healthpromotion=disease

prevention.(b)¼Healthcare

maintenance

andtreatm

ent.(c)¼Healthcare

system

navigation.

57

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

Finally, we chose the mode of survey administration. To effectively view all ofthe stimuli and demonstrate the skills, we determined that the survey needed to beadministered using a computer through either Web-based or in-person datacollection.

Nine in-person cognitive interviews were conducted using a laptop computerand with an interviewer present. Five males and four females participated rangingin age from 21 to 78 years (mean age is 42 years). Five participants had collegedegrees and four participants had less than this. Respondents completed the surveyusing the keyboard and mouse after viewing each stimulus on the screen. Despite anintroduction at the beginning of each question instructing respondents to base theiranswers only on the information in the stimulus, some respondents tried to use priorknowledge or logic to answer questions. For example, respondents who were fam-iliar with a given health topic sometimes answered questions based on their ownexperiences. In other cases, respondents used logic or common sense to answer aquestion. Respondents tended to have more difficulty answering a question thatrequired making an inference, and when the wording in the survey question wasnot exactly the same as the wording in the stimulus. We revised the 38 items basedon the cognitive testing results for use in the next phase of the study, which includedpilot data collection.

Data Collection Methods

Sample and Setting

We pilot tested the instrument using KnowledgePanel1, created by KnowledgeNetworks, an online Non-Volunteer Access Panel. Potential panel members are cho-sen via a statistically valid sampling method and using known published samplingframes that cover 99% of the U.S. population. Address-Based Sampling (ABS),which is based on the U.S. Postal Service Delivery Sequence File, was used to selecta probability sample of all U.S. households. This sample also comprises cell-phonehouseholds as well as non-Internet households. ABS is one of the most innovativemeans of obtaining nationally representative samples at minimum cost. Samplednon-Internet households are provided a laptop computer and free Internet service.KnowledgePanel consists of about 50,000 U.S. residents, aged 18 years or older,including persons of Hispanic origin that were selected probabilistically (for moreinformation about the panel, see http://www.knowledgenetworks.com/knpanel/index.html). Between October 7, 2009, and November 19, 2009, a total of 2,212Knowledge Network panelists aged 18 or over were invited to participate in thesurvey. Respondents received $15 for completing the survey.

Other Measures

In addition to the 38 health-literacy items, we also administered the short-form ofthe Test of Functional Health Literacy in Adults (S-TOFHLA) (Wallace, 2006).The TOFHLA and S-TOFHLA are timed reading comprehension tests that use themodified Cloze procedure, in which every 5th to 7th word in a passage is omittedand replaced with a blank space. The patient must select a word to fit into the blankspaces from the four multiple-choice options provided for each space. Baker,Williams, Parker, Gazmararian, and Nurss (1999) reported Cronbach’s alpha of

58 L. McCormack et al.

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

0.68 for numeracy and 0.97 for the reading comprehension items of the S-TOFHLA.The overall correlation between the S-TOFHLA and the REALM was 0.80.

For validation analyses, we also asked participants to self-report their perform-ance on the kind of skills being assessed in the survey. Specifically, we asked howeasy or difficult it is to remember information they read versus hear; how easy ordifficult it is to understand information they read versus hear; and how easy or dif-ficult it is to explain a health issue to their doctor, find health information they need,and locate health information on the Web. Responses for each of the seven itemsincluded very difficult=difficult=somewhat easy=very easy. Socio-demographic char-acteristics and selected health-related data on respondents were available fromKnowledge Networks.

Hypotheses

We hypothesized that participants with higher education levels and those who reportedless difficulty with skills related to health literacywould have higher scores on the instru-ment. Given the different focus of the S-TOFLHA and this new measure (e.g., readingcomprehension versus skills-based), we hypothesized that the two instruments would bemoderately correlated, with the highest correlations for the print-based skill sets(print-prose and print-document), which involve more reading.

Statistical Analysis

We assessed several psychometric properties of the health-literacy items, includingthe percentage of correct responses and the correlation between each item and thetotal score, excluding the item of interest. We also estimated two-parameter logisticitem response theory (IRT) models, using the Multilog software program (Thissen,Chen, & Bock, 2003). These models estimate a discrimination and threshold para-meter for each item. The IRT discrimination parameter indicates how well the itemdistinguishes between persons with high versus low health literacy, as estimatedbased on the other items included on the scale. Ideally, items should have discrimi-nation parameters of 1.0 or higher. The threshold indicates the difficulty of the item;items with larger thresholds are more difficult. We conducted higher order confirma-tory factor analyses using the Mplus software program (Muthen & Muthen,1998–2007) to assess whether the items clustered into five first-order factors basedon skill set areas (print-prose, print-document, print-quantitative, oral, andInternet-based information seeking), as well as a single overarching second-orderfactor representing health literacy. We examined several fit indices, including thecomparative fit index (CFI), Tucker-Lewis fit index (TLI), and root mean squareerror of approximation (RMSEA), to determine the most appropriate factor model.Sampling strata and weights were incorporated into the analyses to account for thesurvey design.

Based on the psychometric properties, we reduced the item pool by removingitems that performed less well. Specifically, items were candidates for removal if theyhad low item-total correlations, factor loadings, or IRT discrimination parametersand=or very high percentages correct, suggesting poor discrimination. Additionally,we reviewed the content of items with similar threshold parameters to determine ifthey were redundant and should be removed from the scale. Balancing the statistical

Measuring Health Literacy 59

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

results with content validity considerations, we developed the final 25-item scale andcomputed scale scores as the percentage of items answered correctly.

To assess the construct validity of the scale, we conducted a series of analysis ofvariance (ANOVA) procedures to compare health-literacy scores by socio-demographic characteristics and participants’ reports of their self-reported difficultywith various health-literacy-related skills. The latter items included a 4-pointresponse scale ranging from very easy to very difficult, which was collapsed into easyversus difficult due to skewness of the response distribution. Further, we calculatedthe correlations between participants’ scores on each of the four skill-set areas andthe S-TOFHLA. Because no instruments have been validated for computer-basedadministration, few options were available from which to choose.

Finally, we investigated possible cut-points for classifying participants intothree categories based on their health literacy levels: proficient, basic, and belowbasic. We conducted a series of receiver operating characteristic (ROC) analysesto determine which cut-points optimally distinguish participants based on theirself-reported difficulty with understanding information they read and their highestlevel of educational achievement. For example, to determine the cut-point todistinguish proficient versus basic literacy, we compared persons who reported thatunderstanding information they read is very easy with persons who reported thatthey find this task is not very easy (i.e., persons who respond somewhat easy, some-what difficult, or very difficult). For each analysis, the cut-point was selected basedon the point that maximizes the sum of sensitivity and specificity for distinguishingthe two groups.

Results

Data Collection Results

Of the 2212 individuals sampled, 1559 panelists responded to the survey, for acompletion rate of 71%. Among the respondents, 889 (57%) were able to see the testvideo and consented to complete the main survey. Because not all panelists agreed tocomplete the survey or to see the test video, the potential exists that survey estimatescould be biased due to non-response and non-coverage, if not corrected. Therefore,the design-based sampling weights of study participants were adjusted upward tocompensate for persons who failed to respond at both response levels, using aresponse propensity model-based approach. It took participants 45 minutes, on aver-age, to complete the entire survey, which included the 38-item health literacy items,seven questions to self-report on one’s health literacy skills, and the approximate7-minute S-TOFHLA.

Participant Characteristics

The respondents were equally distributed across four age categories and about halfwere female (see Table 2). Respondents were categorized into three education-levelgroups: more than a high school education (n¼ 316); a high school education(n¼ 295); and less than a high school education (n¼ 278). Among the 278 respondentsin the lowest education-level group, one-fourth (n¼ 77) had an 8th grade or lower edu-cation level. About two thirds of respondents were White, 13% were Black, 17% wereHispanic, and 6% classified themselves in to the ‘‘other’’ race=ethnicity category.

60 L. McCormack et al.

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

Table 2. Mean health literacy scores by participant characteristics and self-reportedskills

Health literacy data

Participant characteristic N Weighted % Mean SE p

GenderMale 458 48 70.3 1.5 .469Female 431 52 68.8 1.5 REF

Age (years)18–29 180 22 68.7 2.1 .61730–44 205 25 70.7 2.5 .27145–59 255 27 71.3 1.8 .14360þ 249 26 67.2 1.6 REF

EducationLess than high school 278 31 58.8 1.9 <.001High school graduate 295 33 68.1 1.8 <.001More than high school 316 36 80.3 1.2 REF

Race=ethnicityNon-Hispanic White 664 64 73.8 1.2 .038Non-Hispanic Black 83 13 55.9 3.0 .095Hispanic 80 17 65.1 2.9 .951Other 62 6 64.8 3.8 REF

Marital statusMarried 489 46 73.5 1.3 <.001Not married 400 54 66.1 1.7 REF

Employment statusEmployed 470 51 73.8 1.5 .048Retired 161 15 67.8 1.8 .845Disabled 81 11 54.5 2.4 .002Unemployed 177 23 68.4 2.1 REF

Geographic regionNortheast 161 18 66.9 2.5 .263Midwest 206 22 69.5 2.6 .742South 338 38 70.1 1.7 .838West 184 22 70.7 2.2 REF

Self-reported skillsI am good at mathAgree 429 46 76.6 1.4 <.001Disagree 450 54 64.0 1.5 REF

Remembering information I readEasy 565 65 73.3 1.3 <.001Difficult 317 35 63.4 1.7 REF

Remembering information I hearEasy 584 66 72.5 1.2 .001Difficult 295 34 64.5 1.8 REF

(Continued )

Measuring Health Literacy 61

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

Reliability and Validity of the Instrument

The health-literacy scale demonstrated good internal consistency, with a Cron-bach’s alpha of 0.86. The psychometric properties of the individual items areshown in Table 3. The percentage of correct responses ranged from 24% to91%. The higher order confirmatory factor analysis model fit well (CFI¼ 0.95,TLI¼ 0.98, and RMSEA¼ 0.03). The factor loadings for the four skill-set area fac-tors are shown in Table 2; the loadings of these four factors on the overall healthliteracy factor are as follows: print-prose (0.98), print-document (0.98),print-quantitative (0.95), oral (0.85), and Internet (0.81). Almost all items had fac-tor loadings and item-total correlations of 0.40 or higher and IRT discriminationparameters of 1.00 or higher, indicating good discrimination. One exception is item13, which differs somewhat from the other items in that it requires respondents toperform a mathematical calculation. It is also by far the most difficult item, withonly 24% of respondents answering it correctly. This item was retained to ensurecontent validity of the scale by measuring quantitative skills, which are a compo-nent of health literacy.

On average, respondents answered 70% of the items correctly. The compari-sons by sociodemographic characteristics indicated that respondents with highereducation levels, who were non-Hispanic White, who were married and employedhad higher health-literacy scores relative to their counterparts; whereas respondentswho were disabled had lower scores than respondents who were unemployed.Across all of the self-reported skill items, respondents who reported less difficultywith the skill had higher scores on the health-literacy scale. Respondents tended to

Table 2. Continued

Health literacy data

Participant characteristic N Weighted % Mean SE p

Understanding information I readEasy 665 73 75.1 1.0 <.001Difficult 211 27 56.7 2.2 REF

Understanding information I hearEasy 691 77 73.3 1.1 <.001Difficult 189 23 58.4 2.3 REF

Explaining a health issue to mydoctorEasy 682 74 72.1 1.2 <.001Difficult 196 26 63.3 2.3 REF

Locating health information on theInternetEasy 654 72 73.6 1.1 <.001Difficult 222 28 60.6 2.3 REF

Finding health information I needEasy 645 72 73.3 1.1 <.001Difficult 231 28 61.7 2.4 REF

Note: REF¼ reference category.

62 L. McCormack et al.

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

Table 3. Psychometric properties of health literacy items

Content area=stimulus Item

%correct

Item-totalcorrelation

Factorloading

IRTslope

IRTthreshold

Print-prose

CholesterolIf a person is at high risk for

heart disease, which ofthe following levels oflow density lipoprotein(LDL) cholesterol isbest?

76 0.44 0.67 1.51 �1.15

Which set of low densitylipoprotein (LDL) andhigh density lipoprotein(HDL) levels is best?

66 0.49 0.61 1.32 �0.78

BurnsWhich of the following is

probably not asecond-degree burn?

67 0.41 0.59 1.20 �0.73

Lactose intoleranceWhich of the following

is a symptom of lactoseintolerance?

91 0.36 0.65 1.44 �2.17

StrokeWhich of the following is

not a sign of a stroke?90 0.44 0.77 1.92 �1.79

Print-document

Drug side effects labelWhich of the following

problems could be causedby this medicine?

82 0.27 0.48 0.96 �1.86

Hospital mapIf Joe was visiting someone

in room 130 and wantedto go to the cafeteria,which of these placeswould he pass if he tookthe shortest route?

81 0.44 0.68 1.51 �1.50

Which of the followingentrance is closest to theelevator?

80 0.40 0.59 1.22 �1.42

Medicine recordIn the example listed in the

first row of the table,when should themedicine be taken?

59 0.37 0.53 1.00 �0.47

(Continued )

Measuring Health Literacy 63

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

Table 3. Continued

Content area=stimulus Item

%correct

Item-totalcorrelation

Factorloading

IRTslope

IRTthreshold

Portion controlA person is making a salad

and wants to add oneserving of chopped,uncooked carrots. Howmuch should she use?

64 0.38 0.43 0.73 �1.10

A person is cooking dinnerfor himself and he wantsto include one servingfrom the meat and beansgroup. What should hechoose?

75 0.39 0.65 1.48 �1.07

Insurance formHow much will the

insurance company payfor the physical therapyreceived on 7=22=09?

53 0.39 0.56 0.98 �0.34

How much does the patienthave to pay for thelaboratory servicesreceived on 7=15=09?

80 0.44 0.57 1.06 �1.73

Print-quantitative

Nutrition labelHow many grams of fiber

are in two servings?78 0.45 0.72 1.68 �1.22

If a person is on a 2,500calorie diet, what percentof the daily value ofsaturated fat wouldhe get from oneserving?

24 0.13 0.34 0.66 2.15

Prostate cancer graphMore men die from prostate

cancer than from othercauses. Would you saythis is true, false, or areyou not sure?

80 0.46 0.72 1.64 �1.35

Who is more likely to die ofprostate cancer?

82 0.42 0.73 1.55 �1.50

(Continued )

64 L. McCormack et al.

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

Table 3. Continued

Content area=stimulus Item

%correct

Item-totalcorrelation

Factorloading

IRTslope

IRTthreshold

Oral

Telephone recordingIf a person was worried

about his cough, whatnumber should he press?

58 0.37 0.60 0.96 �0.37

If a person wanted to checkon the date and time ofan appointment shealready made, whatnumber should she press?

58 0.38 0.57 0.89 �0.48

Sleep apneaWhat do the muscles in the

throat typically do whena person is sleeping?

86 0.40 0.68 1.26 �1.89

LungesWhat part of the body do

lunge exercises work?89 0.46 0.94 2.26 �1.76

Internet

CaloriesJohn weighs 200 pounds

and he walked at amedium pace on a firmsurface for 30 minutes.How many calories didhe burn?

59 0.52 0.72 1.73 �0.35

Kate weighs 150 pounds.Which activity wouldburn the most calories?

54 0.52 0.71 1.75 �0.16

Heart attack riskWhat does this tool do? 68 0.57 0.87 2.23 �0.71John is 39 years old and

smokes. His bloodpressure is 130=90 andhe’s on blood pressuremedicine. His HDLcholesterol is 50 and histotal cholesterol is 230.What is his estimated 10year risk of a heartattack?

55 0.59 0.90 2.53 �0.25

Note: Factor loadings based on higher-order confirmatory factor analysis with fourfirst-order factors and one second-order factor (CFI¼ 0.95, TLI¼ 0.98, RMSEA¼ 0.03).

Measuring Health Literacy 65

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

perform less well on tasks that required higher level skills, including some math-ematical manipulations and the application of information to a specific scenario.As hypothesized, correlations between the health literacy domains and S-TOFHLAwere highest for the print-prose, print-document, and print-quantitative skill areaswith correlations of 0.47, 0.45, and 0.41, respectively (Table 4). The correlationswere much lower for the Internet and oral literacy domains, which require fewerreading skills (r¼ 0.31 and r¼ 0.27, respectively; Table 4). Together, the resultsof the mean comparisons and correlations support the construct validity of theinstrument.

Finally, the ROC analyses comparing the health-literacy scores to the categoriesfor respondents’ reporting difficulty understanding the information they read sug-gested a cut-point of 82 for distinguishing those who find the task very easy versusall others (sensitivity¼ 0.63, specificity¼ 0.67) and a cut-point of 70 for distinguish-ing among those who find the task easy versus difficult (i.e., very=somewhat easy vs.very=somewhat difficult) (sensitivity¼ 0.71, specificity¼ 0.65). Comparisons byeducation level indicated a cut-point of 82 for differentiating respondents with somecollege education versus no college education (sensitivity¼ 0.61, specificity¼ 0.72)and a cut-point of 74 for differentiating respondents with at least a high schooleducation (i.e., high school or college) versus respondents with less than a highschool education (sensitivity¼ 0.66, specificity¼ 0.68). Based on these analyses, weclassified participants into three groups: proficient literacy (score �82), basic literacy(score of 70–81), and below basic literacy (score <70). In our sample, 40% ofparticipants have proficient literacy, 22% have basic literacy, and 38% have belowbasic literacy.

Discussion

This new health-literacy instrument fills an existing gap in an important area ofmeasurement, demonstrates robust psychometric properties, and is moderatelycorrelated with an existing measure of literacy. Like the NAAL, it reflects a rangeof tasks and skills that adults are likely to face in their daily lives in the context ofthe U.S. health care system. Unlike the NAAL, however, this instrument measuresthe ability to obtain and use health information from print as well as non-printsources, which is more consistent with how people more typically receive their health

Table 4. Correlation of health literacy domains and S-TOFHLA

Scale OverallPrint-prose

Print-document

Print-quantitative Internet Oral

Health LiteracyDomainsOverall 1.00Print-Prose 0.77 1.00Print-Document 0.87 0.61 1.00Print-Quantitative 0.73 0.49 0.57 1.00Internet 0.79 0.51 0.55 0.48 1.00Oral 0.66 0.36 0.50 0.37 0.42 1.00

S-TOFHLA 0.47 0.45 0.41 0.34 0.31 0.24

66 L. McCormack et al.

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

information today. The mean score across the 25-item instrument was 70% correct.This is comparative with the NAAL, which aimed for a 67% probability of doing atask correctly (Kutner et al., 2006).

We advocate for measuring health literacy using a skills-based approach. Withthis as a priority for measurement, we used a computer-based data collectionapproach. This instrument is well suited for self-administered data collection viathe Web and=or in-person data collection. Web-based data collection offers theadvantages of cost and time efficiencies. The generalizability of Web-based data col-lection is increasing. An interviewer could be present with in-person data collection,and if an individual could not complete the survey on the computer, an alternate for-mat could be administered, similar to an approach taken with the NAAL. Othermodes of data collection are possible, but only for administering a subset of theitems.

We identified a range of skills that individuals often need to monitor andmanage their health in periods of health and illness. We included Internetinformation-seeking skills in the assessment, and given that a computer is requiredto complete the survey, some basic computer skills are needed unless an individualhas assistance. Thus, one could argue that a person’s health-literacy skills are contin-gent on or at least related to their computer skills. Being able to navigate searchengines and websites has become increasingly important, as 75% of Americans usethe Internet and, in 2008, 75% of these Internet users looked for health or medicalinformation online (Pew Internet & American Life Project, 2009). The need for theseskills will only increase as health-information technologies infiltrate the modernhealth care system, including the use of personal health records (PHRs).

Other methodological and conceptual issues are also worthy of consideration.The instrument uses real-world stimuli, as opposed to solely plain-language-approved stimuli. Thus, survey items needed to use the terminology consistent withthese stimuli, which drove up the reading level somewhat. We plan to revisit selectedsurvey items following this pilot study and adjust the reading level if possible. Thisraises questions about the influence of materials in the current health care system onhealth literacy and its measurement. If a respondent answers a question incorrectly,does that mean his or her health-literacy skill is deficient, or is the stimulus deficient,or is it a combination of the two? We are currently examining scoring of the items asit relates to the complexity of the stimuli and the difficulty of the survey items. Wealso are exploring the use of fictitious diseases or conditions to better control forprior knowledge. Also, there is merit to considering whether or not the stimuli couldbe changed or updated over time.

There is growing recognition of the need to account for the demands of the pub-lic health and health care systems when measuring health literacy at the individuallevel (Baker, 2006). In addition to increases in system-level demands, the expecta-tions about consumers’ roles and responsibilities as active participants in their healthcare are also expanding (McCormack, Treiman, Peinado, & Alexander, 2009; Olsen,Aisner, & McGinnis, 2007). These expectations include taking proactive steps, suchas obtaining recommended preventive health services, eating a healthy diet and get-ting regular physical activity, recognizing signs of illness and disease, self-managingchronic illnesses, and navigating the health insurance system (Hibbard, 2009). Thesereal-world expectations assume that consumers will use valid and often complexinformation to support these behaviors. This expanded role for consumers may raiseparameters for what constitutes health literacy.

Measuring Health Literacy 67

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

The overall aim of this study was to develop a publicly available skills-basedinstrument to measure health literacy. The HLSI can be used in intervention researchstudies that seek to improve health-literacy levels and for large-scale surveillance.We will continue to analyze the data, looking at the relationships between healthliteracy and health outcomes, and develop a short form of the instrument.

References

Agency for Healthcare Research and Quality. (n.d.). CAHPS item set for addressing healthliteracy. Retrieved January 29, 2010, from http://www.cahps.ahrq.gov/content/products/HL/PROD_HL_Intro.asp?p=1021&s=215

Baker, D. (2006). The meaning and measure of health literacy. Journal of General InternalMedicine, 21, 878–883.

Baker, D., Williams, M., Parker, R., Gazmararian, J., & Nurss, J. (1999). Development of a brieftest to measure functional health literacy. Patient Education and Counseling, 38, 33–42.

Berkman, N. D., Dewalt, D. A., Pignone, M. P., Sheridan, S. L., Lohr, K. N., Lux, L., et al.(2004). Literacy and health outcomes: Summary, evidence report=technology assessment no.87 (Agency for Healthcare Research and Quality Publication No. 04-E007-1). RetrievedApril 10, 2010, from http://www.ahrq.gov/clinic/epcsums/litsum.pdf

Davis, T. C., Crouch, M. A., Long, S. W., Jackson, R. H., Bates, P., George, R. B., et al.(1991). Rapid assessment of literacy levels of adult primary care patients. FamilyMedicine, 23(6), 433–435.

Hibbard, J. H. (n.d.). Consumers in a complex and dynamic health care environment.Retrieved February 4, 2010, from http://www.unitedhealthfoundation.org/download/HibbardPaper.pdf

Institute of Medicine (IOM). (2009). Measures of health literacy: Workshop summary.Washington, DC: The National Academies Press.

Kutner, M., Greenberg, E., Jin, Y., & Paulsen, C. (2006). The health literacy of America’sadults: Results from the 2003 National Assessment of Adult Literacy (NCES 2006–483).Washington, DC: National Center for Education Statistics. Retrieved April 10, 2010,from http://nces.ed.gov/pubs2006/2006483.pdf

McCormack, L., Treiman, K., Peinado, S., & Alexander, J. (2009). Environmental scan ofpatient roles and responsibilities: Recommendations for a communication initiative. Reportprepared for the Institute of Medicine. Research Triangle Park, NC: RTI International.

Muthen, L. K., & Muthen, B. O. (1998–2007). Mplus user’s guide (5th ed). Los Angeles, CA:Muthen & Muthen.

Nielsen-Bohlman, L., Panzer, A. M., & Kindig, D. A. (Eds.). (2004). Health literacy: Aprescription to end confusion. Washington, D.C.: The National Academies Press.

Olsen, L., Aisner, D., & McGinnis, M. (Eds.). (2007). The learning healthcare system: Work-shop summary (IOM roundtable on evidence-based medicine). Washington, DC: TheNational Academies Press.

Paasche-Orlow, M. K., Schillinger, D., Greene, S. M., & Wagner, E. H. (2006). How healthcare systems can begin to address the challenge of limited literacy. Journal of GeneralInternal Medicine, 21(8), 884–887.

Parker, R. M., Baker, D. W., Williams, M. V., & Nurss, J. R. (1995). The test of functionalhealth literacy in adults: A new instrument for measuring patients’ literacy skills. Journalof General Internal Medicine, 10(10), 537–541.

Parker, R. M., & Kindig, D. A. (2006). Beyond the Institute of Medicine health literacy report:Are the recommendations being taken seriously? Journal of General Internal Medicine,21(8), 891–892.

Pew Internet and American Life Project. (2009). Trend data. Retrieved May 25, 2010, fromhttp://www.pewinternet.org/Static-Pages/Data-Tools/Download-Data/Trend-Data.

68 L. McCormack et al.

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

aspx and http://www.pewinternet.org/Static-Pages/Data-Tools/Download-Data/~/media/Infographics/Trend%20Data/January%202009%20updates/Internet%20Diffusion%201%206%2009.jpg

Ratzan, S. C., & Parker, R. M. (2000). Introduction. In C. R. Selden, M. Xorn, S. C. Ratzan,& R. M. Parker (Eds.), National Library of Medicine current bibliographies in medicine:Health literacy. NLM Pub. No. CMB 2000-1. Bethesda, MD: National Institutes ofHealth, U.S. Department of Health and Human Services.

Thissen, D., Chen, W.-H., & Bock, R. D. (2003). MULTILOG 7 for Windows: Multiple-category item analysis and test scoring using item response theory [Computer software].Lincolnwood, IL: Scientific Software International, Inc.

Sudore, R., & Schillinger, D. (2009). Interventions to improve care for patients with limitedhealth literacy. Journal of Clinical Outcomes Management, 16(1), 20–29.

Wallace, L. (2006). Patients’ health literacy skills: The missing demographic variable in pri-mary care research. Annals of Family Medicine, 4, 85–86.

Weiss, B. (2009). NAAL data: To use or not to use. In Measures of health literacy: Workshopsummary. Washington, DC: The National Academies Press.

Weiss, B., Mays, M., Martz, W., Castro, K., Dewalt, D., Pignone, M., et al. (2005). Quickassessment of literacy in primary care: The newest vital sign. American Family Medicine,3, 514–522.

Appendix: Example Stimuli and Survey Items

Expanding Portions

Are you eating a variety of healthy foods, exercising, and still struggling with yourweight? Some people may need to pay closer attention to portion control (managingthe amount of food that they eat) as their total calorie intake determines their weight.

A serving isn’t what they happen to put on their plate. It’s a specific amount offood defined by common measurements, such as cups, ounces, or pieces. The servingsizes represented here are part of the Mayo Clinic Healthy Weight Pyramid—a foodpyramid designed to promote weight loss and long-term health. Use these servingsizes in conjunction with a diet based on a variety of healthy foods. Add theright amount of regular physical activity, and a person will be well on their wayto enjoying good nutrition and controlling their weight.

Vegetables

Until they’re comfortable judging serving sizes, you may need to use measuring cupsand spoons. A half a cup of cooked carrots, for example, equals one serving. Hereare the recommended serving sizes for other vegetables:

Food Serving size

Raw leafy vegetables ¼ 2 cupsRaw vegetables, chopped ¼ 1 cupChopped, cooked or cannedvegetables

¼ 1=2 cup

Measuring Health Literacy 69

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

Meat and Beans

Familiar objects can help a person picture proper portions for meat, poultry, fish,and beans. For example, a 3-ounce serving of fish is about the size of a deck of cards.Here are the serving sizes for meat and meat substitutes:

Food Serving size

Cooked skinlesspoultry or fish

¼ 3 ounces

Cooked lean meat ¼ 1 1=2 ouncesCooked legumes ordried beans

¼ 1=2 cup or about the size ofan ice cream scoop

Egg ¼ 1 medium

A person is making a salad and wants to add one serving of chopped, uncookedcarrots. How much should she use?

Select one answer only

Source: Used with permission from Mayo Foundation for Medical Education and Research.‘‘Mayo,’’ ‘‘Mayo Clinic,’’ ‘‘MayoClinic.com,’’ ‘‘Mayo Clinic Health Information,’’ and thetriple-shield Mayo logo are trademarks of MFMER. All Rights Reserved.

Explanation of Benefits

ABC Insurance Company Plan

Member: John DoePatient: Jane Doe

Datesofservice

Type ofservice Submitted

Notcovered Covered Co-pay

Planliability

Patientresponsibility Note

7=22=09 Physicaltherapy

140.00 0.00 140.00 140.00 0.00 140.00 A

7=15=09 Laboratory 170.00 66.00 104.00 30.00 74.00 30.00 BTotal 310.00 66.00 244.00 170.00 74.00 170.00

How much will the insurance company pay for the physical therapy received on7=22=09?

70 L. McCormack et al.

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013

Select one answer only

How much does the patient have to pay for the laboratory services received on7=15=09?

Select one answer only

Measuring Health Literacy 71

Dow

nloa

ded

by [

Ohi

o St

ate

Uni

vers

ity L

ibra

ries

] at

12:

00 0

6 Ju

ne 2

013