1
Class 10
Creating Scores and Change Scores, Presenting Measurement Data,
Selecting Standard Survey Items
November 29, 2007
Anita L. Stewart Institute for Health & Aging
University of California, San Francisco
2
Overview of Class 10
Creating summated scales and presenting measurement information
Creating and presenting change scores “The rest of the survey”
– Locating standard survey questions
3
Creating Likert Scale Scores
Translate codebook scoring rules into program code (SAS, SPSS):– Reverse all items that are not already in
desired direction (e.g., higher = better)– Average all items
»Allows score if 1 item is answered
– Apply missing data rule if different »e.g., if more than 50% items missing
5
Review Summated Scores
Review scores for out-of-range values, outliers, expected mean
For scores with problems, review programming statements, locate errors and correct
Repeat process until computer algorithm is producing accurate scores– To test programming accuracy, calculate scores by
hand from 2 questionnaires» Check that they match computer generated scores
6
Testing Scaling Properties in Your Sample for Multi-Item Scales
Obtain item-scale correlations– Part of reliability program
– Each item correlates at least .30 with the total scale (corrected for overlap)
7
Testing Scaling Properties in Your Sample for Multi-Item Scales (cont)
Calculate internal-consistency reliability (Cronbach’s alpha) for multi-item scales in your sample – Regardless of reliability in other studies
Internal consistency should be at least .70– If lower, see if deleting items <.30 will
improve it
8
Presenting Measurement Results (Handout)
Present for each final scale:– % missing– Mean, standard deviation– Observed range, possible range– Floor and ceiling effects, skewness statistic– Range of item-scale correlations
» Number of item-scale correlations > .30
– Internal consistency reliability
9
Overview of Class 10
Creating summated scales and presenting measurement information
Creating and presenting change scores “The rest of the survey”
– Locating standard survey questions
10
Change Scores are Important Variables!
Creating change score variables is complex– Requires thought ahead of time
Don’t rely on your programmer Include specification of change scores in
your codebook
11
Three Types of Change Scores
Measured change– Difference in scores between baseline and
follow-up Percentage change
– Measured change as percent of baseline score Perceived change
– How much change respondent reports (from some prior time period)
12
Measured Change
Difference in scores from baseline to follow-up
Example measure administered at baseline and 1 month after treatment– Pain in past 2 weeks, 0-10 numeric scale,
10 = worst pain
13
Measured Change (cont)
Hypothetical results– Time 1 (baseline) - score of 5
– Time 2 (one month) - score of 8 How should change be measured?
14
Measured Change (cont)
Time 1 (baseline) - score of 5Time 2 (one month) - score of 8
How should change be measured? Two options:
– Time 2 minus time 1
– Time 1 minus time 2
15
Measured Change (cont)
Time 1 (baseline) - score of 5Time 2 (one month) - score of 8
Option one: time 2 minus time 1= +3 Option two: time 1 minus time 2 = -3 Interpretation of change score?
16
Interpretation of Change Score
What do you want the change score to indicate? – Positive change score = improving?– Positive change score = worsening?
Scoring thus depends on:– Direction of scores on original measure (is higher
score better or worse?)– Which was subtracted from which?
17
Define Change Score Before Calculation: Algorithms
You want positive score = improvement
If high score on measure is better– Time 2 minus time 1
If high score on measure is worse– Time 1 minus time 2
You want positive score = decline
If high score on measure is better– Time 1 minus time 2
If high score on measure is worse– Time 2 minus time 1
18
Example: You Want Positive Score To Indicate Improvement
Hypothetical subject: Improved Subtract score nearest “worst” end from
score nearest “best” end
(worst) 0 1 2 3 4 5 6 7 8 9 10 (best)
time 1 time 2
19
Example: You Want Positive Score To Indicate Improvement
Subtract score nearest “worst” end from score nearest “best” end
(worst) 0 1 2 3 4 5 6 7 8 9 10 (best)
time 1 time 2
Time 2 minus time 1 = +4 (improved by 4 points)
20
Example: You Want Positive Score To Indicate Improvement (Scale Reversed)
Hypothetical subject: Improved Subtract score nearest “worst” end from
score nearest “best” end
(best) 0 1 2 3 4 5 6 7 8 9 10 (worst)
time 2 time 1
21
Example: You Want Positive Score To Indicate Improvement (Scale Reversed)
Subtract score nearest “worst” end from score nearest “best” end
(best) 0 1 2 3 4 5 6 7 8 9 10 (worst)
time 2 time 1
Time 1 minus time 2 = +4 (improved by 4 points)
22
Recommendation: Make Change Score Intuitively Meaningful
If high score on measure = better
Calculate change score so positive change score = improved– Time 2 minus time 1
If high score on measure = worse
Calculate change scores so positive change score = improved– Time 1 minus time 2
23
Interpreting “Measured Change” Scores: What is Wrong?
In a study predicting utilization of health care (outpatient visits) over a 1-year period as a function of self-efficacy…
A results sentence:– “Reduced utilization at one year was associated
with level of self efficacy at baseline (p < .01) and with 6-month changes in self efficacy (p < .05).”
24
Interpreting “Measured Change” Scores: Making it Clearer
“Reduced outpatient visits at one year were associated with lower levels of self efficacy at baseline (p < .01) and with 6-month improvements in self efficacy.”
Old way:– “Reduced utilization at one year was associated
with level of self efficacy at baseline (p < .01) and with 6-month changes in self-efficacy.”
25
Three Types of Change Scores
Measured change– Difference in scores between baseline and
follow-up Percentage change
– Measured change as percent of baseline score Perceived change
– How much change respondent reports (from some prior time period)
26
Presenting Change Scores in Tables: What is Wrong?
Change in anxiety over a 1-year period for two groups
1 year change in anxiety p
Exercise group - 40 < .001Education group +4 ns
27
Presenting Change Scores in Tables: Making it Clearer
Change in anxiety over a 1-year period for two groups
1 year change in anxiety p
Exercise group - 40 < .001Education group +4 ns
*Negative score indicates decreased anxiety (change scores are 1-year minus baseline)
28
Reliability of Change Score
Difference scores have been criticized as having low reliability
Nunnally (1994) considers alternatives and suggests this may not be as large a problem as previously thought (p. 247)
Nunnally JC and Bernstein IH. Psychometric Theory, Third Edition, McGraw-Hill, New York, 1994.
29
Percentage Change
Measured change divided by baseline score
Example: pain measure, higher is more pain– change score of -2, baseline score of 6
– 2/6 = 33% reduction in pain
30
Example of Percentage Change Problem with Likert Scales
You want a positive change to indicate improvement (and high score is better)
Subtract score nearest “worst” end from score nearest “best” end
(worst) 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 (best)
time 1 time 2
Time 2 minus Time 1 = change of +4 (improved by 4 points)Change of 4 / baseline score of 8 = 50% improvement
31
Example of Percentage Change Problem with Likert Scales (cont.)
You want a positive change to indicate improvement– high score is worse
Subtract score nearest “best” end from score nearest “worst” end
(best) 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 (worst)
time 2 time 1
Time 1 minus Time 2 = change of +4 (improved by 4 points)Change of 4 / baseline score of 16 = 25% improvement
32
Percentage Change Scores Only Work for Ratio-Level Measures
Can do percentage change only on scales with a true zero– zero represents the absence of the trait in question
Ratio scores - weight in pounds Person weighs 150 pounds
– Gains 10, gained 15% of original weight– Loses 10, lost 15% of original weight
33
Three Types of Change Scores
Measured change– Difference in scores between baseline and
follow-up Percentage change
– Measured change as percent of baseline score Perceived change
– How much change respondent reports (from some prior time period)
34
Perceived Change (Retrospective Change)
How much has your physical functioning changed since your surgery?1 - very much worse2 - much worse3 - worse4 - no change5 - better6 - much better7 - very much better
35
Perceived Change (Retrospective Change) – Better Response Choice?
How much has your physical functioning changed since your surgery?-3 Very much worse-2 Much worse-1 Worse 0 No change 1 Better 2 Much better 3 Very much better
36
Perceived/Retrospective Change
Perceived change enables respondent to define physical functioning in terms of what it means to them
Measured change is a change on specific questions that were contained in the particular measure
37
Example of Measured Change
Baseline and 6-month limitations:
– Difficulty walking
– Difficulty climbing stairs Measured change: change on these 2 physical
functions If person had no change walking or climbing
stairs– Score would be “no change”
38
Example of Perceived Change To what extent did your physical functioning
change over the past 6 months? – Much worse– Worse– No change– Better– Much better
If person has more trouble bending over, and considers this as part of physical functioning, they will report becoming worse
39
Perceived/Retrospective Change Recommend including both types of
measures to assess change– Measured change enables
» Comparison with other studies» May be more sensitive because has more scale levels (if
multi-item measure)» Investigator defines clinically relevant outcomes
– Perceived/Retrospective change enables» Person to report on domain using their own definition» Picks up changes “unmeasured” by particular measure
40
Overview of Class 10
Creating summated scales and presenting measurement information
Creating and presenting change scores “The rest of the survey”
– Locating standard survey questions
41
Locating “Standard” Survey Questions
MD characteristics Comorbidity, chronic conditions Medical history, family history Health behaviors
42
Demographics – Just About Everywhere
Basic demographics Socioeconomic status Financial information (assets, income,
wealth) Employment, occupation Retirement Health insurance
43
Take Away Point:
Don’t write these yourself Use standard questions from appropriate
existing surveys
44
National and State Surveys
Population surveys Tend to have single-item measures rather
than multi-item scales– Good for “standardized” survey items
45
State Surveys
http://www.chis.ucla.edu/ California Health Interview Survey
(CHIS) “Questionnaires”
See contents of 2006 CHIS: adults and adolescents
46
National Surveys
Behavioral Risk Factor Surveillance System Questionnaires
http://www.cdc.gov/brfss/questionnaires/questionnaires.htm
See contents of 2006 BRFSS – English and Spanish
47
MacArthur Research Network on Socioeconomic Status and Health
Measures of economic status, occupational status, education, and perceived social status– Includes rationale
http://www.macses.ucsf.edu/Research/Social%20Environment/notebook/economic.html
Also basic demographics
48
Center for Aging in Diverse Communities (CADC)
Recommends items measuring socioeconomic status– Education, income, race/ethnicity, place of
birth/generation, English language proficiency, financial hardship
Main website: http://medicine.ucsf.edu/cadc/cores/measurement/index.html
49
Cancer Research Measures
The Division of Cancer Epidemiology and Genetics
Demographics, medical history, family history, other risk factors
http://dceg.cancer.gov/QMOD/
50
Non-English Language?
California Health Interview Survey– Numerous languages
Spanish language surveys– SALSA– Hispanic Health and Nutrition Examination Survey
(HHANES)– National Mexican Health and Aging Study – Behavioral Risk Factor Surveillance System (CDC)
51
NCHS National Health Care Surveys: Surveys of Physicians
Family of provider-based surveys Provide information about
– organizations and providers
– services rendered
– patients they serve Measures of physician variables including
practice characteristics
http://www.cdc.gov/nchs/nhcs.htm
52
Basic Measures About Children?
Commonwealth Fund Survey of Parents with Young Children– Parent administered
CHIS for adolescents (self-administered) and children (parent-administered)