Upload
godfrey-dalton
View
216
Download
0
Tags:
Embed Size (px)
Citation preview
Building Wave Response Rates in a Longitudinal Survey: Essential for Nonsampling Error Reduction
or Last In - First Out?
Steven B. CohenSteven B. Cohen
Fred Rohde and William YuFred Rohde and William Yu
Agency for Healthcare Research and QualityAgency for Healthcare Research and Quality
Advancing Advancing Excellence in Excellence in Health CareHealth Care Purpose of DiscussionPurpose of Discussion
Need for essential longitudinal data on health care Need for essential longitudinal data on health care coverage, use and expenditures to inform health care coverage, use and expenditures to inform health care policy and practicepolicy and practice
Description of the Medical Expenditure Panel Survey Description of the Medical Expenditure Panel Survey (MEPS): purpose, longitudinal design and analytical (MEPS): purpose, longitudinal design and analytical capacitycapacity
Focus on field efforts to achieve target response ratesFocus on field efforts to achieve target response rates
Advancing Advancing Excellence in Excellence in Health CareHealth Care Purpose of DiscussionPurpose of Discussion
Evaluations of the quality of the MEPS Evaluations of the quality of the MEPS nonresponse adjustment strategies nonresponse adjustment strategies
Determination of characteristics for Determination of characteristics for cases cases fielded at end of field period (EOF) & fielded at end of field period (EOF) & conversion of temporary refusals (TNR). conversion of temporary refusals (TNR).
Examine ROI for inclusion of these cases. Examine ROI for inclusion of these cases.
Advancing Advancing Excellence in Excellence in Health CareHealth Care Purpose of DiscussionPurpose of Discussion
Examine impact on annual and longitudinal Examine impact on annual and longitudinal response rates; completion of self response rates; completion of self administered questionnaires (SAQ)administered questionnaires (SAQ)
Impact on key survey estimates of health Impact on key survey estimates of health insurance coverage and expendituresinsurance coverage and expenditures
Implications of alternative field strategiesImplications of alternative field strategies
Advancing Advancing Excellence in Excellence in Health CareHealth Care
Medical Expenditure Panel Medical Expenditure Panel Survey (MEPS)Survey (MEPS)
Annual Survey of 15,000 households:Annual Survey of 15,000 households: provides national estimates of health care use, expenditures, provides national estimates of health care use, expenditures, insurance coverage, sources of payment, access to care and insurance coverage, sources of payment, access to care and health care qualityhealth care quality
Permits studies of:Permits studies of: Distribution of expenditures and sources of paymentDistribution of expenditures and sources of payment Role of demographics, family structure, insuranceRole of demographics, family structure, insurance Expenditures for specific conditionsExpenditures for specific conditions Trends over timeTrends over time
Advancing Advancing Excellence in Excellence in Health CareHealth Care
Key Features of MEPS-HCKey Features of MEPS-HC
Survey of U.S. civilian noninstitutionalized populationSurvey of U.S. civilian noninstitutionalized population Sub-sample of respondents to the National Health Interview Sub-sample of respondents to the National Health Interview
Survey (NHIS)Survey (NHIS)– Linkage to NHISLinkage to NHIS
Oversample of minorities and other target groupsOversample of minorities and other target groups Panel Survey – new panel introduced each yearPanel Survey – new panel introduced each year
– Continuous data collection over 2 ½ year periodContinuous data collection over 2 ½ year period
– 5 in-person interviews (CAPI) 5 in-person interviews (CAPI)
– Data from 1st year of new panel combined with data from Data from 1st year of new panel combined with data from 2nd year of previous panel2nd year of previous panel
Advancing Advancing Excellence in Excellence in Health CareHealth Care
MEPS Overlapping PanelsMEPS Overlapping Panels(Panels 8 and 9)(Panels 8 and 9)
MEPS Household Component MEPS Panel 8 2003-
2004
Round 2 Round 3 Round 4 Round 5
Round 1 Round 2 Round 3
MEPS Panel 92004-2005
1/1/2003 1/1/2004
Round 1NHISNHIS20022002
NHISNHIS20032003 Round 4 Round 5
Advancing Advancing Excellence in Excellence in Health CareHealth Care
MEPS Household ComponentMEPS Household ComponentSample DesignSample Design
Oversampling of policy relevant domainsOversampling of policy relevant domains
19961996 Minorities (Blacks & Hispanics)Minorities (Blacks & Hispanics)
19971997 MinoritiesMinoritiesLow income Low income
Children with activity limitationsChildren with activity limitations
Adults with functional limitationsAdults with functional limitations
Predicted high expenditure casesPredicted high expenditure cases
ElderlyElderly
1998-20011998-2001 MinoritiesMinorities
2002+2002+ Minorities, Asians, Low IncomeMinorities, Asians, Low Income
15,000 households; ~35,000 persons15,000 households; ~35,000 persons
Advancing Advancing Excellence in Excellence in Health CareHealth Care Design SpecificationsDesign Specifications
Target Precision Target Precision Specifications for national Specifications for national and regional estimates; policy and regional estimates; policy relevant subgroupsrelevant subgroups
Overall Design effect of 1.6Overall Design effect of 1.6 200 PSU design (Max)200 PSU design (Max) Overall/round specific survey Overall/round specific survey
response rate requirementsresponse rate requirements
Linkage to NHISLinkage to NHIS Multistage designMultistage design Disproportional samplingDisproportional sampling Longitudinal designLongitudinal design Minimize survey cost for Minimize survey cost for
fixed precisionfixed precision
MEPS, 1996–2006: Number of uninsured, under age 65
65.863.962.961.761.962.2 59.1 58.561.762.0
50.149.848.147.045.945.742.642.0
44.5 44.2 43.8
35.834.433.732.031.331.528.731.032.131.6
0
20
40
60
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006
Any time in year First half of year Full year
Nu
mb
er
in m
illi
on
s
Source: Center for Financing, Access, and Cost Trends, AHRQ, Household Component of the Medical Expenditure Panel Survey, 1996–2005 Full-Year and 1996–2006 Point-in-Time Files
Advancing Advancing Excellence in Excellence in Health CareHealth Care Trends in ConcentrationTrends in Concentration
27
38
55
70
97
28
39
56
70
97
28
38
56
69
97
23
33
50
65
97
0102030405060708090
100
Top 1% Top 2% Top 5% Top 10% Top 50%Population ranked by expenditures
1977 1987 1996 2005
Per
cent
age
of e
xpen
ditu
res
Source: National Medical Care Expenditure Survey, 1977; National Medical Expenditure Survey, 1987; Medical Expenditure Panel Survey, 1996 and 2005.
Advancing Advancing Excellence in Excellence in Health CareHealth Care MEPS Field ForceMEPS Field Force
Westat is data collection organizationWestat is data collection organization 500 interviewers 500 interviewers Sample allocated in ~ 200 PSUs, spread across all Sample allocated in ~ 200 PSUs, spread across all
50 states50 states extensive training modulesextensive training modules Information on socio-demographic characteristics of Information on socio-demographic characteristics of
households available based on linkage with NHIShouseholds available based on linkage with NHIS Remuneration for survey participationRemuneration for survey participation
Advancing Advancing Excellence in Excellence in Health CareHealth Care
Tool Chest of Methods to Maximize Tool Chest of Methods to Maximize Survey ResponseSurvey Response
Recruitment of experienced and bilingual interviewerRecruitment of experienced and bilingual interviewer 10+ days training (including procedures for obtaining signed 10+ days training (including procedures for obtaining signed
consents)consents) Uses of MEPS data as reference materials for interviewersUses of MEPS data as reference materials for interviewers Periodic retraining and special trainings (e.g. methods to Periodic retraining and special trainings (e.g. methods to
improve response rates)improve response rates) Respondent remunerationRespondent remuneration Advance mailings from co-sponsors of surveyAdvance mailings from co-sponsors of survey Monthly planning calendar and MEPS DVDMonthly planning calendar and MEPS DVD Daily emails to interviewers regarding interviewing progressDaily emails to interviewers regarding interviewing progress Multiple contacts for refusal conversionsMultiple contacts for refusal conversions
Advancing Advancing Excellence in Excellence in Health CareHealth Care
MEPS Target Response Rates by Round MEPS Target Response Rates by Round and Overalland Overall
Response Rate
NHIS 90%*
Round 1 80%
Round 2 95%
Round 3 96%
Year 1 of Panel 66%
Round 4 97%
Round 5 98%
Pooled Response Rate-Two Panels 65%
*NHIS response rate among households designated for MEPS.Note: Year 1 and the Overall response rate include the NHIS response rate.
Advancing Advancing Excellence in Excellence in Health CareHealth Care
Person Level (survey attrition) Person Level (survey attrition) Nonresponse Adjustment CovariatesNonresponse Adjustment Covariates
Factors associated with survey attrition (after R1)Factors associated with survey attrition (after R1)
– Indicator for initial refusal to R1 interview Indicator for initial refusal to R1 interview
– Family sizeFamily size
– AgeAge
– MSA, census regionMSA, census region
– Marital status (family reference person)Marital status (family reference person)
– Race/ethnicityRace/ethnicity
– Education of reference personEducation of reference person
– Employment statusEmployment status
– Health insurance statusHealth insurance status
– Total expenditures (in yr 1 for yr 2 adj.)Total expenditures (in yr 1 for yr 2 adj.)
– # doctor visits (in yr 1)# doctor visits (in yr 1)
– Self reported health statusSelf reported health status
Advancing Advancing Excellence in Excellence in Health CareHealth Care
Person Level Adjustments:Person Level Adjustments:Annual EstimatesAnnual Estimates
Each MEPS panel weighted separatelyEach MEPS panel weighted separately Nonresponse adjustment for complete Nonresponse adjustment for complete
nonresponse and for survey attritionnonresponse and for survey attrition Final Poststratification adjustment –CPS Final Poststratification adjustment –CPS
12/31:12/31:
age, race/ethnicity, sex, region, MSA age, race/ethnicity, sex, region, MSA status, status, poverty statuspoverty status
Advancing Advancing Excellence in Excellence in Health CareHealth Care
Testing for Panel EffectTesting for Panel Effect
Advancing Advancing Excellence in Excellence in Health CareHealth Care
Capacity of MEPS to Produce Comparable Capacity of MEPS to Produce Comparable NHIS Estimates of Health Insurance NHIS Estimates of Health Insurance
CoverageCoverage
Advancing Advancing Excellence in Excellence in Health CareHealth Care
Characteristics of Respondents Fielded at Characteristics of Respondents Fielded at End of First Round or Temporary RefusalEnd of First Round or Temporary Refusal
Initial Refusals:Initial Refusals:Higher likelihood:Higher likelihood: MSA residence; Northeast region; white Non-MSA residence; Northeast region; white Non-
Hispanic; elderly; excellent health; some high Hispanic; elderly; excellent health; some high school; family size 2+; Attrite in future waves school; family size 2+; Attrite in future waves of data collection of data collection
End of Field Period CasesEnd of Field Period CasesHigher likelihood:Higher likelihood: Race: Asian or BlackRace: Asian or Black in excellent healthin excellent health Attrite in future waves of data collectionAttrite in future waves of data collection
Advancing Advancing Excellence in Excellence in Health CareHealth Care
Testing for Reluctant Response Effect Testing for Reluctant Response Effect on Coverage Estimateson Coverage Estimates
-------------------------------------------------------
DF Wald F P-ValueDF Wald F P-Value--------------------------------------------------------------------------------------------------------------OVERALL MODEL 22 107.98 0.0000OVERALL MODEL 22 107.98 0.0000
JULY INTERVIEW JULY INTERVIEW 1 2.38 0.1244 1 2.38 0.1244TEMP. REFUSALS 1 0.92 0.3393TEMP. REFUSALS 1 0.92 0.3393SEX SEX 1 98.02 <0.0001 1 98.02 <0.0001RACE/ETHNICITY 3 58.42 <0.0001RACE/ETHNICITY 3 58.42 <0.0001MARITAL STATUS 4 16.90 <0.0001MARITAL STATUS 4 16.90 <0.0001EDUCATION 4 10.94 <0.0001EDUCATION 4 10.94 <0.0001POVERTY STATUS 4 43.97 <0.0001POVERTY STATUS 4 43.97 <0.0001MSA STATUS 1 4.34 0.0382MSA STATUS 1 4.34 0.0382INDIVIDUAL INCOME 1 35.52 <0.0001INDIVIDUAL INCOME 1 35.52 <0.0001MEDICAL $ 1 35.79 <0.0001MEDICAL $ 1 35.79 <0.0001--------------------------------------------------------------------------------------------------------------
-2 * Normalized Log-Likelihood Full Model: 13037.78-2 * Normalized Log-Likelihood Full Model: 13037.78Pseudo RPseudo R2:2: :0.194167 :0.194167
Advancing Advancing Excellence in Excellence in Health CareHealth Care Mean Number of Contacts by MonthMean Number of Contacts by Month
R1 Interview Month
Me
an
Nu
mb
er
of C
on
tact
s
5
10
15
20
< Mar Mar Apr May Jun Jul
Overall
< Mar Mar Apr May Jun Jul
O refusals
< Mar Mar Apr May Jun Jul
1+ refusals
Panel 9Panel 10
Advancing Advancing Excellence in Excellence in Health CareHealth Care
Mean Number of Temporary Mean Number of Temporary Refusals by MonthRefusals by Month
R1 Interview Month
Me
an
Nu
mb
er
of R
efu
sals
0
2
4
6
8
< Mar Mar Apr May Jun Jul
Overall
< Mar Mar Apr May Jun Jul
O refusals
< Mar Mar Apr May Jun Jul
1+ refusals
Panel 9Panel 10
Advancing Advancing Excellence in Excellence in Health CareHealth Care
Conditional Response Rates by Month Conditional Response Rates by Month of Round 1 Response: Panel 9of Round 1 Response: Panel 9
R1 Interview Month
Re
spo
nse
Ra
te
0.2
0.4
0.6
0.8
< Mar Mar Apr May Jun Jul
Overall
< Mar Mar Apr May Jun Jul
O refusals
< Mar Mar Apr May Jun Jul
1+ refusals
1st-year response rate1st-year SAQ response rate1st- and 2nd-year SAQ response rateLongitudinal response rate
Advancing Advancing Excellence in Excellence in Health CareHealth Care
Panel/temp. Panel/temp. refusalrefusal
InterviewInterviewdate date N persons N persons
Mean R1Mean R1contactscontacts
Mean R1Mean R1Temp.Temp.
refusalsrefusals
full year full year Cond.Cond.
responseresponse
LongitudLongitudCond.Cond.
responseresponse
Overall P9Overall P9 OverallOverall 18,250 18,250 6.176.17 0.300.30 89.90%89.90% 85.40%85.40%
P10P10 18,170 18,170 6.676.67 0.350.35 88.80%88.80% ..
P9P9 < Jul< Jul 17,537 17,537 5.645.64 0.190.19 90.30%90.30% 85.90%85.90%
P10P10 17,380 17,380 6.166.16 0.230.23 89.30%89.30% ..
P9P9 JulJul 713 713 19.2119.21 2.942.94 79.20%79.20% 71.10%71.10%
P10P10 790 790 17.7817.78 2.942.94 76.70%76.70% ..
0 P90 P9 < Jul< Jul 16,631 16,631 5.425.42 0.000.00 91.00%91.00% 86.80%86.80%
0 P100 P10 16,377 16,377 5.905.90 0.000.00 89.90%89.90% ..
1+ P91+ P9 JulJul 272 272 17.2617.26 7.717.71 77.20%77.20% 62.50%62.50%
1+ P101+ P10 328 328 16.5516.55 7.087.08 73.20%73.20% ..
ExclusionsExclusions July July RR reduction RR reduction .041.041 .959.959
June refusalJune refusal July July .062.062 .938.938
MeasureStandard Field
Operation 50% sample 33% sample 25% sample Exclusion
Difficult Cases Difficult CasesDifficult
Cases Difficult Cases
Mean Expenditures 3479.01 3477.90 3478.24 3521.02 3543.21
Standard Error 91.03 92.02 92.93 94.17 94.94
Responding sample 31589 30769 30495 30359 29948
July cases+June temp refusals 1641 821 547 411
RSE 0.026 0.026 0.027 0.027 0.027
Bias 0 0 0 0 64.2064.20
MSE 8287.10 8467.37 8635.93 8868.86 13135.24
RMSE 91.03 92.02 92.93 94.17 114.61
Ratio of MSE to Standard 1 1.022 1.042 1.070 1.586
Mean no. hours per complete 14.2 22 22 22 12.5
Add’l sample to meet precision goal 670 1,284 2,132
Impact on MSE of Mean Medical Expenditure Estimates for alternative field strategies
MeasureStandard Field
Operation 50% sample 33% sample 25% sample Exclusion
Difficult Cases Difficult CasesDifficult
Cases Difficult Cases
Proportion
Standard Error
Responding sample
July cases+June temp refusals
RSE
Bias
MSE
RMSE
Ratio of MSE to Standard
Impact on MSE of Proportion with Medical Expenses in excess of $10,000 for alternative field strategies
0.0756 0.0756 0.0757 0.077 0.0774
0.0021 0.0022 0.0022 0.0023 0.0022
31589 30769 30495 30359 29948
1641 821 547 411
0.028 0.029 0.029 0.030 0.028
0 0 0 0 0.002
0.00000441 0.00000484 0.00000484 0.00000529 8.08E-06
0.0021 0.0022 0.0022 0.0023 0.0028
1 1.098 1.098 1.200 1.832
Advancing Advancing Excellence in Excellence in Health CareHealth Care
SummarySummary
Need for accurate and reliable national data on Need for accurate and reliable national data on health insurance coverage to inform policy and health insurance coverage to inform policy and practice practice
MEPS design features and analytical capacityMEPS design features and analytical capacity Statistical, methodological and operational design Statistical, methodological and operational design
features to adjust for nonresponse and attritionfeatures to adjust for nonresponse and attrition Evaluation of estimation strategies to correct for Evaluation of estimation strategies to correct for
survey attrition survey attrition Examination of ROI for inclusion of difficult cases Examination of ROI for inclusion of difficult cases Options identified for more efficient and effective Options identified for more efficient and effective
field strategiesfield strategies
Agency for Healthcare Research and QualityAgency for Healthcare Research and QualityAdvancing Excellence in Health CareAdvancing Excellence in Health Care • • www.ahrq.govwww.ahrq.gov