Embed Size (px)
Citation preview
1
Fit for Purpose Community Health Surveys: An Experiment in Three Communities
American Association of Public Opinion ResearchMay 15th 2015
John Boyle PhD, ICFRonaldo Iachan PhD, ICFTala Fakhouri PhD MPH, ICFLewis Berman PhD MS, ICFJames Dayton, ICFMelanie Courtright, Research NowKartik Pashupati, Ph.D., Research Now
2
The Community HINTS (CHINTS) Pilot:Rationale
• National surveys, like the Health Information National Trends Survey (HINTS) are critical for measuring and monitoring the nation’s health.
• However, public health is implemented at the local level, and
• national data has limited usefulness for estimating local needs and evaluating local programs because:
o Lack the sufficient sample size to produce reliable “local” estimates
o Increasing local sample sizes using the same methods would be expensive
o Not designed to address topics that are specific to subpopulation or communities
o Not usually timely enough for local uses
o Annual estimates
o Pre-post evaluations
o Quick response to emerging issues
3
The CHINTS Pilot:Rationale
• Community surveys, that parallel national surveys (HINTS) could significantly improve actionable information for measuring and monitoring the nation’s health at both the national and local level;• Compare differences between national and local health risk factors, behaviors and outcomes
• Compare differences between communities in health, risk factors, behaviors and outcomes
• Compare differences within communities before and after program interventions
• Compare national trends with community trends
• Aside from county level Behavioral Risk Factor Surveillance surveys (BRFSS), very few communities conduct surveys that can be compared to national and other community estimates (e.g., NYC, LA)• Probability surveys are the standard
• Dual frame RDD (particularly cell stratum) is very expensive at community level
• ABS mail surveys have greater coverage (drop points) and language biases at the community level
• So, community level probability surveys are expensive compared to local budgets
• Advantages of the community surveys are more hypothetical than demonstrated
• So, how can we generate parallel community and national surveys to generate these types of national and local estimates?
4
The CHINTS Pilot:Proof of Concept
• Objective:
Test the utility of a non-probability web-survey in producing relatively unbiased community estimates using a general population online panel.
• Methods:
o Purposive Sampling using Census balanced sample of panel members.
o Post-Stratification Weighting: After data collection, the completed sample was further adjusted with post-stratification weights.
o Replication in three communities.
o Comparison of community findings against community based probability samples, and national probability estimates.
5
The CHINTS Pilot:Sample Design
• Site Selection
• Regions (East, West, Midwest)
• Cities of various size and composition
• Availability of both American Community Survey (ACS) and Behavioral Risk Factor Surveillance System (BRFSS) data from comparability
• Two out of three were selected from Cancer Prevention and Control Research Network (CPCRN) for relevance
• Specific Sites
o Cleveland (Cleveland-Elyria, OH)
CPCRN: Case Western Reserve University
o Seattle (King County, WA)
CPCRN: University of Washington Seattle
o New York City (New York City, NY)
No CPRCN
6
The CHINTS Pilot:Sample
• General population online panel owned by Research Now®
o Research Now operates several national panels with more than 3,000,000 members in the United States
o For this project, all respondents were recruited from the e-Rewards (ERI) panel
o ERI is a double opt-in panel
o Initial opt-in to the panel itself is by invitation only
o Invitees to panel come from partner organizations (e.g., airlines, hotels, retailers)
o Secondary opt-in for individual survey invitations
o Industry standard quality checks are performed regularly in which responders are removed if they consistently provide poor quality or inconsistent data
o Checks include speeding checks, overuse of non-response options (e.g. “don’t know”), gibberish open-ends, and illogical or inconsistent responding
o Fingerprinting technology to ensure no duplicate responders
o Limits on survey participation to avoid “professional” survey takers
o Responders are incentivized for participation in the form of currency that can then be redeemed for rewards
7
The CHINTS Pilot:Sample Selection
• Sample selected based on profiled information for panelists
o Selection parameters included responder postal code (confirmed in the survey)
o Demographic parameters (age, gender, race, ethnicity and education) also used to ensure representativeness
o Profiling data is periodically updated as panelists are regularly prompted to re-enter profiling criteria to ensure accuracy
• Responders received a direct e-mail invitation to the survey (as opposed to web traffic or routed sample)
o Panelists received an initial survey invitation, potentially followed by a reminder (no sooner than 36 hours after the initial invite)
o General subject lines/survey invitation text used to limit any potential bias
o Completed surveys were monitored for representativeness, and additional invitations were sent out in order to balance the completes to match census parameters to the extent possible
8
The CHINTS Pilot:Data Collection Protocol
Invitations 3,987 2,962 7,766
Completes 506 513 521
Response RateRR=Completes/Invitations
12.7% 17.3% 6.7%
Cleveland New York CitySeattle
Method notes: Initial launch was conducted in NYC where too much sample was released, yielding 220 quota outs after 500+ completes target achieved. Comparable response rate for NYC after accounting for quota outs would be about 9.5%.
9
The Community HINTS (CHINTS) Survey:Unweighted Comparisons of Demographics
10
The CHINTS Pilot:Unweighted frequencies – Gender
39.4
60.6
36.4
63.6
47.451.9
0
20
40
60
80
100
Male Female
Cleveland
BRFSS CHINTS ACS
44.5
55.5
34.1
65.9
49.6 50.4
0
20
40
60
80
100
Male Female
Seattle
BRFSS CHINTS ACS
42.5
57.5
45.5
54.546.8
53.2
0
20
40
60
80
100
Male Female
New York City
BRFSS CHINTS ACS
11
The CHINTS Pilot:Unweighted frequencies – Age Categories
16.3
33.1
49.2
26.1
38.4 35.532.337.1
30.6
0
20
40
60
80
100
18-34 35-54 55+
Seattle
BRFSS CHINTS ACS
23.7
33.441.5
24.6
38.0 37.434.9 34.530.6
0
20
40
60
80
100
18-34 35-54 55+
New York City
BRFSS CHINTS ACS
14.2
30.9
54.0
19.4
44.136.6
26.934.2
38.9
0
20
40
60
80
100
18-34 35-54 55+
Cleveland
BRFSS CHINTS ACS
12
The CHINTS Pilot:Unweighted frequencies – Race and Hispanic Ethnicity
78.4
13.9
2.9 3.8
82.8
8.74.0 4.6
73.4
18.8
4.2 3.6
0
20
40
60
80
100
NH white NH black Hispanic NH other
Cleveland
BRFSS CHINTS ACS
80.7
3.0 4.510.5
71.0
4.3 4.3
20.5
66.2
5.6 7.8
20.4
0
20
40
60
80
100
NH white NH black Hispanic NH other
Seattle
BRFSS CHINTS ACS
57.2
10.7
19.4
10.4
48.0
20.0 22.5
9.6
34.6
21.927.2
16.4
0
20
40
60
80
100
NH white NH black Hispanic NH other
New York City
BRFSS CHINTS ACS
13
The CHINTS Pilot:Unweighted frequencies – Education
38.8
60.1
28.3
71.7
40.8
59.2
0
20
40
60
80
100
HS or less More than HS
Cleveland
BRFSS CHINTS ACS
18.3
81.4
7.6
92.4
25.7
74.3
0
20
40
60
80
100
HS or less More than HS
Seattle
BRFSS CHINTS ACS
33.8
65.0
22.3
77.7
43.4
56.6
0
20
40
60
80
100
HS or less More than HS
New York City
BRFSS CHINTS ACS
14
The CHINTS Pilot:Comparability before weighting – Conclusion 1
• Overall
• Unweighted CHINTS is comparable to unweighted BRFSS
• Gender
o Like the BRFSS sample, the CHINTS panel sample is predominantly female.
• Age
o Both samples under-represent the younger groups
• Race and Hispanic Ethnicity
o The racial and ethnic composition of the CHINTS panel sample is similar to the population (ACS) in each site.
• Education
o CHINTS over-represents people with higher education – biggest distinction from BRFSS
15
The Community HINTS (CHINTS) Pilot Survey:Weighted Estimates of Key Health Measures
Demographic raking to ACS estimates for age, gender, race and Hispanic ethnicity, education and marital status
16
The CHINTS Pilot:Weighted Estimates – General Health
57.3
30.3
13.5
3.9
59.2
35.3
12.6
1.3
0
20
40
60
80
100
Excellent or VeryGood
Good Fair Poor
Seattle
BRFSS CHINTS
52.2
30.3
13.5
3.9
50.9
35.3
12.6
1.3
0
20
40
60
80
100
Excellent or VeryGood
Good Fair Poor
Cleveland
BRFSS CHINTS
50.2
30.3
13.5
3.9
51.2
35.3
12.6
1.3
0
20
40
60
80
100
Excellent or VeryGood
Good Fair Poor
New York City
BRFSS CHINTS
17
The CHINTS Pilot:Weighted Estimates – Height (inches) and Weight (Pounds)
Height (inches) Mean
BRFSS CHINTS
Cleveland 67.1 67.2
Seattle 67.2 67.3
New York City 66.4 66.2
Weight (pounds) Mean
BRFSS CHINTS
Cleveland 178.7 191.8
Seattle 172.9 175.5
New York City 167.5 172.8
18
The CHINTS Pilot:Weighted Estimates –Time Since Last Routine Checkup
72.1
13.76.6 6.8
0.8
68.0
15.3
5.411.3
0
20
40
60
80
100
< 1 year 1 to < 2 year 2 to < 5 year 5+ years Never
Cleveland
BRFSS CHINTS
71.9
14.07.6 5.4
1.1
70.9
13.66.7 8.8
0
20
40
60
80
100
< 1 year 1 to < 2 year 2 to < 5 year 5+ years Never
New York City
BRFSS CHINTS
60.6
17.012.6
8.41.4
67.1
14.27.7
11.1
0
20
40
60
80
100
< 1 year 1 to < 2 year 2 to < 5 year 5+ years Never
Seattle
BRFSS CHINTS
19
The CHINTS Pilot:Weighted Estimates – Conclusion 2
• After weighting for demographics, CHINTS and BRFSS health estimates are generally equivalent
• No clear need to weight based on general health status after demographic weighting
20
Do Community Estimates Matter?A few illustrative comparisons across sites and with HINTS
21
The CHINTS Pilot:A Comparison across sites – Overall Profile from survey
Cleveland Seattle New York City
Age (% 18-34 years) 26.3 32.2 34.1
Race and Ethnicity (% Non-Hispanic white) 74.0 66.4 34.8
Education (% More than High School) 58.9 74.3 56.7
Income (% $100,000 or more) 18.2 33.1 17.4
Is there a particular doctor, nurse, or other health professional that you see most often?
73.7 66.2 58.6
Insured (Overall) 92.0 96.5 90.6
Insurance through employer 46.2 53.0 37.0
Insurance purchased directly 6.0 4.8 6.8
Medicare 16.9 10.1 10.5
Medicaid 9.6 5.0 20.1
Other 21.3 27.0 25.5
22
The CHINTS Pilot:A Comparison across sites – Most recent routine checkup
About how long has it been since you last visited a doctor for a routine checkup?
68.0
15.3
5.4
11.3
0.0
67.1
14.2
7.711.1
0.0
70.9
13.6
6.78.8
0.0
67.7
13.9
7.2 6.62.9
0
20
40
60
80
100
Within the past year 1 but less than 2years
2 but less than 5years
5+ years Don't know
Cleveland Seattle New York City HINTS
23
The CHINTS Pilot:A Comparison across sites – Methods used to Exchange Information
In the past 12 months, have you used any of the following to exchange medical information with a health care professional?
35.7
6.9 6.2
1.2 2.64.8
58.4
52.4
6.6 7.5
0.21.8
4.1
41.3
35.0
13.1
6.7
2.3
6.7 7.8
57.3
20.6
6.44.5
0.9 2.0
8.0
72.3
0
20
40
60
80
100
E-mail Text message App Videoconference
Social media Fax None
Cleveland Seattle New York City HINTS
24
The CHINTS Pilot:A Comparison across sites – Tobacco Use
44.5
13.5
26.9
36.7
63.9
28.2
7.5
20.2
68.3
80.8
34.1
13.2
25.0
62.6
76.2
44.5
19.2 20.0
61.765.3
0
20
40
60
80
100
Ever Smoked Currently Smoking 1 pack+/day Stopped for 1 day orlonger
Considering quitting
Cleveland Seattle New York City HINTS
Current Smokers
25
The CHINTS Pilot:A Comparison across sites – Screening for Cancer
23.4
78.5
53.8
76.4
40.9
81.6
65.1
86.9
27.9
84.3
48.6
76.8
33.2
77.9
56.4
71.0
0
20
40
60
80
100
Told you can choose tohave a mammogram?
Had a mammogram in thepast 2 years
Told you can choose tohave a colon cancer test?
Ever had a test for coloncancer
Cleveland Seattle New York City HINTS
Breast Cancer ScreeningWomen 50 – 74 years
Colon Cancer ScreeningAll Adults 50 – 74 years
26
The CHINTS Pilot:A Comparison across sites – Your Cancer History
Have you ever been diagnosed as having cancer?
8.4 8.6 8.78.1
0
5
10
15
20
All Cancers (weighted)
Cleveland Seattle New York City HINTS
27
The CHINTS Pilot:City Comparisons – Conclusion 3
• Demographic and health system characteristics vary across local communities and the nation
• CHINTS weighted estimates reflect variability across the three sites for some but not all indicators
• Differences are generally consistent with the demographic and health system differences between the communities
• Differences in health and health information related behaviors between communities offer valuable insights into needs, opportunities and outcomes of great importance to national and local health care programs
• Ability to track health and health information trends at the local level should strengthen national as well a local health programs
28
The CHINTS Pilot:Lessons Learned
• The CINTS pilot demonstrated the viability of collecting useful internet panel data rapidly and inexpensively for different types of communities
• The demographic weighting adjustments minimize potential biases, no health related weighting was required
• For many outcomes, CHINTS weighted estimates seemed no more biased than the BRFSS weighted estimates
• Higher response rates should be achievable in future studies when more rigorous sample controls and follow-up contract protocols for the studies are implemented by the panel organization
• Cost to conduct community comparison surveys using web panels is well within the resources of most local health departments
