Managing Survey Change - City · 2018. 11. 15. · Managing Survey Change Resource Tradeoffs and Quality Metrics. Brad Edwards . ESS Eric-City Methodology Seminar Series. April 16,

Managing Survey ChangeResource Tradeoffs and Quality Metrics

Brad Edwards ESS Eric-City Methodology Seminar SeriesApril 16, 2018

CHANGEWhat makes it special?

Difference, from the Greek diaphora

More interesting than stasis

Focus on continuing surveys

Most survey literature about design, methods, error

Survey management literature much sparser‣Response rates and costs‣ Lifecycle, 3MC‣Adapted for continuing surveys

2

Change in the Context of Survey Management

Managing Survey Change | Resource Tradeoffs and Quality Metrics

Repeating Survey Cycles

3

Data Harmonization

Data Collection

Pretesting

Sample Design

Questionnaire Design

Adaptation

Translation

Interviewer Recruitment, Selection & Training Instrument Technical Design

Data Dissemination Tenders, Bids & Contracts

Study, Organizational & Operational Structure

Ethical Considerations

in Surveys

SurveyQuality


Broad topic, layers of complexity

Contrast with one-off‣Harder to design‣Become easier than one-off in repetition

Tension between continuity and Total Survey Error paradigm, other quality dimensions

4

Survey Change


Survey Inertia

Strong tendency to minimize change Striving for state of maximum

efficiency, minimal error Incremental improvements

diminish Big change = high risk, high cost

10-20years

Rule of ThumbBig changes usually needed for one reason or another every

5Managing Survey Change | Resource Tradeoffs and Quality Metrics

Value (cost/quality) declines

Cost constraints

New opportunities‣Technology advances (or retreats)‣ Survey methods

Organization stress, capabilities

6

Some Signals of Need for Big Change


Some Continuing “Survey Health” Metrics

Topic constructs, societal and population changes‣Changes in mode use

Change in response rates by domain or group, or overall Change in breakoffs, comments,

help requests over time Data user or usage changes

Negative change in slope of number of publications by year

Negative change in slope of number of total users since start over time


8

Users (National Health & Aging Trends Study)

0

1000

2000

3000

2013 2014 2015 2016 2017

Number of Registered Users—Cumulative by Quarter


Relevance, timeliness, accessibility

Catastrophic fail

Time since last major change

Change for change’s sake

9

Other Reasons for Change


Dimensions of Change

Scale Resource tradeoffsOperations Data continuity Pretesting and risk assessment Politics and staging Schedule Cost

= 3 times new design costMajor legacy redesign cost


Case StudyCAPI Modernization on the Medical Expenditure Panel Survey (MEPS)

12

U.S. Health Care Utilization and Costs

The only developed nation with no single payer for health careHealth care is 18% of U.S. economy A fragmented system, not well-integrated Poorly understood by most consumers Large disparities in health care, health

outcomes, and cost Large population segment with

no health care insurance coverage


Complex topics (health conditions, health insurance, employment, provider organizational structure) Roles of individuals, families,

providers, and insurersHigh degree of nesting (5 layers)

Many-to-many relationships Severe underreporting of events

and costs Episodes of care Inter-round processing

13

Challenges for Health Survey Research Methods


Sponsored by Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services

Produces annual estimates of health care use and costs, health insurance coverage for civilian U.S. population Sample drawn from National

Health Interview Survey (NHIS) interviews the year before Each year data are collected from

about 26,000 families 5 CAPI interviews with families over

2.5 years, supplemented by medical provider data

14

MEPS | The Gold Standard


15

MEPS Data Collection Rounds, Reference Periods

Round 1 Data Collection





Panel 24






Panel 22






Panel 232017

2019

2018

2020

2021


16

MEPS and ESS: Scale and Complexity

MEPS ESS

Countries 1 23

Languages 2 20+

Survey Age in Years 21 16

Longitudinal? Yes No

Hierarchical Levels 5 1

Variables 4000+ 600+


17

MEPS Hierarchical Levels of Data (“Nesting”)

Reporting Unit (Household Family)

Health Insurance Provider

Person

Event

Condition

Job


1990 1995 2000 2005 2010 2015 2020

1996First CY

MEPS

2007Blaise/WVS

data collection software

1997Rotating Panel

Introduced with Panel 2

2018Blaise COTS data

collection software

18

MEPS Timeline of Major Changes


The Blaise|WVS Era

Solution to DOS “sunset”

No commercial off-the-shelf (COTS) Windows product (including Blaise) could scale up to handle MEPS’ nested arraysWestat coupled a Blaise rules

engine with a SQL “back-end”, hence Blaise|Westat Visual System, running in Windows XP

&


The Blaise|WVS Era (2)

“Literal” translation of data collection instrument, but incorporated medical provider directory “Walls” prevented back-up

into some critical sectionsModerate/severe performance

problems (>1 sec/answer entry) Engineering changes became

more difficult as Windows and other software products evolved

&


21

Measurement Errors Persisted… Event driver, and looping through each

family member for each provider probe and event, drove respondents to under-report, a conditioning effect Problem becomes more severe

with each succeeding interview Interviewers tried to ease respondent

burden with shortcuts they devised, increasing interviewer effects on measurement error


Panel Doctors Outpatient Emergency Hospital Stays Medicines

Round 5

P17 1.798 0.131 0.068 0.032 1.515

P18 2.001 0.185 0.071 0.038 1.571

P19 2.137 0.183 0.066 0.035 1.611

P20 1.916 0.181 0.070 0.036 1.577

Round 3

P18 2.867 0.259 0.115 0.051 1.842

P19 2.800 0.299 0.112 0.051 1.852

P20 2.629 0.220 0.104 0.045 1.807

P21 2.698 0.256 0.096 0.046 1.775

22

Monitoring Reports: Events per Person


In 2013, Microsoft announced support for Windows XP would end in 2014, raising major security issues

In 2014, the MEPS in Blaise/WVS (with Blaise 4.6) moved from XP to Windows 7, but only a temporary solution… Blaise 4.6 not supported by

StatNetherlands in Windows 7 Performance problems

worsened in Windows 7Major design changes became

increasingly difficult

23

The Biggest Tradeoff: To Go or Not to Go


AHRQ decided to bite the bullet, transitioning MEPS from Blaise/WVS to Blaise 4.8 Benefits of Blaise 4.8

‣ Improved performance‣Much less need for walls‣Greater flexibility in probing for events‣More efficient way to collect dates‣More flexible approach to provider directory‣Much greater potential for future design changes

24

CAPI Modernization (aka “Tech Upgrade”)


Maximize interview performance

Make only minimal design changes during tech upgrade period

Maintain schedule for all deliverables throughout transition

Replace Cheshire (legacy from 1996) with Blaise 4.8 for home office processing

Create a data exchange format and a single round structure for interfacing with data delivery tasks in SAS

Combine single rounds into a multi-round structure in Oracle

25

Transition Approach


26

Technical Architecture for CAPI Modernization

Cheshire format Cheshire structure


Interface:SAS2ORA

Oracle format Cheshire structure

SQL format WVS/Cheshire structure


Interface:SAS2ORA

Oracle format Cheshire structure

Blaise format Blaise/DC structure

Blaise format Blaise/MHOP structure

DEX single round structure

Oracle format DEX multiround structure

Phas

e 1

1996

-200

6Ph

ase

2 20

07-2

017

Tech

Upg

rade

2018

-

Data CollectionHome Office Processing

Interface to Data Delivery

Data Delivery


Cheshire Cheshire SAS Oracle

Blaise/WVS Cheshire SAS Oracle

Blaise Blaise SAS Oracle

Requirements statement Specifications for data collection

instrument and home office processing Programming Testing

Interviewer training Launch in the field Blending new data with old

for data deliveries coveringCY before launch Delivering analytic files

with the new data

27

CAPI Modernization | 8 Major Sets of Activities


0

2

4

6

8

10

12

14

16

18 Staged design/programming by group to dovetail with ongoing delivery schedule

1. People (administrative, demo-graphic data, or ADMIN/DEMO) is one of 4 highest in number of deliveries, must occur first

2. Events (UEGN) highest, part of most deliveries

3. Insurance (HINS) arguably most complex

28

Number of Annual Deliveries by Analytic Group


29

Schedule of Blaise Programming Cycles by Group

Load/environment set-up; Internal Programmer Spec

JAN 16

FEB 16

MAR 16

APR 16

MAY 16

JUN 16

JULY 16

AUG 16

SEP 16

1/11/16

2/18/16

4/19/16

7/12/16

Deliver to AHRQ

Deliver to AHRQ

Develop Cycle 1

Test Cycle 1

Develop Cycle 2Test Cycle 2

Develop Cycle 3

Test Cycle 3

Full Group 1&2 Test 1

Full Group 1&2 Test 2


Develop Cycle 1

Test Cycle 1


Develop Cycle 3

Test Cycle 3

Develop Cycle 1

Test Cycle 1Develop Cycle 2

Test Cycle 2

Group 1 | People Group 2 | Events Group 3 | Insurance Group 4 | Odds & Ends




Full Group 1 Test

Discovery: inherited specifications lacked definitions for key concepts (e.g., who was eligible for MEPS)

Created senior resource demands that could not be met by tech upgrade designers or analysts

To keep schedule from sliding further, additional resources identified to break through

30

Group 1 (People) Specifications Bump


31

Events per Family at Various Cut PointsPanel/Round 50 90 99 99.9 99.99 100

P20R1 1 9 27 59 91 92

P20R2 4 18 53 104 138 225

P19R1 1 9 29 66 116 148

P19R2 4 17 55 112 152 188

P19R3 4 20 29 146 204 208

P19R4 4 18 54 145 174 186

P19R5 3 16 52 98 193 258

P18R1 1 7 22 49 103 110

P18R2 4 19 55 119 158 167

P18R3 5 20 63 142 233 322P18R4 4 8 51 92 129 141

P18R5 3 15 45 90 145 181


Blaise requires a maximum number defined for each array Event array subject of intense discussion in 2015

‣Maximum events ever reported by any household in 15 years was about 300

‣AHRQ hoped the tech upgrade would improve reporting and decided on the requirement of 500 event-size array

‣Very large arrays depressed performance in CAPI Crisis in Group 2 programming in spring/summer 2016 resolved

by importing lead Blaise architect from Statistics Netherlands

32

Group 2 Requirements & the Big Event Tradeoff


As design and programming problems emerged, some schedule slippage was allowed to occur in tech upgrade – early decision in prototyping experience to delay implementation from 2016 to 2018

Ongoing deliveries took precedence in conflict

Design and programming problems had to be solved in some other way:

33

Tradeoffs: Tech Upgrade vs Ongoing Deliveries

Working smarter Adding staff or consultants Eliminating or condensing some

QC and review stages down the road

Shortening schedule for training materials development Overlapping groups more


Early deliveries each year from Westat to AHRQ – “end-of-round”, raw files, snapshot files – not Public Use Files (PUFs) – a black box to WestatWestat prioritized tech upgrade on PUFs, rationalizing variables and tables

with new, more efficient naming conventions As AHRQ analysts became more familiar with redesign, became apparent

to Westat old segment and variable names were used in many AHRQ programs that check the non-PUF files Tradeoff decision favored data delivery, minimizing disruption to QC/PUF

review at AHRQWestat produced new schema for naming conventions

34

Tradeoff: Future Efficiency vs Current Demands


Big Change in Environment

In late 2017, U.S. passed major tax reform legislation, eliminating universal health insurance mandate.Measuring trends in health

insurance coverage, impact on use, cost of health care central to AHRQ mission

Tradeoffs Early indications of error profile of

tech upgrade traded off against report quality in February: flash reports (metrics: events by type; comments per interview) Data took priority over reports:

monitoring reports delivered in March with partial data, while focusing on transforming the Blaise data into first stage of PUFs

35

Timeliness, Monitoring Reports, and Data


Monitoring Reports Data Year (2017) Data Year (2018) CAPI Development

Apr‘18

• Send AHRQ Test Data corresponding to UTIL/DEMO + HINS Monitoring Reports

• Send AHRQ Production Monitoring Report HINS

Condition Coding Wave 2: Data from SRD

Send AHRQ SRD production data (covering UTIL, DEMO and HINS monitoring reports)

AHRQ R2/R4 Alpha testing;

Test 3: P22R3/P21R5 Blaise data to Cheshire

First test of field materials R2/R4 from SRD

BFOS build (for FIT testing)Send AHRQ SAQs for approval

Updating edit programs for HINS R2/R4 Beta build

• (If needed) Test 4: P22R3/P21R5 Blaise data to Cheshire;

• Send AHRQ UEGN and PCND basic edit specifications;

• Begin FY database preparation (Apr 24); start of DEMO imputation

Send AHRQ full UEGN, PMED, ADMN/DEMO, HINS Transformation to SRD specs

• Westat R2/R4 Beta testing• FIT test 1

36

Swimlane Timeline, TechUp Deliveries to AHRQ (April)


37

Data Delivery Processing Activities (2018)FY2016 PUFsCheshire Processing

FY2017 PUFsCheshire Processing

FY2018 PUFsNew SRD/MRD Processing

UBGNCONDWGTSPMEDPRPL

ADMNDEMOEMPLCODEHINS

UEGNHLTH

CONDPCND

DCS/SAQDSDY

ADMNDEMOEMPLHINSHLTHPCNDWGTS

April May June July SeptAug


38

Spring 2018 Home Office Processing Data Flow

Complete Blaise cases

RU case level data cleaning

DQC loading daemon

Consolidated Blaise DB in field data model

Cleaned Blaise DB in DCQ data modelSpring 18

Data Release (Set in MHOP)

Blaise to Data Delivery transformation

Single Round DEX databasesSpring 18

Data Delivery


New systems built for receipt of authorization forms for Medical Provider Component (MPC) and for data quality control (DQC) – formerly comments review activity

Interview date took priority in both activities, as before

Priority shifted in both groups, in different directions

‣ Receipt focused on Rounds 1-3 to field for 2-4

‣ DQC had to focus on completed interviews with comments that needed program resolution, passing all other cases along to MPC processing

Result: MPC delivery dependent on completed interviews with authorization forms moving into MPC processing. With the two activities out of sync, 30% fewer provider/person pairs available for MPC delivery (quality metric)

39

Missed Tradeoff: Comments vs Authorizations (March)


40

Future Metric: Round 2 Response Rate Changes

90.0%

91.0%

92.0%

93.0%

94.0%

95.0%

96.0%

97.0%

98.0%

Week 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Fin

P22 94.5 95.4 95.5 95.5 95.4 95.2 94.8 94.7 94.5 94.1 94.0 93.8 93.7 93.9 94.0 94.0 94.0 94.1 93.3 94.5

P21 93.8 96.4 96.5 96.3 96.5 96.1 95.5 95.0 94.8 94.5 94.1 93.8 93.7 93.7 93.8 93.8 93.9 93.8 93.7 93.8

P20 94.8 96.5 96.9 96.8 96.7 96.4 96.0 95.8 95.4 95.1 94.9 94.6 94.4 94.3 94.1 94.0 93.8 94.0 93.8 94.8

P19 95.6 96.3 96.6 96.8 96.7 96.2 95.9 95.5 95.4 95.2 94.9 94.8 94.6 94.4 94.6 94.6 94.6 94.4 93.6 95.6

P18 95.9 97.3 97.4 97.3 97.3 97.4 97.1 96.8 96.6 96.5 96.3 96.0 95.8 95.6 95.5 95.3 95.1 95.0 95.0 95.9


41

More Efficiency Metrics Programmer hours per question,

per variable compared to estimate Data technician hours per comment

compared to previous years Interviewer hours and contact

attempts per completed interview by round


42

More Quality MetricsHelp desk calls per interviewer by week;

total by type by week Annual interviewer attrition rate

compared to prior years (about 20%) Interview administration time: total and

per section compared to previous years Benchmarking differences compared to

prior years (<3% is standard)


Some Final ThoughtsOn leadership, tradeoffs, and quality

Leader Requirements

Alertness to need for change Courage Toughness Strong management and

methodological skills

Organization Requirements

Flexibility and nimbleness Strong capabilities Adequate resources Culture with focus on quality

and measurement

44

Big Changes Need Strong Leaders and Support


Politics are important – building solid support and understanding its limits

Change more difficult than initiation in some ways – many constraints, sometimes change gets choked

Need for change must be balanced against risks

45

Tradeoffs in Making a Go/No-Go Decision


46

Managing a Quality Spiral

Data Harmonization

Data Collection

Pretesting

Sample Design

Questionnaire Design

Adaptation

Translation

Interviewer Recruitment, Selection & Training Instrument Technical Design

Data Dissemination Tenders, Bids & Contracts

Study, Organizational & Operational Structure

Ethical Considerations

in Surveys

SurveyQuality


Thank YouFor more information please contact:Brad Edwards | [email protected]

And thanks to Doris Lefkowitz (AHRQ) and Wendy Hicks (Westat) for useful comments on drafts of this presentation

mailto:[email protected]

Documents

Managing Survey Change - City · 2018. 11. 15. · Managing Survey Change Resource Tradeoffs and Quality Metrics. Brad Edwards . ESS Eric-City Methodology Seminar Series. April 16,