Upload
alexa-andersen
View
219
Download
0
Tags:
Embed Size (px)
Citation preview
The Roots of Total Survey Design
Lars LybergStockholm University
QMMS Seminar Leinsweiler, Nov 7-9, 2010
Early thinkers Hansen and colleagues, U.S.
Bureau of the Census Deming, U.S. Bureau of the Census
and consultant Kish, University of Michigan Dalenius, Statistics Sweden and
Stockholm University
What were they thinking about? Nonsampling errors Balancing errors and costs Design criteria The limitations of sampling theory Standards Similarities between survey
implementation and the assembly line
4
Deming (1944) “On Errors in Surveys”
American Sociological Review! First listing of sources of problems,
beyond sampling, facing surveys The 13 factors
Deming’s 13 factors
The 13 factors that affect the usefulness of a survey
-To point out the need for directing effort toward all of them in the planning process with a view to usefulness and funds available
-To point out the futility of concentrating on only one or two of them
-To point out the need for theories of bias and variability that correlate accumulated experience
6
Their difficult position They had to promote Neyman’s theory But his theory basically assumes very
small nonsampling errors They were in a first-things-first situation They promoted vigorous controls
hopefully leading to small biases They discussed what a Bayesian
approach might offer
Lines of thought I “There is as yet no universally
accepted ‘survey design formula’ that provides a solution to the design problem (Dalenius 1967)
That’s why textbooks devote little space to design
Important to control specific error sources
Lines of thought II The U.S. Bureau of the Census is a
statistical factory. The main product is statistical tables (Deming and Geoffrey 1941)
Concentration on QC of error sources, evaluation, and survey models
Disentangling the design process
Lines of thought III Hansen-Hurwitz-Pritzker 1967
Take all error sources into account Minimize all biases and select a minimum-
variance scheme so that Var becomes an approximation of (a decent) MSE
The zero defects movement that later became Six Sigma
Dalenius 1969 Total survey design
The design process Criterion of effectiveness: Minimum MSE
per unit of cost while meeting other requirements such as timeliness of results (not just minimum variance)
Good survey design calls for reasonably effective control of the accuracy through appropriate specifications for survey procedures and adequate control of the operations, i.e. proper design of the total system
Mean squared error (MSE) MSE=Var+B2+(Relevance
error)2+Interaction MSEZ(y)=E(y-Y)2+(Y-X)2+(X-Z)2
+2(Y-X)(X-Z)Z is the ideal goal, X is the defined goal,
and y is the actual result Hansen-Hurwitz-Pritzker call them
requirements (Z), specifications (X) and operations (y)
Design issues X-Z is crucial in the design
situation Do we want an approximate
solution to the right problem or an exact solution to the wrong problem?
The design approach (Dalenius and Hansen et al) Specify the ideal goal Z Analyze the survey situation
(financial, methodological and information resources)
Construct a small number of alternative designs
Evaluate the alternatives by reference to associated MSE and costs
The design approach (cont’d) Make a decision
Use one of the alternatives Use a modification of one of them Do not conduct the survey
Develop the administrative design Feasibility The signal system A self-contained design document (tree) Plan B
What does this tell us?
All error sources should be taken into account
There is very little process talk such as the need for CQI
However, the common situation was: no process view, no controls
Concern about costs and effectiveness of all these controls
The user is a somewhat distant player
The user The user was hiding under terms such
as subject matter problem, study purpose or “the four key functions of a statistical system” (reporting, analytic, consulting, research) Tukey 1949
But there were federal statistics users conferences in the U.S. from 1957. Dalenius provides more than 200 references on users in a 1967 ISI paper
Who identifies the requirements? Usually seen as one fictive person
An official An administrator A statistician acting as a subject-
matter specialist Requirements define the population,
types of measurement, time dimensions and statistics needed
The designer’s role vis-à-vis the requirements To critique the suggested
requirements To suggest QC procedures, construct
dummy tables to check the decision-making and perform sensitivity analysis
To act as the devil’s advocate and discuss specific result interpretations with the user
Kish’s contributions The neo-Bayesian view
Appreciates the literature by Schlaifer, Ericson, Edwards, Lindman and Savage on Bayesian methods in survey sampling and psychometrics
For instance, judgment estimates of measurement biases may be combined with sampling variances to construct more realistic estimates of the total survey error
More from Kish Experiments and sample surveys
might not be sufficient. Other investigations “collecting data with considerable care and control” but without randomization and probability sampling might be necessary.
Kish’s view on design Multipurpose is great from an economical
point of view. If one principal statistic can be identified
that alone can decide the design If a small number of principal statistics
can be identified a reasonable design compromise is possible
If statistics are too disparate a joint design might not be possible
Kish on economic design Requires joint consideration of sampling
and nonsampling errors Sometimes demands prior or pilot
studies of sufficient size Requires information about unit
variance Emphasizes a small total error Appreciates the fact that a reduction of
one source might increase total error
Examples of decisions Frame needs updating? Reference period? Acceptable respondent rules? Number of callbacks? Allocation of callbacks? How much and what kind of editing? Mix of modes?
Kish summed up Get a good balance between different
error sources We need to know how error structures
behave under different design alternatives Relevant information should be recorded
during implementation (paradata) Many practical constraints The multipurpose nature calls for a
compromise
Hansen, Dalenius and colleagues on standards General standards
Measurable survey plan, self-contained plan, replications should generate similar results, cost-efficient, sufficiently simple plan
Standards for error control Relevance control, control of accuracy (should
be dominated by variance terms) Minimum performance standards Check that standards yield the results expected
Hansen, Dalenius and colleagues summed up One should be guided by common sense,
experience and theory Design and execution is a management and
systems analysis problem A survey is an economic production process Survey goals must be identified Standards must be dynamic End the practice that sampling error is viewed
as the total error They predicted the CASM movement
More from Hansen, Dalenius and colleagues The examination of design alternatives is costly
and time-consuming There is a risk of overcontrol and inadequate
control. Consequences of large errors must guide any relaxation but they don’t talk about CQI
One might have to compromise relevance to get controllable measurements or abstain from the survey
Keep bias near zero and allow variance at expected levels
What happened? Still no “design formula” General design principles exist for some areas Still a concentration on some error sources
more than others CASM happened We got standards The TSE paradigm accepted but has some
promotional problems Many of the early thoughts were just that, very
little practice, but still useful