S1 Ciceral, A. (1982) Interviews, surveys and the problem of ecological validity.pdf

8/10/2019 S1 Ciceral, A. (1982) Interviews, surveys and the problem of ecological validity.pdf

1/11

Interviews, Surveys, and the Problem of Ecological ValidityAuthor(s): Aaron V. CicourelSource: The American Sociologist, Vol. 17, No. 1 (Feb., 1982), pp. 11-20Published by: American Sociological AssociationStable URL: http://www.jstor.org/stable/27702491.

Accessed: 28/10/2013 15:50

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at.http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of

content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms

of scholarship. For more information about JSTOR, please contact [email protected].

.

American Sociological Associationis collaborating with JSTOR to digitize, preserve and extend access to The

American Sociologist.

http://www.jstor.org

This content downloaded from 64.76.142.170 on Mon, 28 Oct 2013 15:50:13 PMAll use subject to JSTOR Terms and Conditions
http://www.jstor.org/action/showPublisher?publisherCode=asahttp://www.jstor.org/stable/27702491?origin=JSTOR-pdfhttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/stable/27702491?origin=JSTOR-pdfhttp://www.jstor.org/action/showPublisher?publisherCode=asa


2/11

The

Problem

of

Ecological

Validity

11

tions that

a

classic

can

serve.

We

may

believe

that students'

minds

are

expanded

by

reading

Durkheim

without

our

having

to believe Durkheim has many true gener

alizations

about the

causes

of suicide.

George

Herbert

Mead

can

symbolize

what

is

distinctive

in

symbolic

interactionism

even

if

we

cannot

quite

figure

out

how to

test

the

hypothesis

of

the

independence

of

the

F

from

the

me,

and

to

turn

it into

a

puzzle

for routine

science.

And

orte

can

enjoy

the taste

of

Marx's

famous

passage

in The 18th

Brumaire

about

French

peas

ants

forming

a

vast

mass,

without

that

beauty being undermined when we find

some

regions

of

modern

France

where the

peasants

vote

Communist.

What

is

destructive

about

admiration of

the

classics,

then,

is

the

halo

effect,

the

belief that

because

a

book

or

article is

useful

for

one

purpose,

it

must

have

all

the

virtues.

INTERVIEWS, SURVEYS, AND THE PROBLEM OF

ECOLOGICAL VALIDITY*

Aaron

V.

Cicourel

University

of

California,

San

Diego

The

American

Sociologist 1982,

Vol. 17

(February):

11-20

Despite

the

fact

that

virtually

all

social

science

data

are

derived

from

some

kind

of

discourse

or

textual

materials,

sociologists

have

devoted little

time

to

establishing explicit

theoretical

foundations

for

the

use

of

such

instruments

as

interviews

and

surveys.

A

key

problem

always

has been

the lack

of

clear theoretical

concepts

about the

interpretation

of

interview

and

survey

question and answer frames. We lack a theory of comprehension and communication that can

provide

a

foundation for

the

way

that

question-answer

systems

function,

and

the

way

respondents

understand

them. The

paper

briefly

describes

the

possible

relevance

of

linguistic

and

cognitive

processes

for

improving

our

understanding

of

interviews

and

surveys.

The

theoretical

foundations

of

interviews

and

surveys

also

must

address

the

way

that

artificial

circumstances

become

necessary

to

guarantee

adequate study

designs.

These

artificial

circumstances

often

violate

ecological

validity,

or

the

way

interviews

and

survey

questions

are

constructed,

understood,

and

answered,

as

contrasted

with

the

way

that

field

notes

and

tape-recordings

of

natural

settings

are

used

to

address the

same

or

comparable

substantive

and

theoretical

issues.

Social scientists

have relied

on

inter

views

for

a

long

time.

There

is

little

reason

to doubt their value and utility formany

theoretical

and

practical

purposes.

There

exists

a

huge

literature

on

the

virtues

and

drawbacks

of interviews

that

use

open

ended

questions

and

surveys

that

use

close-ended

questions.

Yet there is

little

in

the

way

of

theory

that

would

link

inter

*

Presented

at

the

thematic section

Fact

or

Ar

tifact:

Are

Surveys

Worth

Anything?

held

at

the

1980

American

Sociological

Association

Meetings,

New

York,

August

27,

1980.

The other

speaker

was

Howard

Schuman,

taking

a

less critical view of

sur

vey

research. I am

grateful

to Michael

Cole,

Roy

D'Andrade,

and

Hugh

Mehan for

their

valuable

re

marks and

suggestions

on a

much

longer

first draft

of

the

paper.

[Address

correspondence

to:

Aaron

V.

Cicourel;

Department

of

Sociology; University

of

California,

San

Diego;

La

Jolla

CA

92037.]

views and

surveys

to

more

general

issues

of communication and

comprehension.

Those researchers who are convinced that

interviews and

surveys

are

basic

research

tools for the

sociologists

are

concerned

about

improvements

in

survey

design

and

use,

but

see

little

point

in

challenging

their

routine

use.

In this

paper

I

want to

suggest

a

few

cognitive

and

linguistic

issues

that

can

clarify

our

understanding

of

the

pro

cesses

and

mechanisms

underlying

the

use

of interviews

and

surveys.

I

also

want

to

suggest

some

theoretical

ideas

that

can

strengthen

the

ecological validity

of

inter

view and

survey

methods and

findings.

The

necessity

of

writing

a

brief

paper

does

not

permit

me

to

discuss old

issues

about

current

interview

and

survey prac

tices

that

I

hope

are

obvious

to

http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp


3/11

12

The American Sociologist

sociologists:

for

example,

the

way

that

preliminary

qualitative

interviews

nor

mally

precede

the construction

of inter

view schedules and surveys; and theway

that

pre-testing

with

in-depth

interviews

helps

to

create

questions

that

respondents

can

understand,

while

helping

to

create

answer

categories

that

reflect

the

thinking

of

respondents

and

not

simply

the

re

searcher.

Hence

I will

avoid

discussing

the

range

of

practices

that

are

necessary

to

insure

quality

control,

such

as

using

different

types

of

questions

on

the

same

topic,

exploring

the

significance

of

changes

in

wording,

and

other

procedures

too numerous to mention here.

Interviews

and

surveys

usually

occur

within

a

broader

context

of interaction

that

includes

complex

cognitive

and

lin

guistic

activities within

a

set

of

in

stitutionalized

and

emergent

socio

cultural

constraints.

The

questions

used

in

surveys

almost

always

are

framed

in

a

textual

format

that

displays

features

in

common

with

I.Q., aptitude,

and

reading

tests.

Virtually all social science

data

are

de

rived

from

some

kind of

discourse

or

tex

tual

materials

such

as

reports,

written

ac

counts

of

observations,

interviews,

audio

or

video

recordings

of

natural

settings,

historical

or

contemporary

documents,

minutes

of

meetings,

newspapers,

maga

zines,

and

the

like.

Questionnaires

mailed

to

respondents

presuppose

something

about

the

way

people

are

able

to

analyze

textual

materials.

For

example,

in

re

sponding

to

questions

in

a

reading

test,

the

respondent must utilize several sources of

knowledge

that

the

researcher

used

in

interviews

and

surveys,

therefore,

pre

sumes

a

theory

of

communication

and

comprehension

that

seldom

is

addressed

by

sociologists.

The remainder

of

the

paper

will

suggest

aspects

of communica

tion

and

comprehension

that

can

help

es

tablish

some

theoretical

foundations

for

interviews

and

survey

research.

Aspects ofmemory and comprehension

presupposed

in

surveys

and

interviews

Recent work

on

learning

and

reading

comprehension

(Bransford

et

al., n.d.)

reminds

us

that

our

research

instruments

stringently

control

the

information

re

sources

available

to the

subject

or

respon

dent.

In

tests and

questionnaires

normal

procedures presuppose an agreement

or

social

contract

between

researcher

and

respondent:

the

contract

does

not

permit

the

use

of

other

persons

(nor

the inter

viewer)

in

order

to

decide

on

the

meaning

of

the

question

and

the

appropriateness

of

a

response.

Normal

group

or

peer

sources

of

help

are

blocked.

The

test

or

question

naire

item

is

assumed

to

be

self

explanatory

or

self-contained.

These

con

ditions

contrast

with

the

possibility

of

consulting

a

friend

or

colleague

or

return

ing to a textual report, newspaper article,

book,

and

the

like,

during

or

after

an

ini

tial

reading

of

the

text.

The interview

and

survey

seek

to

re

strict

the

question

frame,

and

in

the

case

of

surveys,

the

answer

frame.

The

goal

is

to

restrict

the

question

in

such

a

way

as

to

anticipate

and

even

designate

(in

surveys)

the

range

of

responses

that

can

be

used.

The

aggregation

of

responses requires

a

few choices

that

either

are

formally

part

of

the questionnaire

or

are

imposed

on

open-ended

responses.

We

impose

infor

mation

processing

restrictions

on

the

re

spondent

because

they

enable

us

to

aggre

gate

and summarize

a

large

amount of

in

formation

in

a

fairly

succinct

way.

But

we

pay

a

price,

and

we

need

to

understand

the

costs

in

order

to

improve

the

reliability

and

validity

of

interview

and

survey

data.

We

need

a

better

understanding

of

the

role

of

memory

and

the

way

questions

are

comprehended.

Norman

(1973)

has

noted

several relationships between the organi

zation

of

memory

and

answers

to

ques

tions:

a

question

may

not

evoke

an

appro

priate

recall

if

it

is

phrased

differently

from

the

storage

format.

Norman

calls

this

the

paraphrase

problem

because

the

best

answer to

a

question

may

be

the

use

of

another

question

by

the

respondent

in order

to

clarify

what

is intended

by

the

original question.

Norman's

reference

to

memory

brings

up

the

problem

of

how

people

store

information

and combine

general

and

specific

sources of informa

tion

in

order

to

reveal

what

they

think

is

addressed

by

the

question

they

are

asked.

Norman

is

interested

in

the

pre-processing

that

occurs

before

an

answer to

a

question



4/11

The Problem

of

Ecological Validity

13

can

be

produced.

In

the

case

of

surveys

we are

faced with

the

paradox

that the

respondents

are

not

encouraged

to

pro

vide us with reasons or explanations about

their

answers,

yet

such

information

gives

us

clues

about

how the

question

was

un

derstood.

Norman

(1973)

states

that retrieval

of

information from

memory

requires

con

struction

by

the

respondent

because

of the

paraphrase problem.

It

is

difficult,

there

fore,

to

show

the

way

questions

and

an

swers are

articulated

by respondents

be

cause no

simple algorithm

can

be iden

tified that would enable

us

to

specify

a

sequence of instructions or steps or ac

tions that

directly

connects

questions

and

answers.

We

need,

therefore,

an

under

standing

of the

comprehension

process

and

the construction

of

responses

in

inter

views

and

surveys.

Aspects

of

language

presupposed

in

surveys

and interviews

I want to

suggest

a

parallel

between

as

pects

of modern

linguistic theory

and

sur

veys.

Language

can

refer

to

a

lexicon and

individual

words

as

a

set

of

carefully

con

structed

ideals that

can

be studied

and

described

independently

of

actual lan

guage

use

in

social

settings.

The

linguist's

syntax-based

theory

of

language provides

the ideal

structures

the

survey

researcher

needs

for

constructing

standardized

ques

tionnaires

whose forced-choice

responses

can

be

analyzed independently

of

the

way

persons

in

daily

life

actually

discuss

or

pose and answer questions of each other

within the

constraints of

daily

practices

in

socio-cultural

organizations.

The

linguist's

normative

theory

of lan

guage

describes idealized

prescriptive

and

proscriptive

rules

that

are

recipes

for de

ciding

what

are

socially acceptable

or

un

acceptable

sentences.

Survey questions

are

equally

normative because

they

occur

in

highly

restrictive

settings

that

have little

or

nothing

to

do

with

actual

discussions

or

practices

in

group

or

informal

organi

zational activities. But the

linguist's

and

survey

researcher's

idealized

language

structures and substantive

questions

about the

world of

opinions,

attitudes,

be

liefs,

and

moral

judgments

are an

integral

part

of

the

way

researchers and

the

public

governmental

agencies

conceptualize

and

interpret

the

world

around

them.

The paradox

we

face is that

our

surveys

and

interviews

only

indirectly

reflect

as

pects

of

the

daily

life

settings

of those

we

interrogate, yet

these

instruments

can

be

the

basis for the

development

of

policy

by

organizations

in

many

complex

nation

states.

If

we

recognize

that

questions

and

an

swers

are

speech

acts

(Austin,

1962;

Searle,

1969),

we can

make

some

sug

gestions

about

the

way

that

the

structure

of

language

can

improve

our

understand

ing of interviews and surveys. Speech act

theory

seeks

to

combine

the

analysis

of

the

propositional

content

of

an

utterance

with

its

illocutionary

force;

the intention

of

a

speaker

to

act on

the

world

by

the

use

of

a

promise,

assertion,

command,

and

the

like.

Speech

acts

enable

the researcher

to

establish

a

functional

meaning

for

an

ut

terance

by

the

way

they

are

classified

as

statements

about

the

world,

a

speaker's

act

on

the

world,

or

a

symbolic

represen

tation

of

an

event

in

the world.

A

meta

language

was

felt

to

be

necessary

for dis

cussion of

speech

acts.

The

way

in which

surveys

and

inter

views

are

conducted

presupposes

a

model

of

conversational

behavior that

has been

ignored

by

most

sociologists.

The notion

of

a

speech

act

model

has

been

extended

to

the

idea

of conversational

postulates

by

Grice

(1975),

and

derived

from his

more

general

notion

of

the

cooperative

principle.

The

principle

refers

to

a

kind of

directive to the speaker to formulate all

aspects

of his

or

her

utterance

in

a

way

that

will

permit

participants

of

a

conver

sation

to

facilitate

to

the

utmost

the

achievement

of

the

explicit

and

tacitly

agreed

upon

aims of

the conversation.

Grice

identifies

four

categories

that

are

designed

to

orient

the

speaker

to

be

as

informative

as

possible

but

not

more

in

formative

than

seems

necessary.

Nor

should

the

speaker

say

anything

believed

to

be

false

or

anything

that lacks

sufficient

evidence.

The

speaker

also is

to be rele

vant

and

to

be

brief

and

orderly

while

trying

not to be

ambiguous

or

obscure

in

his

or

her

use

of

expressions.

Notice

that

the

term

speaker

would

apply

to

both the



5/11

14

The American

Sociologist

interviewer and

respondent

in

the

case

of

surveys

and

interviews.

Another theoretical

issue

contained

in

Grice's work includes the notion of con

versation

implicature.

In

everyday

con

versation,

listeners

are

expected

to

make

inferences that

do

not

necessarily

follow

from the

premises

or

statements

given,

yet

the

statements

are

necessary

for the

com

prehension

of

the

discourse

or

text. The

notion

of

conversational

implicature

is

central

to

surveys

and

interviews

as

sub

sets

of

discourse

and textual

activities

be

cause

our

questions

require

that

the

re

spondent

go

beyond

the

information

given

in survey questions and presume that the

utterances

can

be

expanded

in

order

to

pursue

their

implications

and

derive

co

herence from what is said.

Speech

act

theory

can

help

us

under

stand

the

way

that

variations

in

the

textual

content

and

structure

of

interview

and

survey

questions guide

the

kinds of

in

terpretations

that will

be

made,

and how

these attributions of

meaning

will

influ

ence

the

construction of

a

response

or

the

selection of

an

option

in

a

set

of forced

choice

responses.

Some

aspects

of question-answer

systems

Question-answer

systems

deal with

a

sub-set of

speech

act

theory.

The

ques

tions

employed

are

for

themost

part

direct

attempts

to

elicit information.

In

everyday

English

we

often

use

indirect

speech

acts

and

our

questions

do

not

always

follow

an

interrogative

format.

Scheduled inter

views and surveys presume a respondent

who is

aware

of

the

general procedure.

A

specific

style

of

interrogation

is

used and

respondents

assume

a

response

stance

that

differs

markedly

from routine

every

day

conversation. But this

response

set

is

quite

similar

to

occasions

when

someone

is

being

interviewed for

a

job

or

is

taking

a

test.

The formal

aspects

of

conversation

out

lined

by

Grice

indirectly parallel

some

formal

properties

of

question-answer

systems

described

by

Harrah

(1973).

Whereas Grice

specifies

some

general

conditions

governing

all

conversations,

including

aspects

of the

reasoning

neces

sary

for

successful

exchanges,

Harrah

in

dicates

conditions

that

are

especially

rele

vant to

the

questioner.

In

Harrah's

(1973)

model

the

ques

tioner:

(a)

Is

presumed

to

know

what the

problem

is

about.

(b)

Knows

how

to

express

the

question

in

an

effective

manner.

(c)

Knows what

the

set

of

possible

alterna

tives

can

be.

(d)

Can

claim

that

one

of

the

alternatives

is

true.

(e)

Does

not

know

which alternative

he

wants

to

know.

(f)

Believes

the

respondent

can

help

him

if

particular

question

is

put

properly.

But

there

are

various

logics

of

questions,

notes

Harrah,

and

they

will

vary

according

to

the

social situation

in which

questions

are

used.

Thus

in

a

classroom

setting

a

teacher

puts

questions

to students

and

knows

the

answers

expected.

Harrah

de

scribes

some

of

the

conditions

of

a

Ph.D.

examining

committee

where

their

ques

tions

may

be

directed

as

much

to

each

other

as to

the

student.

In the

Ph.D.

ex

amination

the

knowledge

base of

the

re

spondent

is

presumed

to

be the

problema

tic

issue.

When

a

physician

asks

questions

of

a

patient,

the

knowledge

base of

the

respondent

may

be viewed

as a source

of

new

information.

But this

information

re

quires

particular

types

and

sequences

of

questions

and

answers.

The

patient's

an

swers

also

may

become

problematic

de

pending

on

the

kinds

of

attributions

made

to

the

patient

because

of

age,

mental

status,

social

position,

and

physical

con

dition and appearance, tomention a few

key

variables

that

affect

all

question

answer

systems.

Questionnaire

items

are

not

merely

in

dividual,

self-contained

texts,

but become

the

basis

for

inferring

macro-structures

that resemble

those

reported

by

re

searchers

working

on

text

comprehension

(Kintsch

and

van

Dijk,

forthcoming;

van

Dijk,

1972).

The

respondent

seeks

a

more

comprehensive

understanding

of

the

dif

ferent

questions

asked

despite

the

re

searchers

attempts

to randomize the pre

sentation

of

questions

that

are

linked

by

hypotheses

in the

research

design.

This

search

for

a

pattern

on

the

part

of

the

respondent

is

part

of

an

attempt

to

create



6/11

The

Problem

of

Ecological

Validity

15

an

explanation

that

goes

beyond

whatever

cryptic

information is

supplied by

the

interviewer

or

questionnaire.

The

respon

dent becomes an active participant in the

survey

or

interview

and

seeks

to

develop

his

or

her

own

hypotheses

about

what

is

going

on

and

what

intentions the

re

searcher

projects

by

the

kinds

of

ques

tions asked.

Discourse and

textual

analysis

as

comprehension

A

key

issue

in

the

study

of

comprehen

sion is

trying

to

estimate

what

the

respon

dent brings to the reading test, interview,

or

survey.

There

are

several

strategies

available for

studying comprehension.

In

unpublished

work

by

David

Rumelhart,

comprehension

by

subjects

is

measured

by

asking

them

to

read

a

story

line

by

line

while

indicating

what

is

happening

after

each

line.

An

unpublished

project

in

prog

ress

on

reading

comprehension by

Charles

Fillmore

and Paul

Kay

uses

reading

test

items

(that

strongly

resemble

survey

questions)

as

the basis for interrogating

the

child

about

his

or

her

understanding

of

the

text

of

each

test

question.

More

abstract

types

of textual

analysis

include

the identification of

topics

or

themes

and

their

continuity

over

a

large

textual domain

(Grimes,

1980).

A

large

and

growing

literature

exists

here

that

ad

dresses

textual

analysis

(Dressier,

1977;

Halliday,

1967).

I

will

not

attempt

to

re

view

current

work

on

the

analysis

of

texts

but

only

mention

the

broad

goal

of

iden

tifying those structures thatwould serve

as a

basis for

interpreting specific portions

and

general

aspects

of

a

text.

Particular

conventions

of

language

use,

such

as

fol

lowing

certain forms of

language

structure

when

writing

a

letter,

a

report,

in

structions

for

filling

out

the

necessary

pa

pers

for

a

bank

loan,

and

the

like,

generate

expectations

about how

topics

are

intro

duced,

developed,

and

terminated.

We

need to

know how interview

and

survey

questions

as

texts

are

interpreted by

the

researcher

and

respondents.

Current work

on

the

analysis

of discourse and textual

materials

can

help

us

develop

a

theoretical

foundation for

understanding

and

im

proving

interviews. The theoretical

foun

dations of

interviews and

surveys

must

include

the

way

that

artificial

circum

stances

necessary

to

guarantee

adequate

study designs can violate the ecological

validity

of

findings

vis-?-vis

what

takes

place

in

daily

life

settings.

Restating

the

ecological validity

issue

for

sociology

Social scientists

often

are

so

pre

occupied

with

creating

an

adequate

study

design

that

they

overlook

the

ecologi

cal

validity

problem:

Do

our

instruments

capture

the

daily

life

conditions,

opinions,

values, attitudes, and knowledge base of

those

we

study

as

expressed

in

their

natu

ral habitat?

Recent work

by

Cole

et

al.

(1978)

re

views the

history

and current

efforts

to

deal

with

the

problem

of

ecological

va

lidity.

Cole

et

al.

refer

to

the

revolutionary

impact

of

Wundt's

laboratory

psychology

for the

experimental

study

of mind in

arti

ficially

constructed and

simplified

envi

ronments.

Can

we

extend the

elegance

and control of laboratory research to field

settings?

In

sociology

it is

difficult

to

study

everyday settings

while

using

carefully

formulated

surveys.

Much of

survey

re

search

can

be viewed

as

the

application

of

rigorous techniques

to

data elicited

in

simplified

and

artificial social

envi

ronments.

In

sociology

and

psychology

ecological

validity

remains

a

minor issue

because

studies of

the social

organization

of

the

laboratory and the interview or survey

settings

often

are

relegated

to

minor

roles

when data

are

analyzed.

Psychologists

have demonstrated renewed interest

in

pursuing laboratory-derived problems

in

natural

settings

and

incorporating

every

day

tasks

into

laboratory settings.

In

sociology

this

would

mean

contrasting

the

way

interview

and

survey

questions

are

constructed,

understood,

and

answered,

with the

way

that field

notes

and

tape

recordings

of

natural

settings

are

used

to

address

the

same or

comparable

substan

tive and

theoretical

issues.

Psychologists

are

sensitive

to

problems

associated

with task

definition,

mental

overload,

and

possible

differences

in

the



7/11

16

The

American

Sociologist

way

remembering, thinking,

and

attending

to

activities

occur

in

laboratory

and

non

laboratory

settings.

But

they

do

not

study

the cultural definitions of everyday life

that

are

part

of the

laboratory

setting

and

that

are

invoked

necessarily

when

at

tempting

to conduct

controlled

research

in

non-laboratory settings.

Psychologists

are

not

sensitive

to

what

their

subjects

must

be

able

to

do

to

make

themselves

appear

as

normal

members

of

a

group

and the

larger

society

in

order

to

perform

ade

quately

in

an

experiment

or

in

an

every

day

setting.

The

subjects

and the

exper

imenter

both

must

rely

on

their

everyday

tacit knowledge in order to satisfy stated

and

unstated social conditions

that must

be

followed

if

the

research is

to

be

consid

ered

successful.

Sociologists

are

sensitive

to

the fact

that

many

problems

are

associated with the

way

questionnaires

are

administered,

coded,

and

organized

for

analysis.

But

they

are

insensitive

to

the

information

processing

problems

associated with

these

tasks.

Because

so

many

surveys

are

done

in the

same

culture in which the re

searchers

also

are

native,

and

because

we

gradually

have

socialized

our

respondents

to

be

fairly

docile

to

the

demands

of

sur

veys,

especially

since

everyday

life

cir

cumstances

often force them

to

submit

to

such

activities,

we

have little

knowledge

about

the

social

practices

of

survey

re

search

within

field

settings

and

within

re

search

centers

where

the

analysis

takes

place.

When

we

administer

surveys

in

other

cultures

we

incorporate

natives who

have been trained in the same method and

who

can

tacitly negotiate

the cultural

dif

ferences.

During

everyday

interaction

the

mem

bers

of

a

group

who

routinely

discuss

political,

economic,

and

social

events

are

sensitive

to

group

resources

of informa

tion

and

the

interpersonal

constraints that

are

imposed

on

exchanges,

and also

are

aware

of

the

knowledge

limitations of the

members

of the

group.

The

ecological

va

lidity

issue addresses

the

extent to

which

responses

to

interview

and

survey

ques

tions

reflect

or

represent

the

daily

actions

of

a

collectivity.

We

must

compare

the

way

collective discussions

about

topics

covered in

interviews

and

surveys

parallel

or

differ from

the

way

such themes

are

presented

in

the

formal

setting

created

by

research

goals.

A partial examination of the ecological

validity

issue

can

be found

in

recent work

by

Schuman

(1966),

and Schuman

and

Presser

(1977;

1977-1978; 1979;

forth

coming)

where

they

show that

changes

in

the

wording

of the

questions

often

lead

to

changes

in

response

patterns.

The work

by

Schuman

and

Presser also

contains

valuable

information

on

differences

in

the

use

of

open

and closed

questions.

The

authors

(forthcoming:9-10)

note

that

if

re

spondents

are

given

a

question

about

which they know nothing, many will an

swer

the

question

if

there

is

no

explicit

don't

know

option.

Many respondents

are

willing

to

admit

ignorance.

Respon

dents

also

are

said

to

make

an

educated

(though wrong) guess

about

a

topic

de

spite

their

being

uninformed

about

the

issue. The authors

note that

these

re

sponses

are

like

non-attitudes

in

the

sense

that

there

probably

was

no

prior

thought

about

the

attitude before

pre

sented by the interviewer.

The studies

by

Schuman

and Presser

seek

to

resolve

possible

problems

in

inter

viewing

and

surveys

as a

way

of enhanc

ing

their

reliability

and

validity.

These

studies,

though

limited

in

scope

and

depth,

are

valuable contributions

to

the

minimal work

that

has

been

done

by

re

searchers

working

within

social science

who

strongly

support

these methods

in

their

existing

form.

Many

serious

prob

lems of

reliability

and

validity

remain

be

cause of normal practices that are devoid

of

adequate

concern

for theoretical

foun

dations.

For

example,

we

restrict

our

col

lection of

information

to

a

few

categories

in

order

to

restrict

the

number

of

compari

sons

we

have

to

make. The

respondent's

knowledge

of the world

is

not

a

problema

tic

issue.

In

a

laboratory study

or

in

sur

veys

with

fixed-choice

questions, subjects

or

respondents

bring categorical

mech

anisms

with

them,

but

the

actual

pack

ages

of information

they employ

must

be

either recoded or tailored to the

particular

conditions

of

the

experiment

or

survey

question

and

forced

choices

provided.

The

range

of

speech

acts

becomes

se

verely

restricted

in

survey

research.

The



8/11

The

Problem

of

Ecological

Validity

17

respondent's

cultural

resources

available

for

searching

one's

memory

are

con

strained,

and

this

limits the

respondent's

ability tomake comparisons. The survey

question

introduces co-variation

by

the

wording

of

the

question.

The

following

examples

are

from

a

national

survey

by

E.

C.

Ladd,

Jr. and

S.

M.

Lipset

(as

cited

by

S.

Lang

(1978)).

The

questions

were

sent

to

college

and

university professors.

(1)

The

statements

below relate

to

teaching

and

student

performance.

Does each

correctly

reflect

your per

sonal

judgment?

(i)

The

students

with

whom

I

have

close

contact

are

seriously underprepared

in basic

skills?such

as

those

re

quired

for

written and

oral communi

cation.

(a)

Definitely

yes

(b)

Only partly

(c)

Definitely

no

(2)

Grade

inflation

is

a

serious academic

standards

problem

at

my

institution.

(a)

Definitely

yes

(b)

Only partly

(c)

Definitely

no

(3) American higher education should ex

pand

the

core

curriculum,

to

increase the

number

of

basic

courses

required

of all

undergraduates.

(a)

Definitely

yes

(b)

Only partly

(c)

Definitely

no

The

questions

assume

a

sample

of

re

spondents

familiar

with

the

content

of

the

items

(in

the

present

case

American

col

lege

and

university

professors)

as

well

as

with the

meaning

of

definitely

yes,

only

partly, and definitely no. The questions

and the

responses

reflect

a

co-variation

that

enables

a

reader

to

parse

the

question

easily.

The

answers

permit

easy

aggrega

tion

so

that

these

and

other

questions

can

be

cross-tabulated with

size

of

school,

age

of

respondent,

discipline

or

field of

study

and

the

academic

rank

of

the

respondent.

In

the

present

case

these

questions

were

mailed

out

to

the

informant.

There

was

no

way

for

the

respondent

to

obtain

informal

clarification

about

questions

from

the

interviewer. The

problem

solving

aspects

of

answering

a

fixed-choice

question

are

severely

limited.

There

were

no

open

ended

questions

to

make

it

possible

to

ob

serve

some

elements

of

a

problem-solving

strategy

at

work

(including

the

limitations

of the

respondent's

knowledge

base)

as

the

interviewee and

interviewer

negoti

ated the questions and answers. The

questions

on

the

basic

skills of

students,

grade

inflation,

and

the

idea

of

expanding

the

core

curriculum attribute

an

expertise

to

the

respondent

that

cannot

be

chal

lenged.

There

is

no

possibility

of

exploring

the individual

experiences

of

teachers

within

the

same

subject

area

much

less

across

disciplines.

Nor

can

we

distinguish

respondents

by

the level

of

college

or

uni

versity

classes

they

teach.

We

have

no

information

on

the

background

of their

students.

The

categories

created

by

the

re

searcher

must

be

negotiated

individually

by

each

respondent.

But the researcher's

categories

provide

ready-made

classes

and the

response

set

generates

automatic

criteria for

deciding

class

membership.

It

is difficult

to

interpret

the

meaning

of

these

personal

judgments

vis-?-vis

the

ex

periences

and

knowledge

base of

a

pre

sumed

group

of

experts.

The

expertise

is an automatic creation of identifying

a

population

known

as

being

college

and

university

teachers.

The

responses

we

obtain,

however,

remain

ambiguous

per

sonal

judgments.

The

ecological

validity

of

the

response

is

not

addressed.

Class

membership

and

interview

and

survey

categories

An

important

aspect

of

surveys

and

open-ended

interviews

is the

extent

to

which a concept or class membership is

presupposed

in

the

way

we

elicit

infor

mation

from

respondents.

Our

ability

to

perceive,

remember,

and talk

about

some

object

or

event

as an

instance

of

a

class

or

concept

we

are

presumed

to

know

is

fun

damental

to

the

way

we use

surveys

and

open-ended

questions.

Public

and

private

bureaucracies

so

cialize

their

employees

to

the

use

of

categories

that

can

subsume

a

variety

of

activities

under

identifiable

classes

and

thus confer a

stability

on the world that

enables

them

to

go

beyond

the

informa

tion

given.

We

negotiate

the

assignment

of

an

object

or

event

or

some

aspect

of in

formation

to

a

class

initially

on

the

basis of



9/11

18 The American

Sociologist

expectations

of its

perceptible

elements,

and

then

begin

to

infer

some

of the

ob

ject's

or

event's

nonperceptible

attributes.

This is like saying thatwe construct the

identification of

typical

features that

en

able

us

to

claim

class

membership

for

an

object

or

event,

or we

claim

that

certain

features

suggest

one

or

more

possible

classes that

trigger

a

search

for additional

elements

that

enable

us

to

choose

among

several

possible

candidates of

classes. In

our

perception

of

speech,

for

example,

we

may

be forced

to

imagine

or

recall

or

search

for

information

that

extends be

yond

conventional

or

dictionary interpre

tations of what is said because of the so

cial

setting.

Our

ability

to

make

the

non

perceptible

visible

and

hence

integral

to

the

invocation of

a

class

or

concept

is

a

basic

process necessary

for

all

social

in

teraction

and

bureaucratic

practices.

A recent

paper

by

Medin

and

Smith

(in

press)

distinguishes

between three views

of

concepts.

The

first

or

classical

view

requires

that

a

concept

have

common

properties

which

become

necessary

and

sufficient to define the concept. Every

member

of

a

class

can

be

specified by

the

properties they

all

must

possess

through

a

single

description

of

all

members.

Medin

and

Smith

note

that

attacks of this view

revolve

around

the

properties

of

descrip

tion that

must

be

true

of

all

members.

The

second

view

of

concepts

is called

the

prototype

or

poly

the

tic

position.

The

focus

of

this

view

is

on

the

way

instances

of

a

concept

can

vary

in

th?

degree

to

which

they

all

share

certain

properties.

This view says that the different instances

can

vary

in

the

extent to

which

they

will

embody

the

concept.

A

single

description

of

some

may

again

suffice,

but the

prop

erties

of the

description

are now

only

true

for

most

but

not

all

of

the

members.

Some

instances

of

the class

will

possess

more

of

the

critical

properties

than

will

other

in

stances.

Those

instances

possessing

more

critical

properties

are

said

to be

more

rep

resentative

of

the

concept

in

question.

The third

view

described

by

Medin

and

Smith is called an

exemplar

notion.

This

view

of

a

concept

states

that

no

single

representation

exists

for

an

entire

class

or

concept,

but

instead

only

specific

repre

sentations

of

the

class's

exemplars

occur.

The

example

given

by

Medin and

Smith

for

this

third view is

that

of

the

class

of

persons

who

might

be

called

suicidal.

The example of persons who might be

suicidal is used

by

Medin and Smith

to

compare

the three views

briefly.

They

note

that

the

classical

view failed when

used

by

clinicians because

no

necessary

and

sufficient

common

properties

could

be found

to

define

all

people

who have

suicidal

tendencies.

The

polythetic

view

fails, say

Medin

and

Smith,

because

it falls

short of

revealing

how

someone

decides

that

a

particular

person

is

suicidal.

Fol

lowing

a

suggestion

by

Twersky

and

Kahneman (1973), Medin and Smith con

tend

that

because

clinicians

are

not

likely

to

use a

single

description

of

all

persons

with

suicidal

tendencies,

they

might

in

stead

make the

decision

about

someone

being

suicidal

by comparing

the

individual

to

other

persons

known

to

be suicidal.

The

exemplar

view

would

result

in

the

class

of

people

with

suicidal

tendencies

being

rep

resented

by

separate

descriptions

for

various

people

known

to

be members

of

the class of suicidal persons.

Two

key questions

raised

by

Medin

and

Smith here

are

as

follows:

(1)

Is

it

possible

to

have

a

single

or

unitary

description

for

all

members

of the

class?

(2)

Can

we

say

that

all

of the

properties

specified

in

a

unitary description

are

true

of

all

members

of the class?

According

to

Medin

and Smith

the

classical

view

would

answer

both

ques

tions

with

an

affirmative

response,

while

the polythetic view would say yes to (1)

and

no

to

(2).

The

exemplar

view

would

say

no

to

(1)

and consider

(2)

to

be

irrelevant.

When

we use

fixed-choice

question

naire items

the

respondents

are

expected

to

be

able

to

recognize

the

classes

of ob

jects

stated in each

item

as

self

explanatory.

This

expectation

derives

from the

assumed

pretesting

of

each

questionnaire

item

prior

to

sending

out

or

utilizing

the final

questionnaire.

The

ex

tent to

which

the

respondent

possesses

the

necessary

knowledge

base

in

order to

answer

the

question

is

seldom

a

relevant

issue. The

possibility

that the

concepts

or

classes

presented

to

the

respondent

may



10/11

The

Problem

of

Ecological Validity

19

not

be

defined

clearly

in his

or

her

mind

is

also

seldom

an

issue.

In

both

cases

the

forced-choice

nature

of

the

survey

guar

antees an adequate response so long as

the

respondent

is

willing

to

check off

one

of

the

choices

available. The researcher's

versus

the

respondent's

conception

of

a

particular

class

or

concept

is

presumed

to

be

resolved

by

the

pretest

done

to

the

final

questionnaire.

The

issue of

possible

social

classes used is

not

directly

testable. What

is

testable is

the

way

different

respondents

can

be

distinguished

by

some

measure

of

social

class

as

determined

by

the

way

fixed-choice

questions

are

constructed,

answered, and coded.

Having

created

several

social

classes

by

one

set

of

ques

tions,

we

examine

other

questions

an

swered

differentially

and

can

attribute

the

differences

to

social

class

membership.

Surveys

are

presumed

to

be

hypothesis-driven

and

hence

a

way

of

testing

theory.

But

the

technical

aspects

of

the

instrument

make

it

difficult

to

clarify

theoretical

concepts

or

classes

said

to

have

motivated

the

use

of

the

survey.

Theoretical concepts are subservient to

the

mechanics

of

creating

and

imple

menting questions

and

their

coding.

The

discourse

and textual

processes

and

mechanisms that

provide

the

theoretical

basis

for

surveys

remain

unexamined,

as

are

the

social

constraints

and

practices

of

the

society's

social

stratification

system

that enable

the

researcher

to

utilize

such

an

esoteric

and

indirect instrument

to

learn

about

the

everyday

activities and

beliefs of the

members

of

a

group.

Conclusion

Part

of

the

paper

has

been critical

of

survey

research.

I

have

slighted

several

issues.

For

example,

conscientious

survey

researchers

seek

a

form of

quasi-experi

mentation

with

questionnaire

items

in

which

a

particular

question

frame

is

re

tained but

a

particular

word

(or

perhaps

phrase)

is

altered.

Important

differences

often

are

found

because

of

these

changes

(cf.

Schuman and

Presser,

1981),

even if it

is

not

always

clear what

sort

of

reasoning

we

should

attribute

to

the

respondents.

What

is

important

for

the

survey

re

searcher

is

the

patterning

that

occurs or

emerges

that

gives

us more

confidence

in

the

overall

survey.

When

the

same

ques

tions

are

used

across

different

groups

or

with the same group at different times,

and

similar,

or

the

same,

patterning

emerges,

then

the researcher

feels consid

erably

more

confident

that

the

question

naire is

reflecting something significant

about

the

respondents'

opinions,

atti

tudes,

or

beliefs.

Knowing

that

some

identifiable

group

was

more

in

favor

of

some

action

or

law

this

year

than last

year

is

part

of the formal

patterning

that is

sought.

There still

can

be

problems

here

when

particular

researchers

throw

out

items that do not seem to work, but the

general

idea

is

to

avoid

using question

naire

items

as

if

responses

on a

given

oc

casion

can

be

treated

in

some

absolute

way.

The

goal

is

to

look

at

some

group

or

category

relative

to

other

groups

or

categories

at

specific

periods

of

time,

and

not

how

many

respondents

endorse

a

given

item

at

a

particular

time.

The

survey

researcher

seeks

to

control

the

way

a

data

base is

generated

by

creat

ing restricted conditions under which in

formation is

to

be

elicited,

coded,

and

an

alyzed.

The

conditions

simplify

and

dis

tort

the

daily

life activities

of

those

groups

and

institutions

we

seek

to

understand

and

predict,

but

the

controls and

restricted

data

base

are

highly

valued

by

many

social

scientists

because

they

foster

a sense

of

scientific

rigor

in

our

research.

Another

source

of

control

in

survey

re

search

can

be

found

in

the

enormous

ad

vances

that

have

occurred

in

sampling

theory and the researcher's ability to sam

ple

different

respondents.

What is

more

difficult is

the

sampling

of behavior. In

the

case

of

voting

behavior

we

find

a

fairly

close

correspondence

between what

people

say

in

response

to

a

questionnaire

item and the

way

they actually

vote.

But

other

topics

do

not

fare

as

well,

and

some

not

well

at

all. We

are

not

clear about the

behavior

or

activities

the

survey ques

tionnaire items

are

said

to

index.

People

are

not

very

accurate in

describing

their

own behavior when asked to

respond

to

direct

questions.

The

primary

difficulty

remains the

absence

of

strong

theories.

Instead of

using

strong

theories

we

invari

ably rely

on

the detection

of

patterning

in



11/11

20

The

American

Sociologist

survey

responses

in

order

to

guide

us

in

making

theoretical

explanations

after

the

fact.

Theory

seldom

guides

social

re

search explicitly; we depend

on

research

findings

to

decide

which

theoretical

con

cepts

seem

appropriate.

Sophisticated

survey

researchers

surely

can

find answers

or

replies

to

the

is

sues

I

have

raised

and

will

point

to

the

use

of

other

sources

of data

or

additional

checks

or

strategies

that

I

have

not

cov

ered in

the

paper.

W?

need

strong

theories

to

decide

whether

a

particular

method

and the data

it

yields

tells

us

something

worth

know

ing.We all are forced to deal with the

same

problem

of

interpretation

regardless

of whether

we use

surveys,

census

mate

rials,

vital

statistics,

extensive

interview

ing, participation

observation,

or

audio

or

video

tapes.

The

interpretation

issue is

seldom the focus of

survey

research,

much

less

any

other

type

of

research

in

sociology.

Notions like limited

capacity

processing,

comprehension

of discourse

and textual

materials,

and

language

use

in

socially constrained contexts,

remain

pe

ripheral

topics

in

sociology.

Yet

they

ad

dress the

interpretation

issue

in

several

explicit

ways.

Can

we

afford

the

con

venience of

ignoring

these

issues?

REFERENCES

Austin,

John

L.

1962

How

To Do

Things

With

Words. Cam

bridge,

MA:

Harvard

University.

Bransford,

John

D.,

Barry

S.

Stein,

Tommie S.

Shelton,

and

Richard A.

Owings

n.d.

Cognition

and

adaption:

The

importance

of

learning

to

learn.

Unpublished

manuscript.

Cole,

Michael,

Lois

Hood,

and

Ray

P.

McDermott

1978

Ecological

niche

picking:

Ecological

inva

lidity

as an

axiom

of

experimental cognitive

psychology.

Unpublished

manuscript.

Dressier,

Wolfgang

U.

(ed.)

1977

Current

Trends in

Textlinguistics.

Berlin,

NY:

de

Gruyter.

Grice,

H.

Paul

1975

Logic

and

conversation.

Pp.

41-58 in

P.

Cole and

J.

Morgan

(eds.),

Syntax

and

Semantics.

New

York:

Academic.

Grimes,

Joseph

E.

1980

Context structure

patterns.

Paper

pre

sented at the Nobel Symposium on Text

Processing,

Stockholm,

August.

Halliday,

Michael

A. K.

1967

Notes

on

transitivity

and theme

in

En

glish.

Journal

of

Linguistics

3:199-244.

Harrah,

David

1973

The

logic

of

questions

and

its relevance

to

instructional

science. Instructional Sci

ence

1:447-467.

Kintsch,

Walter and

Teun A.

van

Dijk

forth-

Toward

a

model of

text

comprehension

coming

and

production. Psychological

Review.

Lang, Serge

1978 The

professors:

A

survey

of

a

survey.

The New York Review of Books 25:38-42.

Medin,

Douglas

and Edward

Smith

n.d.

Representation

and

processing

of

lexical

concepts.

Unpublished

manuscript.

Norman,

Donald A.

1973

Memory,

knowledge,

and

the

answering

of

questions. Pp.

135-165 in

R.

S.

Solso

(ed.),

Contemporary

Issues

in

Cognitive

Psychology:

The

Loyola

Symposium.

Washington,

DC:

Winston.

Schuman,

Howard

1966

The

random

probe:

A

technique

for

evaluating

the

validity

of closed

questions.

American

Sociological

Review

21:218-222.

Schuman, Howard and Stanley Presser

1977

Question

wording

as an

independent

vari

able

in

survey

analysis. Sociological

Methods

and

Research

6:151-170.

1977-

Attitude

measurement

and

the

gun

control

1978

paradox.

The Public

Opinion

Quarterly

41:427-438.

1979

The

open

and

closed

question.

American

Sociological

Review 4:692-712.

1981

Questions

and Answers

in Attitude Sur

veys:

Experiments

on

the Effects

of

Ques

tion

Form,

Wording

and Context.

New

York:

Academic Press,

forth-

Public

opinion

and

public

ignorance:

The

com- fine line between attitudes

and

non

ing

attitudes.

American Journal

of

Sociology.

Searle,

John

R.

1969

Speech

Acts.

Cambridge:

Cambridge

Uni

versity

Press.

Tversky,

Amos and

Daniel Kahneman

1973

Availability:

A

Hueristic

for

judging

fre

quency

and

probability.

Cognitive

Psy

chology

5:207-232.

van

Dijk,

Teun

A.

1972

Some

Aspects

of Text

Grammers.

The

Hague:

Mouton.

Documents

S1 Ciceral, A. (1982) Interviews, surveys and the problem of ecological validity.pdf