Oct. 31st, 2005Lisa A. Lauxman, Ph.D

NAE4-HA

The Art and Science Of Creating Useful Surveys

Arizona Cooperative Extension

Survey Design Fundamentals

• Survey is a method of gathering data

• Information collected may include people’s ideas, feelings, plans, beliefs, social, educational, and financial background.

• Questionnaires and interviews are the most commonly used surveys

Survey Purposes

• Survey purposes and methods fall on a continuum. Some have far-reaching generalizable effects

• Other are conducted to meet specific needs

• Identify purpose of survey• Program evaluation• Research• Policy needs

Survey & Questionnaire Components

Questions Instructions Sampling and design Data processing and analyses Pilot testing Response rate Reporting results

Most Common Survey Designs

Cross sectional: Data collected at one point (called a census if all members of a population are surveyed)

Most Common Survey Designs

Longitudinal: Data collected over time (note difference between sampling from a population and following the same people over time

– Trend: Sample a population over time e.g. 17 year olds in 1995 and 17 year olds in 2005

– Cohort: Same population sampled over time e.g. 17 years old in 1990 and 27 years old in 2000

– Panel: Same individuals over time e. g. the 17 year olds in 1995 and same people in 2005

Types of Questions

Close-ended survey items Open-ended survey items: Do you get any

satisfaction? – Find out whether people are satisfied with

product, service or program– Are particularly appropriate when collecting info

about what people like best or least

Writing Items

Survey items may take the form of questions– Example of open-ended question

How courteous are the people who make your appointments?

Survey items may be worded as statements– Example of close-ended statement

Circle your agreement or disagreement with the following: 2) The people who make my appointments are

(a) definitely courteous(b) agree(c) disagree)(d) definitely disagree

Rules for Writing Forced Choice

Each question should be meaningful to respondents

Use standard English Make questions concrete Avoid Biased words or phrases Check your own biases Each question should have only one thought

Examples of Scales Used for Forced Choice

Yes or No Checklist- provides a series of answers.

Respondents may choose just one or more answers Rating scales- 4 types

– Nominal or categorical responses, i.e.. Male or Female– Ordinal- require respondents to place answers in a rank

order, i.e.. High, medium or low – Interval – These have measurement choices with real

meaning between the numbers, i.e.. Annual income– Ratio- Has real meaning between numbers with

equidistance between the numbers. Ex- height and weight

Putting Questions in Order

All surveys should be preceded by an introductions First set of questions should be related to topic described Ask relatively objective questions before subjective ones Move from familiar to the least Follow natural sequence of time See to it that all questions are independent Relatively easy-to-answer questions should be asked at end Avoid items that look alike Sensitive questions should be placed well after the start but

also well before the end Questions should be asked in logical order

Response Scales-Likert-Type

Likert-type Scales– Balance all responses ex. 4, 6– Use neutral responses (no response, don’t know)– Use 4 to 7 point scale– Put negative end of scale first– Use meaningful scale

Endorsement ( Def. True > Def. False) Frequency ( Always > Never) Intensity ( Severe > None) Influence ( Big Problem > No Problem Comparison ( Much more than others > Much less than

others

Test the Survey

Ease of recording responses Check for typos, misspelled words Length Understanding of questions- easy? Difficult? Test for readability Are questions appropriate for people who will be

surveyed? What about data collected? Does it make sense?

How useful is the data? Check for cultural sensitivity

Response Scales- Checklist

Use ranking only if respondents can see or easily remember all choices

Respondents over-report frequent behaviors Big events are more easily remembered Use list to help jog memory (Yes/No to each)- Break

into components Behavior

– Time– Duration– Frequency

RELIABILITY

Test-retest reliability– Most commonly used; same respondents, same survey,

different time; correlation of .70 or better is generally considered accepable; item should NOT be likely to change over time; beware of “practice effect” which can falsely inflate test-retest reliability

Alternate-form reliability– Items differ only in wording such as change order of the et,

response or wording of item; correlation; need large enough sample size to increase the “power” ( the ability to detect difference between forms IF they exist)

RELIABILITY

Internal consistency reliability: – How groups of items “hang together”; Cronbach’s

coefficient alpha; Cronbach’s alpha reflects the homogeneity of a scale.

Inter-observer reliability (.a.k.a. inter-rater reliability):

– Correlation between two observer’s scores on same subject

Intra-observer reliability– Correlation between two observer’s scores on

same subject

VALIDITY

Face validity:Subjective; cursory view by untrained judges;

casual

Content validity: Subjective; not quantified; typically an organized view of what the survey covers in terms of subject matter

VALIDITY

Criterion validity: Compared to another instrument for prediction; Types include:

Concurrent validity: – needs to be relevant, well-known and is like

being compared to a “gold standard”

Predictive validity:– Ability to forecast attitudes, behaviors, outcomes,

events

VALIDITY

Construct validity: most valuable, yet most difficult to obtain; meaning of scale/instrument in practical use, a gestalt, rather than a calculated value; typically results from much effort and often years of work;

Types include: Convergent validity: usually from multiple

investigators from different approachesDivergent /Discriminant validity – much

effort, years of work

Survey Monkey.com

Professional Growth and Development Survey

http://surveymonkey.com/Users/32263842/Surveys/33828911949/EF5500DC-AD39-4186-A9FB-5425E2C3ED1F.asp?U=33828911949&DO_NOT_COPY_THIS_LINK

Arizona Master Gardener Survey

file:///C:/Documents%20and%20Settings/Owner/My%20Documents/Work%20Stuff/SurveyMonkey.com/485F788C-85FC-4E68-856A-A16CB2421AB6.asp.html

Creating Your Own Survey

http://surveymonkey.com/Users/32263842/Surveys/33182449893/F47D9320-AC19-4744-8D5B-9376E1AA9FEA.asp?U=33182449893&DO_NOT_COPY_THIS_LINK

http://surveymonkey.com/Users/32263842/Surveys/108271317032/260979FC-5246-47A3-8467-CC7A6854A795.asp?U=108271317032&DO_NOT_COPY_THIS_LINK

http://surveymonkey.com/Users/32263842/Surveys/83749780641/1F6E31E3-6493-4CA2-A355-0A880364380D.asp?U=83749780641&DO_NOT_COPY_THIS_LINK