Chapter 4 10/28/2013 12:151Geog 3250. Thoughts on researcher- administered surveys? Keep a list of things not to do in a researcher- administered survey
Thoughts on researcher- administered surveys? Keep a list of
things not to do in a researcher- administered survey from this
tongue-in-cheek video: 10/28/2013 12:15Geog 32502
Slide 3
Structured Interviews 10/28/2013 12:153Geog 3250
MethodInstrumentQuestion OrderContext Structured interview
interview schedulestrictusually face-to- face
Surveyquestionnairestrictvarious In-depth Interview Interview
guide, and the interviewer flexible to keep it conversational
usually face-to- face
Slide 4
Structured Interviews: errors and interviewer effects
10/28/2013 12:154Geog 3250 MethodAttitude towards interviewer
effects ErrorsOpportunities for Clarification Structured interview
minimize high: but clarify in exact same way in all situations
Surveyminimize low: due to social distance In-depth Interview use
to sustain rapport and keep the conversation meaningful No errors,
just opportunities for deeper understanding high: clarification may
be the source of meaningful conversation
Slide 5
Structured Interviews and errors Variation due to error can
come from: Intra-interviewer variability: an interviewer is not
consistent in asking questions or recording answers (with the same
respondent or a different one). Inter-interviewer variability: lack
of consistency in asking questions or recording answers between
different interviewers. 10/28/2013 12:155Geog 3250
Slide 6
Structured Interviews: responses 10/28/2013 12:156Geog 3250
MethodResponse modeData recording Structured interview mostly
closed-endedon the interview schedule Surveymostly closed-endedon
the questionnaire In-depth Interview open-ended and
conversationalField notes or audio recording Example of close-ended
question; data recorded on the document
Slide 7
Interview Contexts Work with your neighbour to fill out this
table - as many as you can think of per cell 10/28/2013 12:157Geog
3250 Method of Administration StrengthsLimitations
Researcher-Administered Telephone e.g. with CATI Face-to-Face
Self-Administered Drop-off, mail-back Hand-out, hand-back Email,
0nline
Slide 8
Conducting Structured Interviews: Basic Points Schedule
Interviewers should be familiar with the interview schedule.
Introduction Respondents should be provided with a credible
rationale for the research. Rapport Interviewers should try to
develop rapport with interviewees but refrain from being too chatty
(unlike semi-structured or unstructured) 10/28/2013 12:158Geog
3250
Slide 9
Conducting Interviews: Basic Points, contd. Asking Questions
The question should be asked exactly as stated. Small changes to
wording can make a big difference Similarly, the answers should be
recorded as exactly as possible. Questions should be asked in the
order they are given on the interview schedule. 10/28/2013
12:159Geog 3250
Slide 10
Conducting Interviews: Basic Points, contd. show cards - to
provide detailed information, e.g., Likert scale options
appropriate if long list of possible answers; same set of answer
categories applies to several questions. 10/28/2013 12:1510Geog
3250
Slide 11
Conducting structured interviews: Basic Points Leaving the
interview The interviewer must remain professional and have a set
response to disengage respondents so that they may do more
interviews (i.e., if researcher is motivated to have large sample
size) 10/28/2013 12:1511Geog 3250 Sometimes they want to be done
and move on, sometimes it is the other way around
Slide 12
Conducting (survey) interviews: Basic Points Training Good
interviewing is a skill set to be developed. It usually requires
considerable training and supervision to become good at itbut you
have to start somewhere! 10/28/2013 12:1512Geog 3250
Slide 13
Questionnaires (self-administered) Questionnaires are
essentially structured interviews without an interviewer. They
involve filling out a form which is then returned to the
researcher, often by mail or in person. Because the respondent has
to read the questionnaire without the aid of an interviewer, it has
to be very clear and easy to follow layout matters!! 10/28/2013
12:1513Geog 3250 Example of Statistics Canada self- administered
survey question
Slide 14
Questionnaires (self-administered) Compared to interviews,
questionnaires tend to have fewer open questions, because closed
questions are easier to answer; have simple designs, so the
respondent can complete all sections without difficulty; be short,
to avoid respondent becoming discouraged. 10/28/2013 12:1514Geog
3250
Slide 15
Questionnaires (self-administered) Advantages over structured
interviews: Cheaper, quicker, more convenient to administer No
interviewer effects, including errors in administering the
instrument Social desirability bias seems to be reduced.
Respondents are more likely to give genuine answers to sensitive
questions, e.g., those regarding sexual practices, drug use,
criminal activity, etc. 10/28/2013 12:1515Geog 3250
Slide 16
Questionnaires (self-administered) Disadvantages Researcher
cannot explain the question re: validity. Greater risk of missing
data, lack supervision. Researcher cannot probe. Difficult to ask a
lot of questions. Difficult to ask different kinds of questions,
especially open ones and questions requiring a filter. Order
effects hard to control may answer out of order Literacy issues
Designated respondent may not have completed questionnaire.
10/28/2013 12:1516Geog 3250
Slide 17
Questionnaires (self-administered) Online survey Growing in
popularity Finding participant listsrandom? Email surveys Tend to
be aimed at smaller, more homogenous groups 10/28/2013 12:1517Geog
3250 This one is free and can be restricted to UWO usernames
Slide 18
10/28/2013 12:1518Geog 3250
Slide 19
Questionnaires (self-administered) Researcher-driven diaries A
form of questionnaire Participants record their feelings,
perceptions, actions, etc. on a form shortly after they occur.
Participants should be given explicit instructions on how to
complete the diary, the time periods for recording responses, and
the types of experiences to be recorded. Used for quantitative or
qualitative research. structured like a regular questionnaire with
closed questions, or free text experiences written down without
fixed responses, like answers to open questions 10/28/2013
12:1519Geog 3250
Slide 20
Diaries Advantages of diaries over interviews and
questionnaires: Fairly accurate data, re: frequency or time spent
on behaviours of interest Fairly accurate data about sequencing of
behaviours Good for getting data on sensitive matters, e.g., sexual
practices Disadvantages More expensive than personal interviews
Attrition: people may tire of using them Details may not be
recorded quickly enough. Errors, omissions 10/28/2013 12:1520Geog
3250
Slide 21
Respondent Problems - Structured Interviews and Questionnaires
Response sets the respondent is not motivated to provide a genuine
response answer all the same without reading 10/28/2013 12:1521Geog
3250 I am not paying attention to each question as I respond
Slide 22
Respondent Problems - Structured Interviews and Questionnaires
Acquiescence: trying to please the researcher e.g., respondent
agrees just to be cooperative partial antidote: mix items so that
they have logically opposite positions 10/28/2013 12:1522Geog 3250
Do you like this image?
Slide 23
Respondent Problems - Structured Interviews and Questionnaires
Social desirability: avoiding appearance of being bad/unlikeable
Answers are not sincere, e.g. questions on racism or the
environment 10/28/2013 12:1523Geog 3250 I hope these are the
answers she is looking for.
Slide 24
Respondent Problems - Structured Interviews and Questionnaires
Laziness or boredom: answers just to get the end of the process
E.g., respondent checks all the same answer to get it done
10/28/2013 12:1524Geog 3250 Assume all respondents might be this
lazy?
Slide 25
Feminist (and other) Critiques Structured interviews and
questionnaires - exploitative. asymmetrical power relationship
between the researcher and the respondent. Link to Ethics:
Increased attention given to the rights of research respondents in
recent years, e.g., privacy rights, the right to end the interview
at any time, etc. Is this enough? 10/28/2013 12:1525Geog 3250 Power
imbalances are not always this overt
Slide 26
Questionnaire Design Identify at least one problem with each
question in the following questionnaire snippet: 10/28/2013
12:15Geog 325026
Slide 27
Open or Closed Questions? Closed Questions Present the
respondent with a set of answers from which to choose E.g., see
previous slide Open Questions Response decisions are left
completely to the respondent Answers must be coded into to
numerical data Tend to be used in quantitative research E.g., What
do you think about using alternatives to chemical pesticides on
your lawn? 10/28/2013 12:1527Geog 3250
Slide 28
Open Questions Advantages Respondents can answer in their own
terms Allow for unusual, unanticipated responses No suggested
answers, so responses may expose knowledge and be more genuine Good
for exploring new or changing areas of research Answers may lead to
fixed-choice responses 10/28/2013 12:1528Geog 3250
Slide 29
Open Questions Disadvantages Time-consuming to record answers
Answers have to be coded In self-administered questionnaires,
respondents may balk at the request to write long answers Recording
inaccuracies for verbal answers Why did your degree take longer
than the funding period? Not really sure. I worked away on it
diligently but the analysis just took longer than expected, I
guess. There are worse places to be than grad school. Closer or
more effective supervision was the primary reason. Too much
procrastination and family issues. What code/categories? Can one
response go in multiple categories? 10/28/2013 12:1529Geog
3250
Slide 30
Coding Open Questions Post-coding Data are gathered, then
themes or categories of behaviour are discerned. e.g., an open
question on the legalization of marijuana might have codes very
hostile opposition, indifferent, etc. Each code would then be
assigned a number, and attributed to respondent 10/28/2013
12:1530Geog 3250
Slide 31
Coding Open Questions, contd Pre-coding Themes or categories of
behaviour are decided upon before the data are gathered. May be
done with fixed-response items (i.e. closed- ended) e.g., What is
your view on the legalization of marijuana? Are you strongly
opposed, opposed, neither opposed nor in favour, in favour, or
strongly in favour? Data may be gathered freely through open
responses and sorted into pre-set categories afterward. (What if
there are responses that seem to require a new code?) 10/28/2013
12:1531Geog 3250
Slide 32
Coding Open Questions, contd Basic principles: The categories
must not overlap. The list of categories must be exhaustive. Must
cover all possibilities. Other category is usually required. Coders
should be provided with a coding frame or coding manual that
establishes clear rules about how codes should be applied.
10/28/2013 12:1532Geog 3250
Slide 33
Closed Questions Advantages Easy to process answers
Standardization allows comparison of answers Fixed responses may
help clarify what the question means Easier and quicker for the
respondent to complete Reduced bias in recording answers No
interpretation required by researcher 10/28/2013 12:1533Geog
3250
Slide 34
Closed Questions Disadvantages Loss of spontaneity and
authenticity - relevant answers may be excluded from choices
provided. Antidote: Other category with open area to elaborate (but
you may want to code it afterwards!) Defining categories that do
not overlap. Antidote: Pre-test as open-ended to establish
appropriate and distinct categories. Difficult to make
forced-choice answers exhaustive 24. Why do you oppose the
construction of more wind turbines in your community? Bird/bat
collisions Economic costs outweigh benefits Health threats Impacts
on farm and domestic animals Unfair siting process Other
_____________________ 10/28/2013 12:1534Geog 3250
Slide 35
Closed Questions, contd Disadvantages (contd) Respondents may
differ in their interpretation of the wording of fixed responses.
e.g., the meaning of strongly in strongly agree Respondents may not
find a fixed response that they feel applies to them. Large numbers
of closed questions can reduce rapport in interviews Antidote:
include open-ended responses even if you may not code them?
10/28/2013 12:1535Geog 3250
Slide 36
Types of Questions Personal factual questions e.g., age,
occupation, number of cars owned Factual questions about others,
events etc? Avoid these: makes respondent feel they are being
tested and look stupid Antidote: Questions about perceptions are
better puts them in position of being the expert. This survey
tests, but it is for education, not research purposes 10/28/2013
12:1536Geog 3250
Slide 37
Types of Questions, contd. Beliefs e.g., Do you believe climate
change has happened globally over the last 50 years? Attitudes Very
common, subjective, taste, or perception use of Likert scales E.g.,
Do you feel we should do more to mitigate the effects of global
climate change. Gabe Kotter tests attitudes where participant is
the expert regardless of how they choose 10/28/2013 12:1537Geog
3250 Likert agree scale, often used to create an index from
multiple items
Slide 38
General Rules for Designing Questions Keep the research
questions/hypotheses in mind. Focus on exactly what you want to
know. Be specific. e.g., How many children 18 and under are living
at home? vs. How many children do you have? Put yourself in the
position of the respondent How would you answer the question?
10/28/2013 12:1538Geog 3250
Slide 39
Specific Rules for Designing Questions AVOID: ambiguous terms
Even commonly used terms may be vague, e.g., does income include
tips? Investments income? Does it mean after tax income? long
questions: respondents may lose track of what the question is
asking multiple phenomena in the same question E.g., Do you agree
your network of closest friends and family is helpful and
supportive? 10/28/2013 12:1539Geog 3250
Slide 40
Specific Rules for Designing Questions AVOID: several very
open-ended questions (do unstructured interviews instead?) e.g.,
What do you think of the prime minister? leading questions e.g., Do
you feel the monarchy, which costs taxpayers over $5 million each
year, be abolished? 10/28/2013 12:1540Geog 3250
Slide 41
Specific Rules for Designing Questions AVOID: questions that
include negatives, especially double negatives. e.g., Do you oppose
the rule stating that laptops are not allowed in classrooms?
jargon/technical terms. And be sure that respondents have the
knowledge needed to answer the question. E.g., How would you rate
the level of social capital in your neighbourhood? 10/28/2013
12:1541Geog 3250
Slide 42
Specific Rules for Designing Questions Ensure symmetry between
a closed question and its answers. e.g., avoid: Responses should be
very satisfied, somewhat satisfied, Ensure the answers are
balanced. e.g., avoid: 10/28/2013 12:1542Geog 3250 Dont let
instructors design their own course evaluations?
Slide 43
Specific Rules for Designing Questions Memory Beware of
expecting respondents to recall what theyve done or observed in
detail. e.g., number of hours spent online per week Dont know
option Consider avoiding this(e.g., use neither agree/disagree
instead if you do not want to force them to choose) Thought to
prevent forcing the respondent to choose, but it actually serves to
offer an easy way out. 10/28/2013 12:1543Geog 3250
Slide 44
Exercise Design a survey question that violates one rule of
questionnaire design identified above 10/8/2013 10:15Baxter Geog
215344 Text 335633 and your question to 37607 Or Respond at this
linkthis link poll results here YES peek back up the notes NB You
need to be in slideshow mode (full screen) for any of the links to
work
Slide 45
Specific Rules for Designing Questions Question order All
respondents should receive questions in the same order (unless
testing for order effects). Asking a particular question may affect
the responses given to subsequent questions. Early questions should
be directly related to the announced research topic. Questions
likely to be of interest to respondents should be asked early in
the interview. Sensitive questions (e.g., Have you ever been a
victim of sexual assault?) should be asked well into the schedule,
but not at the very end. 10/28/2013 12:1545Geog 3250
Slide 46
Specific Rules for Designing Questions Question order, contd
Questions should be grouped logically and into related sections.
Within a group of questions, general questions should precede
specific ones. Opinion and attitude questions should precede
behaviour questions. Even if a respondent provides an answer to a
question before it is asked, the question should be repeated at the
appropriate time. i.e., there could be question order issues.
10/28/2013 12:1546Geog 3250
Slide 47
Designing the Questionnaire Provide a clear layout and
presentation Choose vertical or horizontal answers, as appropriate
Vertical tends to be more pleasing to the eye 10/28/2013
12:1547Geog 3250 Vertical layout Horizontal layout
Slide 48
Designing the Questionnaire Provide clear instructions on how
to respond. Is more than one response allowed? Should the choice of
answer be circled? Underlined? Keep a question and its answer
together on the same page. 10/28/2013 12:1548Geog 3250
Slide 49
Vignette Questions Presenting people with one or more scenarios
and asking them how they would respond Anchor the choices in a
realistic situation. Creates distance between question and
respondent. Hopefully more candid response. Weakness: how people
say they would act in a particular situation may be very different
from how they would really act. Can provide useful information, or
at least a starting point for further research. 10/28/2013
12:1549Geog 3250 Michelle and Philip bought their home 5 years ago
and 2 years ago wind turbines were installed nearby.Please provide
your thoughts about their situation in the following 5
questions.
Slide 50
Pilot Studies Used to test whether individual items or the
instrument as a whole operate well. Used with open questions to
generate closed questions for subsequent studies. Provide
interviewers with experience in administering the instrument. Can
be used to ensure that there is variability in the answers given.
10/28/2013 12:1550Geog 3250
Slide 51
Pilot Studies Can identify questions that are embarrassing,
uninteresting, etc. Can identify questions that are difficult to
understand. Can be used to determine the adequacy of the
instructions. Can be used to determine whether the instrument has
satisfactory flow. 10/28/2013 12:1551Geog 3250
Slide 52
Using Existing Questions If certain questions have worked well
for other researchers, they may be appropriate for your study. The
original source must be cited. Allows results from different
studies to be compared. Questions may be modified to suit ones
purposes better. 10/28/2013 12:1552Geog 3250