CBR 303: CBR 303: A Hands-On Approach to Developing Survey A Hands-On Approach to Developing Survey

Tools Tools in Community-Based Researchin Community-Based Research

Welcome & Introductions

Name & affiliation

Experience with CBR and/or survey tools

What you hope to learn today

Objectives

Upon completion of this workshop you will be able to:

Differentiate between what you need to know and what is nice to know

Use various strategies for asking survey questions

Apply different types of response options

Recognize various approaches for administering surveys

Describe how to increase response-rates

Develop a strategy to manage data

Identify various strategies for presenting survey results

Agenda

Review of CBRDefinition and purposes of a surveyIdentify features of ‘best’ surveysSteps in the survey processEstablish research objectives; identify what you

need to know vs. what is nice to knowBegin to develop research questionsRefine research questions

What is CBR?

When you hear “community-based research,” what words or phrases come to mind?

How would you define it?

CBR definition

CBR is a collaborative approach to research that equitably involves all partners in the research process and recognizes the unique strengths that each brings. CBR begins with a research topic of importance to the community with the aim of combining knowledge and action for social change to improve community health and eliminate health disparities.

Minkler & Wallerstein (2003)Community-Based Participatory Research for HealthJossey-Bass (2004)

Principles of CBR

ethicalreview

process oriented

community relevance

social actionoutcomes

joint dataownership

varied methods

capacity building

community initiation

CBR

Introduction to surveys

Why do we do surveys?

What questions can they answer?

What questions can’t they answer?

When is doing a survey a bad idea?

What is a survey?

A process for gathering information, without detailed verification, on the activity being examined.

The systematic collection, analysis and interpretation of information about some area of study.

Information is collected primarily by means of sampling and interviews with the selected individuals.

Survey Goals

To produce detailed data that will allow researchers to:|

describe the characteristics of the group under study,

test theories about the group, and

generalize results beyond just those people who responded to the survey

Purpose for Surveys

Understand the specific activity under review

Identify significant areas warranting special emphasis

Obtain information for use in making organizational changes

Identify future changes/goals for the organization

Features of the “BEST” Surveys *

Specific objectives

Straightforward questions

Sound research design

Sound choice of population or sample

Reliable & valid instruments

Appropriate analysis

Accurate reporting of survey results

Reasonable resources* From The Survey Handbook, Arlene Fink, 1995

Survey process

Assemble your team

Establish your research objectives and questions

Determine your sample

Choose interview methodology

Design questionnaire

Pre-test the questionnaire

Administer survey

Collect and enter data

Analyze data

Present data

1. Assemble your team

Statistician +/ Social Scientist +/Epidemiologist

Community members +/ Community leaders +/ Service providers

Remember to consult your team early and often during planning, implementation analysis and dissemination.

2. Establish Research Objectives & Questions

What do you want to know?

What do you need to know?

What will you measure? Why this factor?

How will the information be used?

Is a survey the best way to get the information you need?

What kinds of resources (time, staff, money) do you have in place to make this happen?

Try not to mix too many objectives into a single survey. Keep it focused.

Example – Research Objective

To identify the factors that lead to early school withdrawal for lower income youth in the community

Don’t Reinvent the Wheel

Before you develop your own tools see what other people have done.

Is it worth redoing a study that has been done 100 times?

What makes your research new? How do others ask similar types of questions?

(check the literature) Can you use or adapt a pre-existing instrument to

meet your needs?

Standardized vs. Homemade Instruments

Standardized Homemade

can compare your results hard to compare

less costly more costly

may not be totally appropriate

can tailor to your needs, may be more relevant

higher credibility credibility depends of investigators

may not exist! always creatable!

Next … Hypothesis

X Y

Lower Socio-economic Status Early School Withdrawal

‘Operationalizing’ Variables

Lower Socio-economic Status

Actual Income Occupation Work

Shift/Conditions Time Spent with

children Ability to provide

educational supports

Early School Withdrawal

Withdrawal from school prior to completion of Grade 12

Marketable skills Relationship with school

staff Relationship with peers Social attitudes

From Concept to Questions Example: Socio-economic Status

Construct Socio-economic status

Indicator Income level

Operationalized Variables

What was your total family income during the past 12 months?

Measurement a) Under $10,000 b) $10,001-$25,000 c) $25,0001 - $50,000d) $50,001-100,000 e) more than $100,001

From Concept to Questions Example: Socio-economic Status

Construct Socio-economic status

Indicators Educational attainment

Operationalized Variables

What is the highest level of school you have completed?

Measurement a) Primary Schoolb) Some High Schoolc) Completed High Schoold) Some College e) Completed Collegef) Graduate/Professional Degree

Exercise: From concept to questions

On your Survey Development Template:

1. Identify the hypothesis (e.g. X causes Y) underlying your objective.

2. List the related factors for each variable.

3. Develop a related research question, operationalizing your variables.

4. Develop a specific measurement for the question.

Important Tips!

Each question on your survey needs to be JUSTIFIED and LINKED to your research objectives

Obtain ALL of the ESSENTIAL information

Obtain ONLY the ESSENTIAL information

Example: Possible question topics

AgeSexSexualityExperiences in schoolLevel of academic completionExperiences of successSESSchool

NeighborhoodWhat their parents do

Extracurricular activities

Hobbies

Relationships with teachers

Relationship with peers

Relationship with parents

Part-time employment

Triaging Questions

High: Absolutely essential

Medium: Very valuable for decision making

Low: Supportive data to enhance understanding

Strategies for asking questionsDifferent Types of Questions

Open-Ended

The respondent answers in their own words

More demanding and time consuming for respondents

Lower response rate

Harder to categorize, analyze and interpret

Good when you don’t know how people will respond, useful in the preliminary pilot stage

E.g. Please describe any barriers that you face accessing our service?

Closed/Structured

The respondent answers from a range of choices

Less time consuming for respondents

Higher response rate

Easier to categorize, analyze and interpret

Requires more front-end effort (designing the questions)

Can oversimplify or make assumptions

Response categories must be inclusive

Limits answers to options presented

Can compromise with ‘other:____________’ category

Types of questions: Two choices

Do you like ice cream?

Yes No

Types of questions: Multiple choice

Which is your favourite ice cream flavour?

Chocolate Vanilla Strawberry Banana Other: __________ (please specify)

Types of questions: Checklist

Please select all of the ice cream flavours that you have eaten in the last month?

Chocolate Vanilla Strawberry Banana Other: __________ (please specify)

Types of questions: Ranking

Please order the following ice cream flavours by preference (where 1=your favourite & 4 = your least favourite)?

Chocolate Vanilla Strawberry Banana

Types of questions: Rating/Likert Scale

Please rate the degree to which you agree/disagree with the following statements:

1. Ice cream is my favourite dessert.

Strongly agree Agree Disagree Strongly Disagree

2. Ice cream's only redeeming feature is that it can have chocolate.

Strongly agree Agree Disagree Strongly Disagree

Helpful options

Excellent – good – fair – poor

Approve/Disapprove, Favour/Oppose,

Too many – about right – not enough

Often – Regularly – Seldom

Always – most of the time- some of the time- rarely – never

More likely – no difference – less likely

(Agree) 1 2 3 4 5 (Disagree)

Evens or Odds

Even numbered scales force a choice

Odd numbered scales allow for a neutral stance

Beware of the forced choice: research shows that those who would have gone for neutral are more likely to go for positive choices when forced!

Beware of neutral categories: research shows that people ‘revert to the mean’ (ie they are most likely to choose neutral).

Types of questions: Numerical

How many ice cream cones have you eaten in the last week? _____

How old are you? ______

What year were you born?_____

Tips for Question Design

Don’t ask questions that you would not answer

Focus on behaviours not labels

Keep your values/assumptions out of questions

Make sure that questions are not too demanding

Avoid abbreviation or professional jargon

Use simple, direct and familiar words/terms (slang can be OK)

Tips for question design cont’d …

Make questions clear, specific and as short as possible

Make sure everyone will interpret the question the same way

Specify the frame of reference (e.g. time, place)

Avoid leading questions

Be aware of ‘social desirability’

Make sure that questions are applicable to all respondents

Ordering your questions

Always give an intro (title, sponsor, purpose etc)

Start with easy questions that everyone can answer (builds trust & confidence)

Introduce sensitive stuff gradually

Ordering your questions

There should be a logic to sequence and flow, use transition statements or subheadings

Move from general to specific

Ask about things in chronological order

Always say thank you at the end and leave room for questions/comments/feedback

Common Survey Errors!

Beware!

Problem questions: The double barrel

Beware of the “ANDs”

Do you like to watch TV andand eat potato chips?

How would you rewrite?

Problem Questions: The Incompetent Experts

Sometimes people don’t know enough to answer the questions or may not be able to estimate the answers

To what extent do you agree with the economic foreign policy of Latvia?

How many cans of soup have you bought in the last year?

Problem questions: The long list

Sometimes having questions with too many response categories can be overwhelming, especially when you are asking folks to rank order them:

Please rank order these TV shows (where 11 is your favourite and 1717 is your least favourite)…

Problem questions: The double negative

I disagree that the statement is false…

Problem questions: Not mutually exclusive

How old are you?

20-25 25-30 30-35 35-45

Problem questions: Vague wording

Do you have a partner?

Yes No

Problem questions: The mismatch

Do you work out?

Strongly agree Agree Disagree Strongly disagree

The Vague Response

When is your birthday? __________

Answer: tomorrow

Sex, drugs and all the good stuff

How do we Ask about Sensitive Topics?

Asking sensitive questions

Don’t ask what you would not answer!

Consider social desirability bias

Some strategies for asking sensitive questions include:

Self-administered questionnaires Ensuring anonymity or confidentiality Careful wording Providing appropriate response ranges

Ways of Asking Threatening Questions

Various techniques have been developed to ask embarrassing or sensitive questions in non-threatening ways.

The following are examples of several such techniques as applied to the question:

"Did you kill your boss?"

Ways to ask: “Did you kill your boss?”

The Casual Approach "Did you happen to have killed your boss?“

The Numbered Card “Please read the number which corresponds with

what happened to your boss.“

1. Natural death2. I killed him/her3. Other (please specify)

“Did you kill your boss?”

The “Everybody’s Doing It” Approach "As you may be aware, many people have been killing their

bosses these days. Would you happen to have killed yours?"

The “Sandwich” Method "Have you ever done any of the following?"

a. Fire-bombed the Parliament buildingsb. Murdered your bossc. Detonated a nuclear device

“Did you kill your boss?”

The Assumption Method "How many of your bosses have you killed in the last

ten years of work?“

The “Other People” Approach

a. "Do you know of other people who have killed their bosses?"

b. “Have you done that yourself?“

“Did you kill your boss?”

The Illustrative Technique

“Which of the following pictures most accurately depicts what happened to your boss?"

Designing a good survey is hard work

Give Yourself Plenty of Time!

Pilot Testing

TEST! TEST! TEST! TEST! TEST! Test with colleagues & participants – the more

the better. This will save you time in the long run!

Discover Poor question wording or ordering or

instructions Suggest additional response categories Time issues

Getting your questions answered….

Who are you going to talk to?

3. Determine Your Sample

Who will you interview?

Who is your clientele?

Will this group provide you with the necessary data to meet the goals of the project?

Keep the sample large enough for a sufficient amount of data but small enough so that it is manageable.

Sampling

Eligibility or inclusion/exclusion criteria

Who do you wantwant to talk to?

Who do you not wantnot want to hear from?

Inclusion & Exclusion

Research Question

How does socio-economic factors impact on students’ ability to remain in school?

Target Population

High School Students

Inclusion Criteria

13-18 years oldStudents in school & Early school leaversParental and youth consent

Exclusion Criteria

Use alcohol, marijuana or other drugs more than once a week

Sampling

Probability Sampling (ideal)Probability Sampling (ideal) Simple random sampling Stratified random sampling Systematic random sampling Cluster random sampling

Non-random sampling (hard-to-reach, pilot)Non-random sampling (hard-to-reach, pilot) Whole group Convenience Snowball Quota

Sample Size

Best way to determine this is to talk to a statistician!Considerations include:

CostEffect Size

Response rate

Getting your questions answered….

How are you going to do it?

4. Choose an Interview Methodology

How will you conduct the interview?

mail out Questionnaires individual interviews case study focus group on-line surveys

Consider the make up of your clientele: language; accessibility language; accessibility options; age; educationoptions; age; education

Types: Face to Face (door to door)

BENEFITS

Interviewer administers survey

Ensures consistency & complete response

Allows for clarification & probing

Good if there are lots of ‘skips’

High response rateTakes care of literacy issues

CONSIDERATIONS

ExpensiveConfidentiality & privacy

concernsInterviewer biasSocial desirability bias may

be stronger

Types: Telephone

BENEFITS

Interviewer administers survey

Ensures consistency & complete response

Allows for clarification & probing

Good if there are lots of ‘skips’

Less expensive than face to face

CONSIDERATIONS

Less response rate than face to face

Miss those without a telephoneHard in this climate of

‘telemarketers – must be SHORT!

Types: Self-completed (mail, web)

BENEFITS

Survey completed by respondent

Low-costCan be good if there is lots

of ‘confidential’ or ‘embarrassing’ questions

Increase anonymity of responses

CONSIDERATIONS

Requires literacyVariable completeness of answers

Bad if there are lots of ‘skips’ or confusing parts

Lower response rate

Factors to Consider when Choosing a Method

Cost, budgetHuman resourcesEquipmentExpertiseWho is likely to be reached/not reachedWhat is your target population most likely to respond to?BiasResponse rateParticipant burdenFollow-up necessary?TimelinesSample size

Getting your questions answered….

But nobody wants to talk to me?

Improving Response Rate

Keep things short & simple

Ensure confidentiality

Target the appropriate population

Make it easy/convenient

Clearly communicate purpose, importance & relevance

Give advance notice

If you can – provide rewards or incentives

Increasing response rate: Layout matters!

Create a bookletPrint on white paper, don’t look like an adNo questions on 1st page – just title No questions on last page – room for commentsOrder fun/easy questions 1st, harder in the middle

and demographics at the endThe first question matters: it should be easily

answerable!The layout should be neat and clean and easy to

followFrom Dillman handout

Getting your questions answered….

What do I with all these surveys?

8. Manage Your Data

Expect to collect data 2-3 weeks after distributing

Stay organized

Remember confidentiality

Determine ahead of time (before or while survey is being developed) data management program (e.g. NVIVO or NuDist for qualitative; SPSS, SAS, Stata for quantative)

Determine ahead of time who will enter data; provide them with adequate training

Talk to your statistician!

9. Analyze Data

Identify themes & patterns

Frequencies (counts of things, how many times things happen; get mean, median, mode)

Descriptives (percentages or raw numbers of demographics and other factors that would be nonsensical to take an average such as gender or ethnicity)

Outliers (things that fall outside the expected values)

My questions are answered….

Presenting Results

Numbers can be deceiving…

10. Report Results

Make sure to acknowledge both the strengths & limitations of your data beginning with strengths!

ALWAYS CONTEXTUALIZE YOUR DATAALWAYS CONTEXTUALIZE YOUR DATA

How many people did you talk to? How did you decide who to talk to?

(random, convenience) Who did you talk to? How easy is it to generalize your results?

Presentation Matters

TablesGraphsPicturesWords

Features of the ‘bestbest’ Surveys

Specific objectives

Straightforward questions

Sound research design

Sound choice of population or sample

Reliable & valid instruments

Appropriate analysis

Accurate reporting of survey results

Reasonable resourcesFrom The Survey Handbook, Arlene Fink, 1995

Reflection & Next Steps

Reflect on and record 1-3 things you learned today.

Record 1-3 related goals you have going forward.

Record 1-3 related next steps you plan to take upon leaving the workshop.

Objectives

Having completed this workshop you should now be able to:

Differentiate between what you need to know and what is nice to know

Use various strategies for asking survey questions

Apply different types of response options

Recognize various approaches for administering surveys

Describe how to increase response-rates

Develop a strategy to manage data

Identify various strategies for presenting survey results

Acknowledgements

Several of the slides presented here today were developed by San Patten, MSc., Community-Based Research Coordinator for the Alberta Community Council on HIV

CBR 303: CBR 303: A Hands-On Approach to Developing Survey A Hands-On Approach to Developing Survey

Tools Tools in Community-Based Researchin Community-Based Research