Sampling and Instrument Creation 1 - Surveys

EDUC 894 Week 4

Plan for Today• Deliverable #1 In• For Next Week

▫Individualized Reading Handouts & Questions▫Draft Informed Consent Forms (&

Instruments…)• Sampling-----------------------------Break-----------------------• Instrument Creation Focus: Surveys

▫What NOT to do: An Example How many problems can you find?

▫What To Do: Principles Designing a Valid & Reliable Survey

• Group Work & Consultations

Sampling - The Objectivist Perspective

• There can be two goals▫To describe, document etc. the reality of something

for those who participated in the study▫To make generalizations about a larger group than

those who participated in the study

• Usually we are looking to generalize from a sample to the population▫To make claims about the population▫To build/test theories that explain the population

• Common Practices▫Use a large group to avoid undue influence of

unusual cases▫Want to select representative individuals

Selecting Representative Participants• A population is a group of individuals that

have the same characteristic(s).

• A sample is a subgroup of the target population that the researcher plans to study for the purpose of making generalizations about the target population.▫Samples are only estimates.▫The difference between the sample estimate

and the true population is the “sampling error.”

Populations and Samples

Population

Sample

Population- All teachers in high schools in one city

- College students in all community colleges

- Adult educators in all schools of education

Sample- All high school biology teachers

- Students in one community college

- Adult educators in five schools of education in the Midwest

Reality CheckThe Population

The group of individuals having one characteristic that distinguishes them from other groups.

The Target Population or Sampling FrameThe actual list of sampling units from which the sample is selected.

The SampleThe group of participants in a study selectedfrom the target population from which theresearcher generalizes to the target population.

College Freshmen

Freshmen in E101 at

SFU

Students who

returned surveys

Strategies for Getting a Representative Sample

• Simple random sampling: Selecting a sample from the population so all in the population have an equal chance of being selected

• Systematic sampling : Choosing every “nth” individual or site in the population until the desired sample size is achieved

• Stratified sampling: Stratifying the population on a characteristic (e.g., gender) then sampling from each stratum

How does the sample affect the claims you can make?• If I do a study about classroom management

strategies in which participants are 6th grade English teachers from a few schools in Vancouver, what populations make sense to generalize to?

English teachers Educators in British Columbia

Middle school teachers 6th grade teachers in the Lower Mainland

K-12 teachers in Canada

Language teachers in the Pacific Northwest

Sampling: Constructionist / Subjectivist Perspectives• The goal is to select people/sites who can best

help us understand a phenomenon / culture etc.▫“Purposeful Sampling” (as opposed to

“representative”)

• Looking to find participants that will help us develop a detailed understanding that might:▫Be “useful” information▫Help people “learn” about the phenomenon▫Give voice to “silenced” people

• Work with a small size to gain in-depth perspective

E.g. 1 individual, 4 cases, 10–20 interviews

Types of Purposeful SamplingWhen Does Sampling Occur?

Before Data Collection After Data Collection Has Started

What is the intent?

To developmanyperspectives

ExtremeCase

Sampling

To describeparticularlytroublesomeor enlighteningcases

TypicalSampling

To describe whatis “typical” to those unfamiliarwith the case

What is the intent?

To take advantageof whatever caseunfolds

SnowballSampling

To exploreconfirming ordisconfirmingcases

Confirming/Disconfirming

Sampling

MaximalVariationSampling

To generate a theory or concept

CriticalSampling

To describe somesubgroup in depth

HomogenousSampling

To describe acase that illustrates “dramatically” the situation

Theory or ConceptSampling

To locate people or sites to study

OpportunisticSampling

Practice Example

•You are conducting a study on teacher adoption and use of iPads

▫Describe an objectivist RQ you could ask and a strategy for representative sampling to help you answer it

▫Describe a constructionist / subjectivist RQ you could ask and strategy for purposeful sampling to help you answer it

Dinner Break

Survey Design

•(Most) surveys are conducted from an objectivist perspective

•Writing a survey is all about making decisions:1. Recognizing when a choice is being made2. Making choices that will create a reliable

instrument that will help you make valid inferences to answer your research questions

3. It’s (usually) not about making the “best” decision in some abstract sense, but making a choice that will be most useful for your purposes

Reliability & Validity• Reliability

▫The accuracy or precision of a measurement procedure X = T + E

▫Often thought of in terms of consistency Measured by correlation

coefficients

• Validity▫The degree to which scores

provide information that is relevant to the inferences to be made from them Content-related validity evidence Criterion-related validity evidence Construct related validity evidence

Limitations of Surveys

•Self-report data▫Notoriously unreliable▫Indicates only what we think, not reality

•Data at one point in time▫Limited w/in person sample

•Retrospective questions rely on memory•The Devil is in the Details

▫Question form, wording, and answer choices can inadvertently dictate responses

•Sampling ▫Who responds isn’t random

Image Source: http://www.nynpa.com/images/NYNPA0081.jpg

Survey Decisions

•Topics Covered•Number & Focus of Items per Topic•Item Sequence & Presentation•Item Type / Format•Item Wording

•Always think about what you will do with the answers you get – i.e. how will you analyze the data?

What NOT to do

•Take the survey posted here (link sent via email)

https://my.sfu.ca/cgi-bin/WebObjects/WebSurvey.woa/wa/survey?17437110

•What problems in each category can you find? ▫Topics Covered▫Number & Focus of Items per Topic▫Item Sequence & Presentation▫Item Type / Format▫Item Wording

Survey Decisions Topics Covered• Remember the idea of content-related validity?

▫Goal: Determine the most popular animals▫Assumed “like” indicates popularity

What pet you had as a child may or may not relate Are the plants in you garden a better indicator?

Feelings about gardening don’t give relevant info Caveat: Questions to lay a foundation

• Define your terms ▫Are you concerned with popularity of animals (in

general) or only pets? Can ask a similar question about plants / garden

plants▫What about ability self-esteem (in general) or

self-esteem as related to academic work? How would this affect your survey design?

Survey Decisions Topics Covered

Self Esteem

Social

Academic

Athletic

Survey

Survey Decisions Number & Focus of Items per Topic•Number: Do you have enough questions

on each topic to give a reliable measure?▫Three questions is usually the absolute

minimum given for a scale•Focus: Do questions give you info you can

actually triangulate?▫How will you compare the names of plants

reported to be in their garden with Likert scales given for specific plants?

▫What will you do with the question about liking “trees without flowers”?

Survey DecisionsItem Sequence & Presentation• Question sets and introductions• Broad to narrow progression

• How satisfied are you with service in the food court?• Do servers in the food court smile when taking your

order?• Will you group or distribute similar content items?

• Putting similar items together will help learners think about what you want them to

• But, if you have multiple items try and get convergence on a construct, putting them near each other will lead to a false reliability

• Technical Concerns• Pagination - scroll forever or endless clickage?• The “required item” dilemma

Survey DecisionsItem Type / Format• Open versus Closed Ended Questions

▫It all depends on what you are looking for, but from an objectivist perspective:

“Use close ended questions where at all possible, if you can limit the possible answers it can really speed up the data crunching when you

are finished”

▫You may need open-ended questions when you don’t know the range of possible answers But then you will have to go through and code

these answers for themes▫Thus, don’t ask an open-question with a limited

number of possible answers or where answers range predictably “Do you like to garden?” “How much do you like dogs?”

Survey DecisionsOpen Ended Question Guidelines• Be specific

▫What did you think of class? BAD▫What 3 things did you like best about class? GOOD▫ What 3 things did you like best about the class content? BETTER

• Leave people enough space to write▫A philosophy of life, love and happiness is not 64

characters or less!• Don’t ask too many open-ended questions

▫Your participants will tire and it will show in their responses How do you know if a sunscreen contains nanoparticles? Ask George W. Bush!

Survey DecisionsClosed Ended Question GuidelinesQuestion Types 1

▫Yes/No▫Lists

• Single choice (“best answer”, radio button)

• Multi (“all that apply”, checkbox)• Can do a series of these in a grid, but

dangers of confusion and auto-pilotListen Speak Read Write

English □ □ □ □

Spanish □ □ □ □

French □ □ □ □

Italian □ □ □ □

Survey DecisionsClosed Ended Question Guidelines

Question Types 2▫ Rating Scales

• Ordinal • E.g. Rank the following from best to worst

• Likert • E.g. Levels of agreement

• Likert-style • E.g. Levels of approval, satisfaction etc.

• Frequency • E.g. Often, sometimes, almost never

• Bipolar • E.g. Outgoing to introverted

Survey DecisionsClosed Ended Question Guidelines

•Focus on Rating Scales▫ Graphic vs. Finite Choice

▫ Think about the following:• Does your scale goes to the extremes?• Does your scale have enough choices?

• Participants are less likely to choose the anchors (n-2)

• Does your scale imply interval spacing?• Will your scale have a “neutral” option?

Mark an X on the line to show your level of agreement

Disagree Agree

Indicate your level of agreement by circling your choice

Strongly Disagree Disagree Neutral Agree Strongly Agree

Survey DecisionsItem Wording 1•Clear & Concise

▫What do you do when you get a headache? UNCLEAR

▫What headache medicine do you use? CLEARER

•Concrete▫Did you enjoy the book? ABSTRACT

▫Did you recommend the book to a friend? CONCRETE

•One idea per question▫Should BC continue its urban density and needle

exchange programs? CONFOUNDED

•Be careful when asking personal questions▫Use categories to avoid emotional responses

Survey DecisionsItem Wording 2•Avoid leading questions

▫Do you think Stephen Harper will continue to be a successful Prime Minister?

▫Do you think Stephen Harper will continue to be plagued by challenges about his failed policies?

•The challenge of reversed questions▫Can be used to break up question sequence or to

get at something in a different way, but big reliability issues I do not like venus fly-traps EXPLICIT

Self-Esteem Survey: Sometimes I doubt myself IMPLICIT

Survey DecisionsThe Big Picture

•Every time you have to make a decision, always think about 3 things1. How will your decision affect

participant responses?2. How will your decision affect your

analysis?3. What does your decision imply about

the underlying construct?

Group Work