SoTL: Research Design

Nancy Gourash Bliwise, PhDSTEP Faculty Development Workshop

Emory UniversityJune, 2014

Research Design in SoTL

Commitment to Outcomes Research

Learning Objectives

Identify the basic elements of four common quasi-experimental designs used in outcomes research

Discuss strategies to “control” factors that might impact validity of conclusions drawn from designs

Be able to identify ethical issues in educational outcomes research and resources for IRB review

Outcome Research is …

Important

Rewarding

Necessary

And often…

Difficult

Time-consuming

Small effect sizes

Type of ResearchDescriptive/Qualitative

Question: What Is? descriptive, generative naturalistic,

observational, constructivist

Samples: small, targeted Assessment: interviews,

observations inductive formative, “thick

description”, expansive

Inferential/Quantitative

Question: What Works? hypothesis testing,

confirmatory empirical, statistical,

comparative Samples: large,

representative Assessment: scores, rates

deductive summative, precise,

reliable

Formal Design Focus of SoTL

Actions as educators Student learning and performance

Question guides choice of design Source Material:

Cook & Campbell (1979) – Quasi-Experimentation: Design & Analysis Issues for Field Settings

Trochim (2006) – Research Methods Knowledge Base Online: http://www.socialresearchmethods.net/kb/

NSF – User-friendly Handbook for Mixed Methods Designs Online: http://

www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf97153

Optimal Design -- Descriptive

No single preferred design

Common features Focus on individual and collective experiences of students Context is described Search for patterns Developmental Multiple assessment methods and assessment

opportunities Subgroup analysis

Optimal Design -- Inferential

Randomized “Clinical” Trial

Real Life

Quasi-experimental designs Non-random assignment to groups

Frequently used (and effective designs Single group repeated measures design

Non-equivalent groups pre-test/post-test design

Cohort design

Single group crossover (ABAB) design

Uninterpretable Designs

One-shot case study

Single-group pre-test/post-test design

Static group comparison

X O

O1 X O2

X O O

Single Group Repeated Measures/Time Series

Add multiple observations to track learning/change

Goal is to study the trajectory of learning Depth of knowledge Understanding of concepts Application Integration/synthesis Testing of ideas

O1 O2 X O3 O4 O5 O6

Non-equivalent Groups Pre-test/Post-test Design

Compare two classes New vs. standard

Willing/interested colleagues Commitment to method

Pre-test/Post-test essential

Targeted measurement

O1 X O2O1 O2

Cohort Design

What if you are the only one who teaches this? Or colleagues are not “willing”

Compare outcomes to yourself But be very, very careful

Risk – nothing else can change Assumes stable cohort

O1 X

O2

Single Group Cross-over Design

Multiple units/cases

Replicate (ABAB)

Standard assessment strategy

XA1O1O2O3 XB1O4O5O6 XA2O7 O8O9XB2O10O11O12

Less “Control”

Rule out competing explanations of findings Argument/Logic Measurement

Student demographics Possible pre-test differences Characteristics of “treatment”

Design Statistical controls Preventive action

Sound Measurement

Existing standards

Independence of observations, where possible

Conceptual comparison

Multiple dimensions/skills

Your Turn…

Choose design that best “fits” your case

Type of class

Possibility of comparison

Assessment strategy

IRB Review

Educational research Human participants

Type of research Non-research/exempt/expedited

Primary Issues Coercion Informed consent

Everything IRB

www.irb.emory.edu

Social-behavioral studies

Multi-disciplinary team Rebecca Rousselle Carol Corkran

Non-Research

New category Systematic investigation, including research

development, testing and evaluation, designed to develop or contribute to generalizable knowledge

PI can make this determination If it is just for your own evaluation, it is not

research

Exempt vs. Expedited

Most surveys, interviews, or observations of public behavior are presumed exempt if the participant cannot be identified (no links from the person to the data) or the responses/data can not harm the participant.

IRB determines this

Coercion

Major issue Students must be free not to participate

Informed Consent

Description of the purpose of the study Who is being studied What will be required Risks/benefits Compensation Legal review Contact information Summary of findings