Work Session 2Methodological Approaches

Office of Research, Evaluation and Policy StudiesXimena D. Burgin, Ed.D.

December 2nd, 2010

Intent of a Study

Purpose statement: Overall directioni.e.,The purpose of this grant is to enhance and expand existing

partnerships. The Partnership and Community Readiness section of the application should demonstrate that a school-community partnership has been in existence for 6 or more months prior to submission of the application (henceforth referred to as preexisting school-community partnership). The preexisting school community partnership should involve at least two of the four required SS/HS partners that have worked together on issues of school safety, drug and violence prevention, and/or healthy childhood development issues for 6 or more months prior to submission of the application.

Work Plan Requirements…

The SS/HS Initiative draws on the best practices of education, justice, social services, and mental health systems to provide integrated and comprehensive resources for prevention programs and prosocial services for youth. This comprehensive plan include six elements to address problems of school violence and alcohol and other drug abuse:♦ Element 1: Safe school environment.♦ Element 2: Alcohol and other drugs and violence prevention

and early intervention programs.♦ Element 3: School and community mental health preventive

and treatment interventionservices.♦ Element 4: Early childhood psychosocial and emotional

development programs.♦ Element 5: Supporting and connecting schools and

communities.♦ Element 6: Safe school policies.

Grant Measurements…

GPRA1 – SS/HS grant sites will experience a decrease in the number of violent incidents at schools during the 3-year grant period.

GPRA2 – SS/HS grant sites will experience a decrease in substance use during the 3-year grant period.

GPRA4 – SS/HS grant sites will increase mental health services to students and families during the 3-year grant period.

Grant Outcomes…GPRA1 – SS/HS grant sites will experience a decrease in the number of violent incidents

at schools during the 3-year grant period.

Outcomes: Objective 1. In each of three years cumulatively, decrease by 60% the percentage of violent incidence occurrence for each of the categories: threatening behavior, unprovoked attack on student, fighting, fighting with or threatening teacher, possession or use of dangerous weapon, arson, and endangering safety of others.

GPRA2 – SS/HS grant sites will experience a decrease in substance use during the 3-year grant period.

Outcomes: Objective 2. In each of three years cumulatively, decrease by 60% the percentage of substance abuse occurrence for each of the categories: alcohol, tobacco, and controlled substances (possession or use).

 GPRA4 – SS/HS grant sites will increase mental health services to students and families

during the 3-year grant period.

Outcome: Objective 4. In each of three years cumulatively, increase by 50% the percentage of students and families accessing and receiving mental health services, as measured by frequency of service requested or inquired about, follow-through in accessing such service, and persistence in pursuing such service to closure.

Is the study feasible?

Tradeoffs between rigor and practicality. You may have to: Control the implementation of your program more

carefully. Ask program participants lots of questions. Evaluate research rigor, desire and practicality.

Is the study feasible?…cont.

Practical considerations for a feasible research project: How long the research will take to accomplish. Ethical constraints Needed cooperation Costs.

GPRAs &Outcomes

(Research Questions)

Quantitative Research Questions

Quantitative research is mainly concerned with the degree in which phenomena possess certain properties, states and characters, and the similarities, differences and causal relations that exist within and between.

Qualitative Research Questions

Qualitative research is concerned with the properties, the state and the character (i.e., the nature, of phenomena). Emphasis on processes and meanings.

Grant Outcomes…

GPRA1 – SS/HS grant sites will experience a decrease in the number of violent incidents at schools during the 3-year grant period.

Outcomes: Objective 1. In each of three years cumulatively, decrease by 60% the percentage of violent incidence occurrence for each of the categories: threatening behavior, unprovoked attack on student, fighting, fighting with or threatening teacher, possession or use of dangerous weapon, arson, and endangering safety of others.

Grant Outcomes…

GPRA2 – SS/HS grant sites will experience a decrease in substance use during the 3-year grant period.

Outcomes: Objective 2. In each of three years cumulatively, decrease by 60% the percentage of substance abuse occurrence for each of the categories: alcohol, tobacco, and controlled substances (possession or use).

Grant Outcomes…

GPRA4 – SS/HS grant sites will increase mental health services to students and families during the 3-year grant period.

Outcome: Objective 4. In each of three years cumulatively, increase by 50% the percentage of students and families accessing and receiving mental health services, as measured by frequency of service requested or inquired about, follow-through in accessing such service, and persistence in pursuing such service to closure.

Qualitative Analysis

Qualitative analysis seeks to capture the richness of people's experience in their own terms. Understanding and meaning emerge from in-depth analysis of detailed descriptions and verbatim quotations.

QUANTITATIVE (Deductive)

• Ho, Ha• RQ• Variables• Predict• Instruments before

study starts• Difference over time.

Differentiating Approaches

QUALITATIVE(Inductive)

• Ho, Ha• RQ• Single concept (central phenomena)• Build themes• Open-ended questions.

Stance-emerge during the study

• Deep understanding.

Research Methodology

Experimental design Correlational design Survey design Grounded theory design Ethnographic design Narrative design Mixed design Action research design

Experimental design

To establish possible cause-effect between IV and DV. Between-group designs

True experiments Quasi-experiments Factorial analysis

Within-group or Individual designs Time series experiments (interrupted, equivalent) Repeated measures experiments Single-subject experiments

Correlational design

To predict scores and explain the relationship among variables.

Raw Data: Self-esteem GPA score1 42 3.82 10 1.43 15 2.54 22 3.1

Resource: http://www.uky.edu/Classes/PSY/313-001/research%20method%20ch2s.ppt

Survey Design

To describe attitudes, opinions, behaviors, or characteristics.

o Cross-sectional survey designs

Resource:http://images.freshmeat.net/editorials/r_intro/images/line-graph-1.jpg

Survey Design…

O Longitudinal survey

Designs

Resource: http://www.nature.com/eye/journal/v21/n6/images/6702322f1.jpg

Survey Design…

o Trend studies

Resource: http://coastalchange.ucsd.edu/images/wet_dry_2.gif

Survey Design…

o Cohort studies

Resource: http://www.ncsall.net/fileadmin/resources/research/br11-graph2.gif

Survey Design…

o Panel studies

Resource: http://www.nsf.gov/news/special_reports/survey/images/dynamic_logo_sm.jpg

Grounded Theory Design

Systematic design Open coding Axial coding Selective coding

Grounded Theory Design…

Emerging design Exists at the abstract level. Grounded in data Meet four central criteria:

fit work relevance modifiability

Grounded Theory Design…

Constructivist design Position between a positivist view and postmodern

view. Interest on views, values, beliefs, feelings,

assumptions, and ideologies of individuals. Explain the experience of the phenomenon or

process rather than facts and acts.

Ethnographic Design

Realist ethnography Confessional ethnography Life history Autoethnography Microethnography Ethnography case study

Ethnographic Design…

Critical ethnography Feminist ethnography Postmodern ethnography Ethnography novels

Narrative Research Design

Autobiographies Biographies Life writing/histories/stories Personal accounts/narratives/documents Narrative interviews So on

Mixed Methods Design

Triangulation Mixed Methods Designs Embedded Mixed Methods Designs

Resource: Hanson et al, 2005

Mixed Methods Design…

Explanatory Mixed Methods Designs Exploratory Mixed Methods Designs

Resource: Hanson et al, 2005

Mixed Methods Design…

Resource: Burke & Onwuegbuzie, 2004

Mixed Methods Design…

Resource: Hanson et al, 2005

Action Research Design

Practical Participatory

Sampling

Random sampling methods: Simple random sampling

Sampling…

Stratified random sampling Cluster random sampling

Resource: http://www.socialresearchmethods.net/kb/Assets/images/sampstrt.gif

Sampling…

Two-stage random sampling: combine Cluster random sampling & Simple random sampling

Resource: http://www.clas.ufl.edu/jur/200411/images/emerson_2.jpg

Sampling

Nonrandom sampling methods: Systematic sampling

Resource: http://www.disector.com/stereology-analysis-graphics/sampling1.gif

Sampling…

Convenient sampling

Resource: http://www.nedarc.org/nedarc/collectingData/participantSelection/probabilitySampling/images/simpleSample.gif

Sampling…

Purposive sampling

Resource:http://www.david.curtis.care4free.net/esfig3.gif

Sample Size

It depends on the type of study. However, the larger the sample the better.

Quasi-Experimental Design

REQUEST FOR APPLICATIONS Evaluation of State and Local Education Programs and Policies CFDA Number: 84.305E

C. Implementing Rigorous Evaluations “The methodological requirements for evaluations under

this program…other experimental and quasi-experimental designs that substantially minimize selection bias or allow it to be modeled may be employed” (pg.7).a. Random Assignment “Studies using random assignment to intervention and

comparison conditions have the strongest internal validity for causal conclusions and thus are preferred whenever they are feasible” (pg.7).

Threats to Validity: Internal

Related to participants: History Maturation Regression Selection Mortality Interaction with selection

Related to treatments: Diffusion of treatments Compensatory

equalization Compensatory rivalry Resentful

demoralization

Related to procedures: Testing Instrumentation

Threats to Validity: External

External: It may affect generalizability: Interaction of selection

and treatment Interaction of setting

and treatment Interaction of history

and treatment

Threats to Internal Validity in Types of Experimental Designs  True Experiment Quasi-Experimental

To Participants: Controlled Potential Threat

History Controlled Potential Threat

Maturation Controlled Potential Threat

Regression Controlled Potential Threat

Selection Controlled Potential Threat

Mortality Controlled Potential Threat

Interaction of selection and maturation, history, and instrumentation.

Controlled Potential Threat

Threats to Internal Validity in Types of Experimental Designs

 

True Experiment Quasi-Experimental

To Procedures:    

TestingPotential Threat if pre and posttest

used

Potential Threat if pre and posttest used

Instrumentation

Potential Threat if instrument or observational

procedures change

Potential Threat if instrument or observational

procedures change

Types of Between-Group Designs Quasi-Experimental Designs

Select Control Group Pretest No treatment Posttest

Select Experimental Group Pretest Experimental Treatment Posttest

Pre- and Posttest DesignTime

Select Control Group No treatment Posttest

Select Experimental Group Experimental Treatment Posttest

Post-Only DesignTime