Upload
amy-cassata-phd
View
66
Download
1
Embed Size (px)
Citation preview
Making Sense of Measuring Implementation in Educational Research
Amy Cassata & Jeanne Century MSPNet Webinar
February 25th, 2015
Today’s Agenda
• Defini&ons, Theory and Background • Implementa&on Measurement and the
Common Guidelines • Design and Measurement Approaches • Analysis Strategies
• Study #1: Exploratory -‐ Nine School Districts – Science 1998-‐2002 • Study #2: Literature Synthesis 2006-‐2010 • Study #3: FOI of Mathema&cs and Science Materials 2007-‐2010 • Study #4: Implementa&on Study – STEM Schools (2) 2010-‐2013 • Study #5: Implementa&on Study – Elementary Mathema&cs 2011 -‐ 2015 • Study #6: Instrument Development/Valida&on Study – IES 2011-‐2015 • Study #7: Implementa&on Study – Computer Science 2013-‐2016
Previous and Current Work
• What is an innova&on? An interven&on?
• What is fidelity of implementa&on? How is it different from “implementa&on” (use)?
• What is the implementa&on process?
Defini&ons
Intervention Component Framework
Interven(on Component Framework
Categories of Cri&cal Components
Structural Interac&onal
Procedural Educa&ve Interac&onal (Leader)
Interac&onal (Par&cipant) (engagement)
This work was funded in part by the Na(onal Science Founda(on
What is the “intervention?” What is the “IT?”
FACTORS
INTERVENTION
OUTCOME B OUTCOME A OUTCOME A’
FACTORS
FACTORS
OUTCOME B OUTCOME A’ OUTCOME C OUTCOME B’
FACTORS
FACTORS
Time
INTERVENTION’ INTERVENTION’’
characteris&cs of the users factors
In the context of the interven&on • self-‐efficacy • understanding of the interven&on • actude toward the interven&on • intrinsic mo&va&on • extrinsic mo&va&on
Descrip&ve characteris&cs • demographic • educa&on • experience
characteris&cs of the users factors
NOT in the context of the interven&on • innova&veness • resourcefulness and coping • networked-‐ness • &me management and organiza&onal skills
Percep&ons • perceived adaptability • perceived visibility • ease of use • perceived effec&veness
characteris&cs of the organiza&on
factors Descrip&ve Characteris&cs • organiza&onal structures • physical environment • popula&on characteris&cs • stakeholder community
support • presence of opportuni&es for
learning inside the organiza&on
People in the Organiza&on • informa&on sufficiency • informa&on sharing • shared beliefs and values • Resources • locus of decision making • u&lity of learning opportuni&es • organiza&onal efficacy • collabora&on • leadership
Organiza&onal Strategies • ongoing improvement
structures • leveraging • implementa&on strategy • family/community
communica&on
characteris&cs of the environment factors
• poli&cal environment • community beliefs and values • descrip&ve community characteris&cs • opportuni&es for learning • network structures • extraneous events or ini&a&ves
• Defini&on of fidelity of implementa&on
• Measuring “dimensions” of implementa&on: structure and process
• Need to iden&fy the interven&on model and the cri&cal elements &ed to intended impacts
Areas of Convergence
Design and Development Efficacy Study Effec&veness Study Scale-‐up Study
Purpose Draw on exis(ng theory and
evidence to design and itera(vely develop interven(ons or strategies.
Tes(ng a strategy or interven(on under “ideal” circumstances
Examine a strategy or interven(on under “typical” circumstances or condi(ons of rou(ne prac(ce
Examine a strategy or interven(on in a wide range of popula(ons, context, and circumstances
Que
s&on
s
Are users able to enact the interven(on as intended(feasibility and fidelity) and if not, why (factors)?
Does it work? (fidelity)
Does it work on a large scale? (fidelity, use, factors)
How does it work in natural seLngs and what what factors support and inhibit implementa(on? (use, factors)
Implem
enta&o
n Measuremen
t • Iden(fying essen(al components
• Organizing components • Disentangling interven(on
components from general quality indicators
• Measures of appropriate grain size
• Instruments for both treatment and control condi(ons
• Affordable designs that will yield sufficient data to differen(ate treatments
• Crea(ng scales and indices represen(ng implementa(on that can be linked to outcomes
• Appropriate analy(c approaches
• Rela(onships between factors and implementa(on
• Rela(onships between implementa(on and outcomes
• Describe implementa(on over (me
• Op(mal implementa(on for par(cular popula(ons
Measuring Implementa&on in Common Guidelines Studies
• Implementa&on measurement is more than just “fidelity of implementa&on.”
• All ini&a&ves ( programs, interven&ons, reforms, innova&ons) are comprised of components.
• Factors influence enactment of components.
• Components and factors are measurable. • Be clear about what the “it” is.
Takeaways
Contact Information: [email protected] [email protected]