Upload
carlow
View
44
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Continuous Improvement Research in Education AND Evaluation of State & Local Education Programs & Policies (Topics 2 & 3 of 84.305H). James Benson, Ph.D. Allen Ruby, Ph.D. National Center for Education Research. Overview . Overview of IES and its mission Requirements for both topics - PowerPoint PPT Presentation
Citation preview
ies.ed.gov
Connecting Research,Policy and Practice
Continuous Improvement Research in Education
ANDEvaluation of State & Local Education
Programs & Policies(Topics 2 & 3 of 84.305H)
James Benson, Ph.D. Allen Ruby, Ph.D.
National Center for Education Research
ies.ed.gov
Overview
• Overview of IES and its mission• Requirements for both topics• Specifics for each topic
– Purpose– The project narrative
• Significance• Partnership• Research Plan
• The project narrative– Personnel and Resources
• Other important sections of the application• Preparing and submitting an application
2
ies.ed.gov
Legislative Mission of IES
• Describe the condition and progress of education in the United States
• Identify education practices that improve academic achievement and access to education opportunities
• Evaluate the effectiveness of Federal and other education programs
3
ies.ed.gov
Organizational Structure of IES
4
National Board for Education
SciencesStandards &
Review Office
Office of the Director
National Center for Education Evaluation
National Center for Education Statistics
National Center for Education Research
National Center for
Special Education Research
ies.ed.gov
IES Grant Programs: Research Objectives
• Develop or identify education interventions (i.e., practices, programs, policies, and approaches) – that enhance academic achievement– that can be widely deployed
• Identify what does not work and thereby encourage innovation and further research
• Understand the processes that underlie the effectiveness of education interventions and the variation in their effectiveness
5
ies.ed.gov
Partnerships & IES Priorities
IES seeks to... • Encourage education researchers to develop
partnerships with stakeholder groups to advance relevance of research and usability of its findings for day-to-day work of education practitioners and policymakers
• Increase capacity of education policymakers and practitioners to use knowledge generated from high quality data analysis, research, and evaluation through wide variety of communication and outreach strategies
(See http://ies.ed.gov/director/board/priorities.asp)
6
ies.ed.gov
Partnerships & IES Research Grant Programs
• Partnerships and Collaborations Focused on Problems of Practice and Policy (84.305H)– To further promote research partnerships between
research institutions and State and local education agencies (SEAs/LEAs)
– Contains 3 topics• Researcher-Practitioner Partnerships in Education Research• Continuous Improvement Research in Education• Evaluation of State & Local Education Programs & Policies
7
ies.ed.gov
The Two Topics
• Continuous Improvement Research in Education (Continuous Improvement) – Well-established partnerships– Goal: To adapt and revise a specific approach, using a continuous improvement
strategy, to address a specific education issue or problem of high importance to the education agency that has important implications for improving student education outcomes
• Evaluation of State & Local Education Programs & Policies (State/Local Evaluation) – New or established partnerships– Goal: To carry out rigorous evaluations of education programs or policies
(programs/policies) that are implemented by state or local education agencies and have important implications for improving student education outcomes
ies.ed.gov
Requirements Shared by Both Topics
• Focus on student education outcomes• Partnership between research institutions and
SEAs/LEAs
ies.ed.gov
Focus on Student Outcomes
• IES funds research to improve the quality of education for all students through advancing the understanding of and practices for teaching, learning, and organizing education systems
• All research must address education outcomes of students– Academic outcomes– Social and behavioral competencies that support
student success in school
10
ies.ed.gov
Student Population
• Students from prekindergarten through postsecondary and adult education– Typically developing students – Students with disabilities or at risk for disabilities• Specific requirements for identifying students at risk for
disabilities status • see http://ies.ed.gov/ncser/definition.asp
11
ies.ed.gov
Ultimate Outcomes of Interest: Student Outcomes
Grade OutcomePrekindergarten School readiness (e.g., pre-reading, language,
vocabulary, early math and science knowledge, social and behavioral competencies)
Kindergarten – Grade 12
Learning, achievement, and higher-order thinking in reading, writing, mathematics, and science; progress through the education system (e.g., course and grade completion or retention, high school graduation, and dropout); social and behavioral competencies important to academic and post-academic success
12
ies.ed.gov
Ultimate Outcomes of Interest: Student Outcomes
Grade OutcomePostsecondary(Grades 13 – 16:baccalaureate and sub-baccalaureate)
Access to, persistence in, progress through, and completion of postsecondary education; for students in developmental programs, additional outcomes include achievement in reading, writing, English language proficiency, and mathematics; success in gateway math and science courses, introductory English composition
Adult Education(Adult Basic Education, Adult Secondary Education, Adult ESL, and GED preparation)
Student achievement in reading, writing, English language proficiency, and mathematics; access to, persistence in, progress through, and completion of adult education programs
13
ies.ed.gov
Applications must be from a Partnership
• Partnership must include at least a research institution and a U.S. education agency
• Applications must include at least one Principal Investigator (PI) from a research institution and at least one PI from an SEA or LEA– PI from research institution: Must have the ability and
capacity to conduct scientifically valid research and expertise in the education issue to be addressed
– PI from SEA or LEA: Must have decision-making authority for the education issue within his or her agency
14
ies.ed.gov
Partnership
• Length of Partnership– Continuous Improvement
• document at least 1 year of collaboration and describe products
– State/Local Evaluation• may be new or existing partnership
• Broad definition of research institution– Ability and capacity to conduct scientifically valid research
ies.ed.gov
Partnership: SEA Partner
• State education agencies – Examples: education agencies, departments, boards,
commissions – Oversee early learning, elementary, secondary,
postsecondary/higher, and adult education– Includes education agencies in U.S. territories and tribal
education agencies
16
ies.ed.gov
Partnership: LEA Partner
• Local education agencies which are primarily public school districts
• Community college districts• State and city postsecondary systems
– If there is a state or city higher education agency that oversees the postsecondary system, include them as an agency partner
– If there is no state or city education agency that oversees the postsecondary system, the system can apply as the sole agency partner
– A postsecondary system that applies as an education agency partner cannot also serve as the research institution partner in the same project
17
ies.ed.gov
Additional Partners
• Partnerships may include more than one State or local education agency if they share similarities and interests
• Non-education state and local agencies may be partners as long as an education agency is a partner
• Partnerships may include more than one research institution if they have shared interests and will make unique contributions
• Partnerships may include other non-research organizations (e.g., issue-oriented or stakeholder groups) that will contribute to the partnership and its work
18
ies.ed.gov
Quick Check
Check the fit between your research and the topic!• Just because you have a partnership, doesn’t mean
the Continuous Improvement topic or the State/Local Evaluation topic is the best grant topic for you
• Potentially better fit– Education Research Grants Program (84.305A) or – Special Education Research Grants Program (84.324A)
ies.ed.gov
Specifics about the Topics
• Continuous Improvement Research in Education
• Evaluation of State & Local Education Programs & Policies
ies.ed.gov
Continuous Improvement: General Purpose
• Promote joint research by partnerships of research institutions and SEAs/LEAs– Addresses an education issue or problem of key
importance to an SEA/LEA– Directly contributes to solving problems faced by an
SEA/LEA
21
ies.ed.gov
Continuous Improvement: Specific Purpose
• Implement, adapt, and revise an educational approach to address the education issue or problem of concern to the SEA/LEA, with the aim of improving student outcomes
• Increase the agency’s capacity to carry out research, development, and implementation
• Contribute to our understanding of how approaches can be adopted to address local conditions and wide implementation
22
ies.ed.gov
Continuous Improvement: Expected Products of the Grant
• Description of the partnership as it developed during the grant
• Description of the approach in use by the end of the project• Description of the process of continuous improvement used
to adapt/revise the approach and the measures used in that process
• Results from an ongoing comparison of student outcomes in sites (e.g., schools/classrooms) where the approach is being adapted and revised, compared to sites that are not trying to implement the approach
23
ies.ed.gov
Continuous Improvement: Expected Products of the Grant
• Recommendations for how the partnership could be maintained over the longer term
• Specific and general lessons from the revisions to the approach and changes made in the education system that improved the approach and its implementation
• Lessons learned from the joint development work performed by the partnership that could benefit other partnerships
24
ies.ed.gov
Education Issue and Approach
• Applicants may propose to address any education issue or problem of priority to the LEA/SEA
• An approach is defined as a policy, program, intervention, practice or combination thereof that addresses a problem/issue of high importance to an education agency, and that has a strong theoretical and/or empirical rationale for improving student education outcomes.
25
ies.ed.gov
The Project Narrative
•Significance•Partnership•Research Plan•Personnel•Resources
26
ies.ed.gov
Significance
In the Significance section, clearly describe… • The education issue/problem• The approach to be adapted/revised• The education system• Current practice in the education system
ies.ed.gov
Significance
• Describe the education issue to be addressed – Its links to student education outcomes– Its importance to the education agency’s decision making – Its importance to other education agencies, policymakers,
and education research
28
ies.ed.gov
Significance
• Describe the proposed approach to be implemented, adapted, and revised in order to address the education issue– Describe the theory of change for the approach– Provide any empirical evidence that the approach can be
successfully implemented– Provide any empirical evidence that the approach can
improve student outcomes
29
ies.ed.gov
Significance
• Describe the education system– Describe the education system(s) where implementation
will occur • E.g., classroom, school, district, multiple districts, state
– Discuss why the approach will need adaptation and revision for successful implementation in this system
– Discuss how the system(s) may need to change to support successful implementation
30
ies.ed.gov
Significance
• Describe current practice in the education agency– Describe how the education agency is currently addressing
the education issue– Discuss why current practice is not satisfactory– Describe the current status of approach within the
education agency
31
ies.ed.gov
The Project Narrative
•Significance•Partnership•Research Plan•Personnel•Resources
32
ies.ed.gov
Partnership
In the Partnership section, clearly describe… • The partnership• Partnership development • Tracking the partnership’s success
ies.ed.gov
Partnership Description
• Describe the partners– The research institution and the education agency – Any other members of the partnership– The partnership’s previous work (1-year minimum) and
resulting products– Partners’ common interests and complementary abilities
– How all members contribute to and benefit from the partnership
– How the partners decided to propose a Continuous Improvement project
– Management structure and procedures to keep the project on track and quality control
ies.ed.gov
Partnership Development
• Planned activities and processes to further develop the partnership– How will these activities and processes contribute to the
research, agency capacity building, and future collaborations?
• Partnership decision-making process– How will you determine next steps in research,
dissemination, capacity building, and future research?• Building the education agency’s capacity to carry out
research, development, and implementation
35
ies.ed.gov
Tracking the Partnership
• Monitoring the success of the partnership– During the project: Maintaining the partnership,
completing the adaptation and revision of the approach, completing the ongoing comparison
– After the project: Opportunities for the partnership to continue and for the agency to be more able to carry out research, development, and implementation
• IES encourages projects to propose additional indicators of partnership success
36
ies.ed.gov
The Project Narrative
•Significance•Partnership•Research Plan•Personnel•Resources
37
ies.ed.gov
Research Plan
In the Research Plan, clearly describe… • A measurement strategy and plan for data collection• The continuous improvement process• An ongoing comparison study of student outcomes
38
ies.ed.gov
Measurement Strategy
• Describe how you will collect data and measure for:– How well the approach is functioning including its:
• Usability: Can intended user physically implement the approach as well as understand it and be willing to use it?
• Feasibility: Is the approach usable within the constraints of the education system?– Progress toward the desired outcomes (as set out in the theory of change)
• Short-term• Intermediate• Final
– What attributes of the approach and the education system need revision• Describe measure construction, including any new measures the
project will need, AND how the measures will inform the improvement process
39
ies.ed.gov
Continuous Improvement Process
• Describe the Improvement Process– Starting Approach: how the approach will be obtained and
initially implemented– Analysis Process: how collected data will be studied and
interpreted to determine adaptations and revisions – Implementation Process: how the iterative revisions of the
approach will be implemented• Describe the Monitoring of the Improvement Process– Infrastructure and processes to keep work on track– Identification of needed changes in the education system
40
ies.ed.gov
Ongoing Comparison Study
• Detail a plan for an ongoing comparison of student outcome between sites taking part in adaptation and revision of the approach and those not involved– Comparability of comparison sites– Timing of comparisons
• From start of project• Not need to be every improvement cycle but should take place every semester
or year)– Student outcomes to be compared (using easily available data)– Analysis plan– Comparison site practice (based on non-intensive study)
• Not expected to have the rigor of a pilot study or an efficacy study (as described in 84.305A or 84.324A)
41
ies.ed.gov
Specifics about the Topics
• Continuous Improvement Research in Education
• Evaluation of State & Local Education Programs & Policies
ies.ed.gov
State & Local Evaluation: Purpose
• Promote joint evaluation research by research institutions and SEAs/LEAs– On an education program/policy of key importance to
SEAs/LEAs– That will directly contribute to SEA/LEA program and policy
decisions – Provide opportunities to develop the partnership through
the evaluation• Foster longer-term research partnerships– Provide and support the use of rigorous research-based
evidence in decision making– Continue practitioner input into research agenda
43
ies.ed.gov
What should the partnerships do during the grant?
Broadly, the partnerships should… • Identify an education program or policy • Implemented by an SEA/LEA
• Of high priority to that agency
• Intended to improve student education outcomes
• Carry out an evaluation of that program/policy
44
ies.ed.gov
Expected Products of the Grant
• Causal evidence of the impact of a clearly specified program/policy implemented by an SEA/LEA – Overall impacts– Impacts under a variety of conditions
• Conclusions on and revisions to the theory of change that guides the program/policy– Contributions to our theoretical understanding of
education processes and procedures
45
ies.ed.gov
Expected Products of the Grant
• If a beneficial impact is found… – The organizational supports, tools, and procedures needed
for sufficient implementation of the core components of the program/policy under routine practice should be identified
• If a beneficial impact is not found…– A determination should be made whether and what type of
further research would be useful to revise the program/policy and/or its implementation
• The financial costs of the program/policy
46
ies.ed.gov
The Project Narrative
•Significance•Partnership•Research Plan•Personnel•Resources
47
ies.ed.gov
Significance
In the Significance section, clearly describe…• The education program or policy to be evaluated– Components– Processes and materials to support implementation– Evidence it is ready to be or already implemented– How it differs from existing practice
ies.ed.gov
Significance
• Its implementation – By an SEA or LEA– Target population and sites– End users of the program or policy and how they are to
carry it out
ies.ed.gov
Significance
• The theory of change– How the program or policy is to effect changes that
ultimately lead to beneficial impacts to student outcomes– Intermediate outcomes in this process
• Rationale for testing its impact on student education outcomes– In widespread use but not well-evaluated– An alternative to common practice that has a theoretical
(perhaps empirical as well) justification– Improvement on previous research
ies.ed.gov
The Project Narrative
•Significance•Partnership•Research Plan•Personnel•Resources
51
ies.ed.gov
Describe the Partnership
• Describe the partners– The research institution and the education agency – Any other members of the partnership– Common interest in and benefits from this evaluation– The process through which they decided to propose a
State/Local project– Past or ongoing collaborations and results from them– Management structure and procedures to keep the project
on track and quality control
ies.ed.gov
Partnership Development Plan
• Partnership’s decision-making process• Improving the education agency’s capacity to
participate in and use education research– Identify the agency’s interests in capacity building– The agency’s specific understanding of the proposed
research design and the validity and generalization of the evidence provided from it
– The agency’s general capacity to understand and use research
ies.ed.gov
The Project Narrative
•Significance•Partnership•Research Plan•Personnel•Resources
54
ies.ed.gov
Research Plan
• State research questions and hypotheses• Describe sample– Define population and how your sample and sampling
procedures will allow inferences to the population– Exclusion and inclusion rules and their justification– Strategies used to increase participation and reduce
attrition• Describe the setting – Discuss implications for the generalizability of your study
55
ies.ed.gov
Research Plan: Design
Rationale for the Selected Research Design• Causal inference • Threats to internal validity• Degree of equivalence at baseline• Bias from overall and differential attrition• Meet WWC evidence standards (with or without
reservations)
56
ies.ed.gov
Research Plan: Design
Preferred Design: Randomized Controlled Trial (RCT)• Note unit of randomization and justify choice• Describe process for random assignment and
maintaining its integrity• Potential Issues– Entire population: Treatment fidelity– Volunteers: Comparison group status– Lotteries: Attrition of non-accepted parties– Staggered roll out: Little time for true comparison
57
ies.ed.gov
Research Plan: Design
Alternatives to the RCT Design• If RCT is not possible, justify why• Alternatives to minimize or model selection bias– Regression discontinuity designs– Well-designed quasi-experimental designs
• Comparative interrupted time series
58
ies.ed.gov
Research Plan: Statistical Power
• Detailed description of power analysis and justification for method used to calculate power– Including assumptions
• Power for main analyses and important subgroup analyses
• Reviewers should be able to check power calculations
59
ies.ed.gov
Research Plan: Outcome Measures
• Student education outcome measures relevant to states, districts, and schools– Often found in administrative data– Can include researcher-developed measures but not as the
primary outcome measures• Provide reliability, validity, and appropriateness• Intermediate outcomes• Link to theory of change
60
ies.ed.gov
Research Plan: Moderators & Mediators
• May explain differential impacts of intervention• Identified in theory of change
• Describe how they will be measured in both treatment and control
• Discuss if doing exploratory or confirmatory analysis of each one examined
• Describe analysis plan
61
ies.ed.gov
Research Plan: Fidelity of Implementation
• Describe how measures capture core components of the program or policy– Note their psychometric properties
• Describe design and implementation of fidelity study• Measure fidelity in both treatment and comparison
groups
62
ies.ed.gov
Research Plan: Fidelity of Implementation
• Discuss how data will be analyzed and will contribute to overall evaluation
• For secondary data analyses using historical data– Requirement can be waived with documentation of lack of
fidelity data
63
ies.ed.gov
Research Plan: Comparison Group Practice
• Describe who makes up comparison group • Detail how you will measure whether they are
similar/different from treatment group• Detail how you will measure what they receive in
place of the treatment • Determine if control group receives components
similar to intervention and how much
64
ies.ed.gov
Research Plan: Analysis
Detailed description of data analysis procedures• Make clear how analyses directly answer your research
questions and can be done based on the design• Quantitative: Statistical procedures, model, and software• Qualitative: Methods to index, summarize, and interpret data• Will quantitative and qualitative data be used for separate or
combined analyses?• Address clustering of students in classrooms in schools • Address missing data• Include plans for analyses of subgroups, mediators,
moderators, and fidelity of implementation
65
ies.ed.gov
Research Plan: Cost Analysis
• Document financial costs of program implementation– Detailed enough for another state or district to use
• Can include a cost-effectiveness or cost-benefit analysis but not required
66
ies.ed.gov
Research Plan: Dissemination Plan
• Identify all your audiences and how you will disseminate the results to them– The education agency partner (ongoing process)– Other education agencies, policymakers, and practitioners– The research community– The public
67
ies.ed.gov
The Project Narrative
•Significance•Partnership•Research Plan•Personnel•Resources
68
ies.ed.gov
Personnel
• Identify all key personnel on the project team – Roles and responsibilities on the project– Qualifications (i.e., expertise and experience) to carry out the
roles and responsibilities– % FTE on the project (one key person should have enough
time to maintain progress of project)– Past success at working in similar partnerships
• PI qualifications for managing a grant of this size and type
• For State/Local Evaluation– Ensure objectivity of evaluation
69
ies.ed.gov
Resources
• Describe the institutional resources of all the institutions involved in the partnership and how these resources will contribute to building the partnership and to the research– Institutional capacity to manage the grant– Resources available at the partner institutions that will be
used– Plans to acquire any major resources not yet in hand– Joint Letter of Agreement by partners (Appendix D)
70
ies.ed.gov
Resources
• If individual schools are taking part…– Schools should document their involvement– E.g., Letters of Agreement in Appendix D
• If secondary data is being analyzed…– The organization holding those data should document their
willingness to provide the data– E.g., Letters of Agreement in Appendix D
• If school staff are taking part…– E.g., through surveys, observations, logs– Discuss how their cooperation will be obtained (e.g., use of
incentives) and their current knowledge of the project
71
ies.ed.gov
Other Important Sections of the Application
• Appendix A• Appendix B• Appendix C• Appendix D• Budget & Budget Narrative
72
ies.ed.gov
Appendix A
Page Limit: 3
• If you are resubmitting an application, use up to 3 pages to discuss how you responded to reviewer comments
73
ies.ed.gov
Appendix B
Page Limit: 15
• Figures, charts, or tables that supplement the project narrative
• Timelines for the project• Examples of measures to be used – E.g., tests, surveys, observation, and interview protocols
• Do NOT include narrative text
ies.ed.gov
Appendix C
Page Limit: 10
• If you are proposing to study an approach, intervention or assessment, you may include examples of materials used in the intervention or assessment, such as…– curriculum material– computer screen shots– assessment items– other materials
75
ies.ed.gov
Appendix D
No Page Limit• Letters of Agreement from all the research partners– Joint Letter from key partners– Separate Letters from other organizations involved– Letters should clearly state the organization’s expected role in
the partnership and their commitments to the project– Similar letters from any consultants and schools taking part– Letters from holders of data should make clear that the data
described in the application will be provided for the proposed use by the project
76
ies.ed.gov
Budget & Budget Narrative
• Continuous Improvement– The maximum award is $2.5 million – The maximum project length is 4 years
• State/Local Evaluation– The maximum award is $5 million – The maximum project length is 5 years
• Award size depends on the project scope• Include a detailed budget form (SF 424) AND a
narrative that links the activities, personnel, etc. from the Project Narrative to the funds requested
77
ies.ed.gov
Application Deadline
Letter of Intent Due
Date
Application Package Posted
Start Dates
August 7, 2014
4:30:00 PM DC Time
June 5, 2014 June 5, 2014 July 1, 2015to
Sept 1, 2015
Important Dates & Deadlines
78
ies.ed.gov
Information Sources
• Request for Applications– http://ies.ed.gov/funding/
• Letter of Intent– https://iesreview.ed.gov/index.cfm
• Application Package– www.grants.gov– Click on “Find Grant Opportunities”, then “Basic Search”, then
type in “84.305” under CFDA Number, then select “84.305H”
• Program Officers
79
ies.ed.gov
Peer Review(Standards & Review Office)
• Compliance screening for format requirements• Responsiveness screening for program requirements• Assignment to review panel– 2 to 3 reviewers (substantive and methodological)– The most competitive proposals are reviewed by full panel
• Many panelists will be generalists to your topic• There will be an expert in every procedure you use
– Panel provides an overall score plus specific scores on Significance, Partnership, Research Plan, Personnel, and Resources
80
ies.ed.gov
Notification
• All applicants will receive e-mail notification that the following information is available via the Applicant Notification System (ANS):• Status of award• Reviewer summary statement
• If you are not granted an award the first time, consider resubmitting and talking with your Program Officer
81