Evaluation, Government Performance and Results Act and Performance ReportsFY06 ERCM Initial Grantee MeetingDecember 7, 2006, San Antonio, TX

U.S. Department of Education, Office of Safe and Drug-Free Schools400 Maryland Avenue, SW / Washington, DC 20202

Tara Hill

Michelle Sinkgraven

Sara Strizzi

Program Analysts, U.S. Department of Education, Office of Safe and Drug-Free Schools

2

Session Objectives

• Review the importance of evaluation• Discuss key evaluation components• Discuss creating performance measures

and an evaluation plan• Highlight the Emergency Response and

Crisis Management (ERCM) Government Performance and Results Act (GPRA) measures for FY 2006 grantees

• Explain how to collect data and report on the GPRA measures on the ED 524B form

3

Why Evaluate ERCM Projects?

Because evaluation….

• Is a way to gather school-based and community data to help improve emergency management plans

• Uncovers new information or consequences that were not anticipated

• Involves multiple stakeholders in the process and creates buy-in for emergency management planning

4

What Evaluation Does…

• Determines if a grant project is accomplishing the performance objectives

• Supports decision-making in the district and community

• Provides data for communicating to stakeholders the benefits and purpose of the emergency management plan

• Builds a knowledge base about what does and does not work in an emergency management plan

• Helps the U.S. Department of Education report on progress

5

What Evaluation Does NOT Do…

• Attempt to judge a program, project, or school system subjectively

• Aim to represent a false picture of program success for administrators or funders

6

Creating an ERCM Evaluation Plan:

Key ComponentsKey questions to consider

• Who will conduct your evaluation?

• Who are the key stakeholders in your grant project and your overall emergency management efforts?

• Have you captured the appropriate data at the beginning of your project?

• What are your key project objectives?

7

Reviewing Project Objectives

ERCM grant objectives

• Project-specific objectives and performance measures• Drawn from needs assessment• As reflected in grant applications• Based on individual school and district issues

• Government Performance and Results Act measures

8

Project-Specific Objectives & Performance Measures

Project Objective:

• A specific, measurable statement about what the project will achieve.

• Hint: Start the objective with the word "To" followed by a verb. • For example, "To increase the number of bus drivers

trained in CPR from 5 bus drivers to 25 bus drivers.

9

Project-Specific Objectives & Performance Measures

(continued)Performance Measure:

• A value or characteristic that can be used to determine the extent to which the objective has been achieved.

• For example, the # of bus drivers trained in CPR between September 1, 2006-March 31, 2008.

10

Government Performance and

Results Act (GPRA) • What is GPRA?• Purpose of the GPRA• ED's response to GPRA• GPRA measures for the ERCM Program• Measure 1• Measure 2• Measure 3

11

What is GPRA?

• The Government Performance and Results Act of 1993 requires all federal agencies to manage activities with attention to outcomes.

• Agencies must:• State intended accomplishments• Identify the resources required• Periodically report to Congress

12

Purpose of the GPRA

• Continuous monitoring of program outcomes results in:

• Improved accountability for expenditures using public funds

• Informed Congressional decision making through use of objective information

• Government focus on results

13

U.S. Department of Education's Response to

GPRA• Develop a strategic plan for 2002-2007• Plan reflects the Department's goals and

priorities• Department goals:

• Goal 1: Create a Culture of Achievement• Goal 2: Improve Student Achievement• Goal 3: Develop Safe Schools and Strong Character• Goal 4: Transform Education into an Evidence-based

Field• Goal 5: Enhance the Quality of and Access to

Postsecondary and Adult Education

• Goal 6: Establish Management Excellence

14

What are the ERCM GPRA Measures?

• Measure 1: Demonstration of increased number of hazards addressed by the improved school emergency response plan as compared to the baseline plan.

• Measure 2: Demonstration of improved response time and quality of response to practice drills and simulated crises.

• Measure 3: A plan for, and commitment to, the sustainability and continuous improvement of school emergency response plans by the district and community partners beyond the period of Federal financial assistance.

15

Measure 1

• Demonstration of increased number of hazards addressed by the improved school emergency response plan as compared to the baseline plan.

16

Measure 1: Increased Number of HazardsDefinitions:• Hazards: Broadly defined as potential threats to

the school / school district whose date or time of occurrence cannot be predicted. Examples include:• Terrorist attack• Tornado• Earthquake• Chemical spill• Unlawful entry by unlawful person• Criminal activity

• Addressed: Written plan is in place that responds to hazards

• Baseline plan: Emergency plan in place prior to grant award date

17

Data Needed for Measure 1

• Number of hazards addressed by the improved school emergency plan

• Number of hazards addressed by the baseline plan

• Measure can include information on number of hazards at the district level and at the school level

18

Completing the 524B

Sample scenario: ABC School District had a basic emergency plan in place prior to the grant that addressed fires, tornadoes, and unlawful entry. At the end of the grant, the district addressed three additional hazards: terrorist attack, bomb threat, and active shooter.

19

Completing the 524B• Box 1.a. Performance Measure:

• Number of hazards addressed by the improved school emergency response plan as compared to the baseline plan.

• Measure Type:• GPRA

• Quantitative Data:• Step 1: Enter 999 in all data fields• Step 2: Provide numerical answers to each of the

following in the "Explanation of Progress" section: Number of hazards addressed in original (baseline) plan Number of hazards addressed in final plan Number of additional hazards addressed by final plan

Formula: # of hazards in final plan - # of hazards in baseline plan

20

Completing the 524B Form

• Additional information to include in the "Explanation of Progress" section:

• Provide a narrative that explains what the numbers mean and how they were calculated.

• Be specific and list additional hazards that have been addressed.

• Describe any barriers that prevented you from meeting your goal, if applicable.

21

FAQs• Can I count facilities improvements as

hazards that my school/school district has addressed?

• What if our district did not include funds to address hazards that were uncovered in our vulnerability assessment?

• Do I need to report school-by-school data on the 524B?

• My application is for a consortium of school districts. How should I report data for this measure—do I aggregate or report on a district-by-district basis?

22

Measure 2

• Demonstration of improved response time and quality of response to practice drills and simulated crises.

23

Measure 2: Improved Response Time and Quality of

Response Definitions:• Response Time: The number of minutes

and seconds it takes to respond to a practice drill or simulated crisis. The type of response depends on the type of drill staged. Drills include, but are not limited to:• Fire drill• Lockdown• Community-wide emergency response drill• Hazardous weather drill• Active shooter drill• Shelter-in-place drill• Evacuation

24

Measure 2: Improved Response Time and Quality of

ResponseDefinitions, continued:

• Demonstration of improved response time: A quantifiable decrease in the amount of time it took to respond to a practice drill or simulated crisis that was staged at the beginning of the performance period and the end of the performance period.

• Quality of response: To respond to this portion of the measure, attach a copy of the after-action report for each drill and clearly state what has been done to make improvements in quality.

25

Data Needed for Measure 2

• Response time:

• Baseline (Drill #1): Response time to drill conducted at the beginning of the performance period or before the grant was awarded.

• Drill #2: Response time (to same type of drill) at the end of the performance period.

• Quality of response:

• A thorough narrative description of the drills documenting lessons learned and a plan for addressing any shortcomings. Attach an after-action report for each drill.

26

Completing the 524B

Sample scenario: ABC School District conducted a tornado drill. An after-action report highlighted confusion about designated safe areas. In response to the report, students were briefed and shelter-in-place signs were hung around school. A few months later, a second tornado drill was conducted. The after-action report attributed the 3 minute, 16 second reduced response time to the changes implemented from the lessons learned in the first (baseline) drill.

27

Completing the 524B• Box 1.a. Performance Measure:

• Demonstration of improved response time and quality of response to practice drills and simulated crises.

• Measure Type:• GPRA

• Quantitative Data:• Step 1: Enter 999 in all data fields• Step 2: Provide numerical answers to each of the

following in the "Explanation of Progress" section: Actual time for drill #1 (baseline drill) Goal (target) time for drill #2 Actual time for drill #2 Difference between drill time #1 and drill time #2

28

Completing the 524B• Additional information to include in the

"Explanation of Progress" section:

• Provide a narrative for the data provided, explaining what the numbers mean and how they were calculated

• Be specific, include information about what went right and wrong and how times were calculated

• Provide the rationale for the drill #2 goal/target and when it was established (I.e. upon completion of drill #1)

• Describe any barriers that prevented you from meeting the goal for drill #2, if applicable

• Include a qualitative analysis of each drill, such as an after-action report

29

FAQs

• Do I need to compare similar types of drills for reporting purposes?

• Does my first (baseline) drill need to occur during the grant performance period or can we use data from a prior drill or simulation?

• What if our grant application specifies running a variety of drills only once?

• Can I report on community-wide crisis drills?

30

Measure 3

• A plan for, and commitment to, the sustainability and continuous improvement of school emergency response plans by the district and community partners beyond the period of Federal financial assistance.

31

Measure 3: Plan for and Commitment to Sustainability and Continuous Improvement

Definitions:

• Plans: The emergency management plan developed under the ERCM grant.

32

Data Needed for Measure 3• Provide the answer to the following

question: "Do we have a plan for, and commitment to, the sustainability and continuous improvement of school emergency response plans by our district and community partners beyond the period of Federal financial assistance?"

• A simple "yes" or "no" is not sufficient.• A copy of the plan should be provided with

your final report.

• In addition, an objective analysis of whether or not the plan meets this measure should be provided.

33

Completing the 524B

• Box 1.a. Performance Measure:• A plan for, and commitment to, the sustainability and

continuous improvement of school emergency response plans by our district and community partners beyond the period of Federal financial assistance.

• Measure Type:• GPRA

• Quantitative Data:• Target, Raw Number: YES• Actual, Raw Number: YES, if your district has met this

standard. If not, NO.

34

Completing the 524B

• Explanation of Progress:

• Provide a detailed answer to the question, "Do we have a plan for, and commitment to, the sustainability and continuous improvement of school emergency response plans by our district and community partners beyond the period of Federal financial assistance?"

• Provide proof of your plan. This could include, but is not limited to, the following:

Copies of updated Partner Agreements Timelines/agenda for upcoming community partner

meetings/drills after the grant period is over Your district's completed emergency plan

35

FAQs

• Do I need to submit a copy of my district's completed emergency plan with my final report?

• What are some ways a district might demonstrate sustainability?

• Do I need to submit signed copies of all Memorandums of Understanding (MOU) at the conclusion of the grant period?

36

QUESTIONS