Thar’s Gold in Them Thar AAR/ IPs

Texas Department of Public SafetyDIVISION OF EMERGENCY MANAGEMENT

Thar’s Gold in Them Thar AAR/IPs


WELCOME AND AGENDA• Purpose and Value of After Action Reports• How to develop an Improvement Plan

Esther Corwin, MEPExercise Training Officer, Texas Division of Emergency [email protected] Info is on handout

mailto:[email protected]


• A consolidation of information gathered during the testing and evaluation of a community’s emergency operations plan through an exercise

• Provides feedback to executive leadership, participating entities and governing agencies in the achievement of the exercise objectives and overall capabilities of the community

• Information gleaned from this process, identifies and guides future improvement actions

WHAT IS AN AAR


WHY WRITE AN AAR• Creates a multi-agency process to review an

exercise or incident• General exercise identification information,

but also includes evaluation information• Facilitates honest and critical appraisal of

actions based on plans and policies


WHERE DO WE LOOK• Exercises to simulate a stressful realistic scene• Exercises to evaluate a new or seldom used

equipment or skill• Incidents presenting unexpected challenges• Incidents having “less than expected” results

OR• Incidents employing innovative actions• Incidents with unexpected positive results

http://www.the3dstudio.com/product_details.aspx?id_product=254449


EXERCISE EVALUATION IS BASED ON• Community policies, Emergency Operations

Plan, agency SOPs and SOGs, MOUs, etc.• Exercise Evaluation Methodology

– S.MA.R.T. objectives– Exercise Evaluation Guides

• Observational Data• Hotwash• Participant Feedback


INCIDENT EVALUATION IS BASED ON• Community policies, Emergency Operations

Plan, agency SOPs and SOGs, MOUs, etc.• Incident Action Plans• Briefings, De-Briefings• Observational Data• Participant Feedback


TYPES OF OBSERVATIONAL DATA• Descriptive reporting

– Typically yields reliable data.• Inferential reporting

– Includes terms like “adequate” or “timely”– Yields inconsistent data

• Evaluative reporting– Includes terms like “efficient” or “successful”– Difficult to reliably to collect


THREE LEVELS OF ANALYSIS


THREE LEVELS OF ANALYSIS

Critical Resource Logistics & Distribution

Onsite Incident Command

Capability Being Evaluated

Capability-Level Analysis

Activity-Level Analysis

Task-Level Analysis

Resource Management – Identify, dispatch, mobilize,

track, demobilize and pay for resources

Direct/Control incident management activities.

Identify the resources based on incident need

Develop all necessary components of an IAP and

obtain approvals

Inventory resources by type/ category that are available to

support this incident

Establish incident objectives and priorities and operational

periods

Leve

l of E

valu

atio

n


EXERCISE EVALUATION GUIDES• Identify the activities, tasks, and performance

measures to be observed• Developed and customized before the exercise • Evaluators complete the EEG by:

– Logging times and actions accurately– Decision making processes and participants– Player roles and responsibilities– Coordination and cooperation– How actions were performed, resources involved– Documenting issues and recommendations


DATA ANALYSIS

The goal is to evaluate the ability of involved functions to perform

target capabilities


PRELIMINARY ANALYSIS• Evaluators organize observations into key

issues and a chronological narrative• At a minimum, should include:

– Description of the assigned function or operation, analyzed by capability, activity, and task

– Documented record of significant observed actions



• Review exercise objectives– What was the intent of the objective?– What would demonstrate the successful

performance of the objective?– If the objective was not met, what factors

contributed to this result?

• Review incident operational objectives

• Review exercise objectives– What was the intent of the objective?– What would demonstrate the successful

performance of the objective?– If the objective was not met, what factors

contributed to this result?

• Review incident operational objectives

REVIEW OBJECTIVES


4 STEPS OF DATA ANALYSIS

1. Identify issues– Compare performance data to standards– Identify differences– Determine consequences

2. Determine root cause– Discuss conditions leading to observed behavior– The “Why” staircase

Fool’s Gold


WHY STAIRCASE

Each step should explain the step above …

… down to the underlying root cause.

Why did it happen?

Why was that?

And why was that?

ROOT CAUSE

And why was that?

Why did that happen?


4 STEPS OF DATA ANALYSIS3. Develop recommendations

– Sustain and improve– Short and long term– What needs to be changed and how– Give references

4. Capture lessons learned– Innovative practice or a piece of knowledge gained from

experience– Provides guidance for approaching a similar problem in

the future– Save time, conserve money, and accelerate preparedness

http://www.123rf.com/photo_5322729_a-gold-or-brass-key-to-find-a-solution-to-a-lock-as-a-jigsaw-puzzle-piece.html


DRAFT AAR• Provides feedback to participating entities and

governing agencies regarding the achievement of objectives and overall capabilities

• Records recommendations for improvement• Establish consensus and buy-in on next steps


THE AAR• Suggested AAR format includes:

– Executive Summary– Overview– Exercise Goals and Objectives– Analysis of Capabilities Demonstrated– Conclusion


• Observations• References• Analysis

– For improvements, list consequences of action or inaction

• Recommendations



• Recommendations



• Recommendations



• Recommendations

OBSERVATIONS AND RECOMMENDATIONS

http://www.123rf.com/photo_10416560_golden-key-in-keyhole-isolated-on-white-background.html


AFTER ACTION CONFERENCE• Planning Team, evaluators, and stakeholders

meet to review and refine draft AAR• Evaluation leads to a disciplined process for

implementing improvement actions and continually strengthening preparedness


AFTER ACTION CONFERENCE• Address identified issues• Develop specific improvement actions• Prioritizes action items• Assigns responsibility to track

implementation


These questions aid development:• What changes need to be made to plans and

procedures to improve performance?• What changes need to be made to

organizational structures?• What changes need to be made to leadership

and management processes?

DEVELOPING IMPROVEMENT ACTIONS

?


More questions to aid development:• What training is needed?• What changes to, or additional

equipment are needed?• What lessons can be learned that will

direct how to approach a similar problem in the future?

DEVELOPING IMPROVEMENT ACTIONS


IMPROVEMENT BENCHMARKS• Must be clearly defined and attainable• Examples include:

– Number of personnel trained in a task– Percentage of equipment that is up-to-date– Finalization of an interagency agreement within a

given amount of time• Include concrete deadlines to track progress

toward full implementation



ACTIVITY – IMPROVEMENT DEVELOPMENT

• Older elementary school in a rural area just before dismissal on a spring day, parents are arriving to take students home

• NWS issues tornado warning estimating less than 10 minutes before impact

• Principal activates “bell” for sheltering, but students/parents confuse it with dismissal

• Some parents enter school to take child and “run” from the storm

As a large group, given a scenario and issue, discuss

possible recommendations and develop a simple improvement

action.


Issue:• Student SafetyBasic Recommendation:• Develop something to

assist the principle in communicating emergency information

• Changes to plans and procedures

• Changes organizational structures

• Changes to leadership/ management processes

• What training is needed• Changes or additional

equipment• Lessons learned for the

future

ACTIVITY – DEVELOP IMPROVEMENT

Jarrell Tornado May 27, 1997


SUMMARY• Objectives and Evaluation Methodology are the

foundation of Improvement Planning• Root Cause analysis helps find the key policy,

procedure, training or equipment to which an improvement could be made

• Applies to Exercises and actual Incidents

Improvements make things better than before


QUESTIONS ???

Any question can lead to a

good idea.

Synergy is the Energy of People.

Esther Corwin, MEPExercise Training Officer, [email protected]

512-424-2198

mailto:[email protected]


Documents

Thar’s Gold in Them Thar AAR/ IPs