CHILD AND FAMILY DISASTER RESEARCH TRAINING AND EDUCATION

1Northwest Center for Public Health Practice

CHILD AND FAMILY

DISASTER RESEARCH

TRAINING AND EDUCATION


Federal Sponsors

NIMH National Institute of Mental Health

NINRNational Institute of Nursing Research

SAMHSA Substance Abuse

and Mental Health Services Administration


Principal Investigators

Betty Pfefferbaum, MD, JD University of Oklahoma Health Sciences Center

Alan M. Steinberg, PhD University of California, Los Angeles

Robert S. Pynoos, MD, MPHUniversity of California, Los Angeles

John Fairbank, PhDDuke University


Evaluating Disaster Mental Health Programs

Part II: From Theory to Practice

Clark Johnson, Ph.D.

Adopted / Modified from materials prepared by: Fran Norris Ph.D., Craig Rosen, Ph.D.

Helena Young, Ph.D.National Center for PTSD


“Disaster research is different from most other fields in that much of the work is motivated by a sense of urgency and concern. Disaster research has both benefited and suffered from this. It has benefited because the cadre of researchers is fluid, and new ideas are accepted and welcomed. It has benefited also because the result has been an impressively diverse database that includes samples from all different regions of the United States[...]. However, disaster research has also suffered from this situation. Scholarship is not always the best because studies often are undertaken under conditions where there simply is not time to absorb a literature that is scattered across a variety of journals and is mixed in quality. Concerns about experimental designs and scientific rigor must often take a back seat to provider beliefs, consumer demands, and clinical necessities. Most of the research is atheoretical and little of it is programmatic. On the basis of this review, we will state our opinion unequivocally that we do not need more research that establishes only that severely exposed disaster victims develop psychological disorders or, worse, that barely exposed disaster victims do not. We need carefully conceived and theory-driven studies of basic process that are longitudinal in design. [...] We need more research that addresses the needs of diverse populations. We need more complex studies of family systems and community-level processes. We need to identify and investigate novel approaches to community intervention, where the intervention itself has been designed to produce collective rather than individual improvements.”

Source :

Norris, Friedman, & Watson. (2002) 60,000 Disaster Victims Speak: Part II. Summary and Implications of the Disaster Mental Health Research.

Psychiatry 65(3), 240-260

We will start the next session in about 10 minutes and will begin with a discussion of the following text


Disaster research is motivated by a sense of urgency and concern.

It has benefited from this because:• The cadre of researchers is fluid, and new ideas are

accepted and welcomed. • The result has been an impressively diverse database

that includes samples from all different regions of the United States


Disaster research is motivated by a sense of urgency and concern.

It has suffered from this situation because:• Scholarship is not always the best because studies

often are undertaken under conditions where there simply is not time to absorb a literature that is scattered across a variety of journals and is mixed in quality.

• Concerns about experimental designs and scientific rigor must often take a back seat to provider beliefs, consumer demands, and clinical necessities.

• Most of the research is atheoretical and little of it is programmatic.


On the basis of this review, we will state our opinion unequivocally that

We do not need • more research that establishes only that “severely exposed

disaster victims” develop psychological disorders or, worse, that “barely exposed disaster victims” do not.

We do need:• carefully conceived and theory-driven studies of basic process

that are longitudinal in design. • more research that addresses the needs of diverse populations. • more complex studies of family systems and community-level

processes. • to identify and investigate novel approaches to community

intervention, where the intervention itself has been designed to produce collective rather than individual improvements.


Let’s pretend we’re starting today

When disaster strikes the Psychosocial health of the children and families in our community will be adversely impacted.

Our goal is to implement a program that is designed to minimize this impact.

The evaluation of this program must be designed to:• Guide development (Proactive / Clarificative)• Monitor implementation (Interactive)• Summarize outcomes and results (Impact)

How should we proceed?


Psychological First Aid (PFA)http://www.ncptsd.va.gov/ncmain/ncdocs/manuals/nc_manual_psyfirstaid.html

PFA is an evidence-informed modular approach for assisting people in the immediate aftermath of disaster and terrorism: to reduce initial distress, and to foster short and long-term adaptive functioning.

It is for use by mental health specialists including first responders, incident command systems, primary and emergency health care providers, school crisis response teams, faith-based organizations, disaster relief organizations, Community Emergency Response Teams, Medical Reserve Corps, and the Citizens Corps in diverse settings.


Ex Ante Evaluation

Overview of key elements• Lessons from the past• Problem analysis and needs assessment• Objective setting• Alternative delivery mechanisms and risk assessment• Added value of this activity• Planning future monitoring and evaluation• Helping to achieve cost-effectiveness

EX ANTE EVALUATION: A practical guide for preparing proposals for expenditure programmes

Available: (http://ec.europa.eu/budget/evaluation/pdf/ex_ante_guide_en.pdf)


Pubmed “Psychological first aid”

Recent publication history "Psychological First Aid"

0

2

4

6

8

10

2002 2003 2004 2005 2006 2007

Year

Pu

bli

shed

Art

icle

s s

May 28th, 2007


Pubmed “Psychological first aid”

23 hits General overview

• 1 - Technology• 6 - Disaster Planning / Policy• 7 - How to’s• 1 - Personal Experiences w/ • 8 - N/A

One article that might help us along• Macy, et.al. (2004).

Community-based, acute posttraumatic stress management: A description and evaluation of a psychosocial-intervention continuum. Harvard Review of Psychiatry. 12,217-228.


Focus on Macy, et. al. (2004). Helps us avoid “re-inventing the wheel”

• Literature – paucity of evidence that CISD is effective (p218) • “Conceptual and practice framework for assessing and intervening with

children, youths, families and their various types of adult caregivers” (p219)• Template for intervention process and practice protocols (p221-222)

• An evaluation study that can be used as a preliminary template (p223)• Stakeholders study• Case records study• Study of training

• Results (p226): Program effective because• It helped communities handle crises• Trained a network of local people to lead or assist with the interventions

• Identifies study’s limitations (p226-7)• No comparison group• No use of standardized / validated instruments• No analysis of quantitative client-outcome data• Long-term effectiveness unclear


Evaluation Study

The study was conducted over a five-month period, between June and October 2003. The design was essentially that of a case study structured to capture PTSM’s essential elements and to enable an assessment of program effectiveness, specifically through a three-component design:

(1) a study of stakeholders in order to assess their views of the program, its impact on individuals and communities, and its quality

(2) a study of case records of interventions with individuals and community groups experiencing traumatic events in order to assess the breadth and depth of the interventions, the manpower and time required, and the effectiveness of the interventions; and

(3) an assessment of the effectiveness of the training that was designed to create a cadre of people to assist with community interventions.


Generate an initial intervention plan

A this point the “plan” is – just a rough sketch of your ideas for:

• Training• Process• ??

Documents include:• Training • Program logic• ??


Ex Ante Evaluation Lessons from the past

Problem analysis and needs assessment• Problem Analysis

• What is the problem to be solved?• What are the main factors and actors involved?

• Needs Assessment• What is the concrete target group• What are the needs and / or interests of this group

Objective setting Alternative delivery mechanisms and risk assessment Added value of this activity Planning future monitoring and evaluation Helping to achieve cost-effectiveness


Problem analysis

Roadmap• Define the key aspects of the situation to be addressed by the

program• Identify factors that are likely to influence the key problem• Identify the main groups of actors that influence or that are

being influenced by the situation• Analyze the cause-effect relations between the factors

identified and the interests and motivation of the actors• Construct a visual presentation of these relationships


Needs assessment

Roadmap• Identify the target population and the most important

subgroups within it• Investigate the situation, motivations and interests of

these groups• Make sure that the identified needs actually

correspond with social, economic and environmental objectives of the community


Ex Ante Evaluation

Lessons from the past Problem analysis and needs assessment

Objective setting• Define general, specific and operational objectives• Define indicators that measure inputs, outputs, results and

impacts Alternative delivery mechanisms and risk assessment Added value of this activity Planning future monitoring and evaluation Helping to achieve cost-effectiveness


Key questions

To Generate: Ask:General Objectives What goal are we working towards?

Specific Objectives What will have changed when we achieve it?

Operative objectives What will be delivered to achieve the goal?

Progress Indicators How will we know if we are on course

Success criteria How can we judge if the action has been successful

Outcome indicators How do we know if the desired change has been effected?


- Inputs - Resources Available for Achieving Goals

Some inputs are tangible resources, such as funding, program staff, office space, supplies, and transportation

Others are less concrete, such as the skills of staff and relationships among staff and with local community leaders

Lack of these resources can greatly limit an organization’s ability to deliver services

Given the unexpected nature of disasters, programs often are initiated before all of the necessary inputs are in place, creating challenges for both the program and its evaluation


-Outputs – The Measurable Units of Products from Program Processes

Evaluations often focus on the outputs of the service delivery process, such as:

• Number of outreach visits concluded• Number of children receiving counseling • Number of people reached in public education• Number of individuals screened and referred for

more extensive treatment

In some cases, evaluations conclude with outputs, which are used as a proxy for outcomes


Indicators Of Outputs And Outcomes

Indicators are the observable measures or standards used to monitor or evaluate program success or outcome (e.g., number of clients receiving services, changes in consumer self-reported symptoms or behaviors, or changes in community conditions)

It is the job of the evaluator to ensure that these criteria are defensible

For indicators of success to be meaningful, they must exhibit good construct validity (measure what they claim to be measuring)


-Processes – Activities or Means to Bring About Program Objectives

Such processes might include:

• outreach to affected people in the community• providing classes or community education on

normal responses to trauma• public relations efforts to increase community

awareness of the agency’s services• training secondary helpers in how to provide

reassurance and support to facilitate recovery• providing brief individual or group counseling or

more extensive intervention• arranging treatment referrals for individuals with

more severe mental health needs


Ready-to-use Data Collection, Data Management, And Reporting Tools

If possible locate and use tools that someone else has developed and validated (!)


Individual Encounter Log Used to document interactions

with individuals or families lasting 15+ minutes and involving participant disclosure.

Captures encounter characteristics, risk categories, participant characteristics, & referrals.

Completed by the crisis counselor immediately after the encounter is over.

Training considerations: Eliciting personal information through “active listening” without asking directly.


Participant Survey (1) Used to obtain feedback

about the program. In one selected week each

quarter, all adults receiving individual or group crisis counseling are given a packet containing a cover letter, survey, pen, and stamped return envelope.

Survey provides some data about immediate outcomes of crisis counseling, such as learning about common reactions to disasters, normalization of feelings and help-seeking, and finding ways to take care of one’s self & family.


Participant Survey (2) Data on disaster experiences

(p. 1) and distress (p. 2) provide information about participant needs.

Distress measure is the SPRINT-E.

Training considerations: The counselor must be convinced that the survey is the recipient’s opportunity to tell the program about community needs and how well program is meeting those needs.


Provider Survey (1) Used to capture crisis

counselors’ opinions about training, resources, services provided, and overall quality of the CCP.

The provider survey is collected anonymously from crisis counselors and their supervisors at 6 and 12 months post-disaster.

A packet containing a cover letter, survey, and pen is given to each crisis counselor together with a stamped return envelope, addressed to an external evaluator (presently NCPTSD).


Provider Survey (2) The survey also measures

worker stress (p. 2). Respondent’s identity is

protected by lack of identifying information, return of the survey to an external evaluator, and aggregation of results.

Training considerations: Conveying reasons for, and importance of, the survey; explaining why high response rate matters.


Resources

For further information about:• Tools• Databases• Evaluation manual

Contact the

SAMHSA’s Disaster Technical Assistance Center (DTAC). (http://mentalhealth.samhsa.gov/dtac/)

• Email -- http://nmhicstore.samhsa.gov/emails/contact.aspx

• Phone – 1-800-308-3515


Ex Ante Evaluation

Lessons from the past Problem analysis and needs assessment Objective setting Alternative delivery mechanisms and risk assessment Added value of this activity Planning future monitoring and evaluation

• What types of evaluations are needed and when should they be carried out?

• Are the proposed methods of collecting, storing and processing the follow-up data sound?

• Is the monitoring system fully operational already from the outset of the program implementation?

Helping to achieve cost-effectiveness


Ex Ante Evaluation

Lessons from the past Problem analysis and needs assessment Objective setting Alternative delivery mechanisms and risk assessment Added value of this activity Planning future monitoring and evaluation Helping to achieve cost-effectiveness


Where are experts and resources

Let your exploration identify the experts and resources that are available

For our hypothetical example we have the following leads:• Fran Norris - [email protected]• Robert Macy – [email protected]• SAMHSA’s Disaster Technical Assistance Center

(DTAC). (http://mentalhealth.samhsa.gov/dtac/)• Email -- http://nmhicstore.samhsa.gov/emails/contact.aspx• Phone – 1-800-308-3515


Blank


Barriers and Challenges to Conducting Program Evaluation

Conducting program evaluation in the aftermath of disasters poses special challenges


Crisis And Chaos

In the immediate aftermath of disasters, decisions need to be made quickly on the basis on limited information. The prejudice is towards action, not deliberation.

During the crisis, there may be little interest in collecting systematic information on how the program is working. This shortcoming makes it difficult to monitor program progress and provides few data with which to later evaluate program achievements

In this context, evaluation may be viewed as arbitrary and burdensome, imposed by outsiders without a stake in serving survivors


Evolving, Adapting Services

The nature of the services may evolve over time as the needs of survivors change

Program models often must be adapted to the community, and providers in the field have a sense of “learning as they go”

Evaluations cannot assume that services are being delivered based on a pre-ordained model. It is essential to continually document program services and delivery strategies in order to be able to evaluate what the program is actually doing at different points in time


Evolving Community Context

Evaluation results are influenced by the community context, which also evolves over time.

• For example, client satisfaction results may be higher during early phases of recovery than during later stages, when disillusionment sets in.

Outreach programs may discover problems that existed prior to the disaster.

• Evaluations must be careful to differentiate new mental health problems from pre-existing problems


Factors That Boost Capacity For Program Evaluation

How do we establish an evaluation infrastructure that will allow us to maximize learning in future disasters.


Advance Political Support

Building evaluation capacity in disaster mental health requires creating an evaluation planning component in State Emergency Disaster Preparedness programs

Embedded in this ethos would be a respect for quality management informed by empirical feedback, and the expectation of accountability

A dialogue among key stakeholders involved in post-disaster recovery -- at federal, state, and local levels -- to set evaluation policy is needed to ensure that the evaluation mandate is feasible, relevant to real-world concerns, and not unduly burdensome


Outcomes – The Societal Benefits While outputs assess “how much” was done,

outcomes focus on “how much good” was done. They are the least well-specified arena in disaster mental health

Outcomes differ over time:

• Immediate outcomes can be observed directly after completing an activity

• Intermediate outcomes derive from immediate outcomes such as alleviation of psychiatric symptoms, reduced substance use, or improved role functioning

• Long-term outcomes are program benefits such as community cohesion or disaster preparedness