GETTING TO OUTCOMES · iv Preface Getting To Outcomes® (GTO) is a user-friendly process for comprehensive planning, implementation guidance, and evaluation of programs and community

GETTING TO OUTCOMES® GUIDE FOR

COMMUNITY EMERGENCY PREPAREDNESS

GETTING TO OUTCOMES®

Patricia A. Ebener

Sarah B. Hunter

Rachel M. Adams

David Eisenman

Joie D. Acosta

Matthew Chinman

C O R P O R A T I O N

https://www.rand.org/pubs/tools/TL259.html

https://www.rand.org/

Limited Print and Electronic Distribution Rights

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited. Permission is given to duplicate this document for personal use only, as long as it is unaltered and complete. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial use. For information on reprint and linking permissions, please visit www.rand.org/pubs/permissions.html.

The RAND Corporation is a research organization that develops solutions to public policy challenges to help make communities throughout the world safer and more secure, healthier and more prosperous. RAND is nonprofit, nonpartisan, and committed to the public interest.

RAND’s publications do not necessarily reflect the opinions of its research clients and sponsors.

Support RAND

Make a tax-deductible charitable contribution at www.rand.org/giving/contribute

www.rand.org

For more information on this publication, visit www.rand.org/t/TL259

Published by the RAND Corporation, Santa Monica, Calif.

© Copyright 2017 RAND Corporation

R® is a registered trademark.

Getting To Outcomes® and GTO® are jointly owned by the RAND Corporation and the University of South Carolina.

http://www.rand.org/t/TL259

http://www.rand.org/pubs/permissions.html

http://www.rand.org/giving/contribute

http://www.rand.org

iii

Preface..............................................................................................................ivAcknowledgments........................................................................................................vAbbreviations....................................................................................................viGlossary...........................................................................................................vii

INTRODUCTION

BriefBackgroundonGettingToOutcomes,CommunityEmergencyPreparedness,andOverviewoftheGuide.......................................................x

CHAPTER ONE

GTOStep1—NeedsandResourcesAssessment...............................................1

CHAPTER TWO

GTOStep2—GoalsandDesiredOutcomes.....................................................17

CHAPTER THREE

GTOStep3—BestPractices.............................................................................27

CHAPTER FOUR

GTOStep4—Fit.....................................................................................................36

CHAPTER FIVE

GTOStep5—Capacities...................................................................................45

CHAPTER SIX

GTOStep6—Plan............................................................................................57

CHAPTER SEVEN

GTOStep7—ProcessEvaluation........................................................................74

CHAPTER EIGHT

GTOStep8—OutcomeEvaluation.....................................................................89

CHAPTER NINE

GTOStep9—ContinuousQualityImprovement(CQI)..................................104

CHAPTER TEN

GTOStep10—Sustainability....................................................................................115

CHAPTER ELEVEN

Summary........................................................................................................127

APPENDIX

ROAD-MAPProcessEvaluationMaterial................................................................129

Bibliography...................................................................................................133

iv

Preface

Getting To Outcomes® (GTO) is a user-friendly process for comprehensive planning, implementation guidance, and evaluation of programs and community initiatives. It is designed to help organizations run programs well and get desired outcomes. This guide is designed to lead the user through the ten steps of GTO and provide supplemental information and resources to improve planning, implementation, evaluation, improvement, and sustainability of a community emergency preparedness (CEP) program. This guide is a streamlined sequence of overviews, tools, and additional resources for each GTO step. Other GTO manuals for drug prevention, underage drinking prevention, positive youth development, homelessness services, home visiting, and teenage pregnancy prevention are available at http://www.rand.org/gto.

This guide is designed for individuals who are considering and/or planning for the implementation (and evaluation) of a CEP program. Emergency planners, personnel from community-based organizations, state and local health department staff responsible for CEP programs, and individuals assessing or evaluating CEP programs may also be interested in this guide.

This publication was supported under a cooperative agreement: CDC’s Collaboration with Academia to Strengthen Public Health Workforce Capacity, grant number 3 U36 OE000002-04 S05, funded by CDC Office of Public Health Preparedness and Response through the Association of Schools and Programs of Public Health (ASPPH). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of CDC, the U.S. Department of Health and Human Services, or ASPPH.

The UCLA Center for Public Health and Disasters promotes efforts to improve community resilience and to reduce the health impacts of disasters and global climate change. For more information on the center, see http://www.cphd.ucla.edu.

RAND Health is a division of the RAND Corporation, a research organization that develops solutions to public policy challenges to help make communities throughout the world safer and more secure, healthier and more prosperous. RAND is nonprofit, nonpartisan, and committed to the public interest. For more information on RAND Health, see http://www.rand.org/health or contact the director at [email protected].

http://www.rand.org/gto

http://www.cphd.ucla.edu

http://www.rand.org/health

mailto:[email protected]

v

Acknowledgments

The authors wish to express our thanks to Mary Leinhos, Ph.D., M.S., our project officer from the Office of Public Health Preparedness and Response, Centers for Disease Control and Prevention (CDC), for her support of this project. We appreciate the participation of the staff from the Public Health Emergency Department at the City of Long Beach Department of Health and Human Services, the Health Disaster Management Department at the Orange County Health Care Agency, and the Preparedness and Response Program at the San Bernardino County Department of Public Health. We would also like to acknowledge Plymouth Village, an older-adult residential community, in Redlands, California. We appreciate the input of our RAND and University of California, Los Angeles colleagues who provided insights and experiences for this work—in particular, Michael Prelip, Alina Dorian, Cathy Lang, Deborah Glik, and Andrea Martel. We thank Jeffrey Chappelle for administrative support. We would also like to thank Melinda Moore and Paul Koegel for their reviews and comments on the manual.

vi

Abbreviations

ASPPH Association of Schools and Programs of Public Health

BRFSS Behavioral Risk Factor Surveillance System

CDC Centers for Disease Control and Prevention

CEP community emergency preparedness

CHIS California Health Interview Survey

CPHD UCLA Center for Public Health and Disasters

CQI continuous quality improvement

EBPH evidence-based public health

FEMA Federal Emergency Management Agency

GTO Getting To Outcomes

HHS U.S. Department of Health and Human Services

LHD local health department

MOU memorandum of understanding

N/A not applicable

PHRETS Public Health Response to Emergent Threats Survey

ROAD-MAP Resilient Older Adults in Disasters—Mentored and Prepared

SMART specific, measurable, achievable, realistic, and time-based

UCLA University of California, Los Angeles

vii

Glossary

Adaptation is the process of changing a program to make it more suitable to a particular population or to an organization’s mission without compromising or deleting the parts of the program that make it effective (often called core components) (Step 2—Goals and Desired Outcomes and Step 4—Fit).

Capacities are the resources (staff, skills, facilities, finances, and others) that an organization needs to implement and sustain a program. Fiscal and resource capacities encompass adequate funding and other basics needed to implement a program as planned (e.g., transportation, food, printed materials, and evaluation resources); technical capacities include the expertise needed to address all aspects of program planning, implementation, and evaluation; access to special materials needed to deliver the program; and the technology appropriate to the implementation of the program, such as computers (Step 5—Capacities).

Continuous quality improvement (CQI) is a systematic assessment using information from evaluation to inform future program planning, implementation, and outcomes (Step 9—Continuous Quality Improvement).

Culture can be thought of as a population’s values, norms, beliefs, religion, customs, rituals, language, and ethnicity/race (Step 4—Fit).

Desired outcomes are specific changes in behaviors and determinants, such as knowledge, that are expected as a result of a program. Well-written desired outcomes are specific, measurable, achievable, realistic, and time-based (SMART). Desired outcomes are also sometimes called objectives (Step 2—Goals and Desired Outcomes). If the behavior or determinant is measured to be at a desirable point before the start of the program, then a desired outcome could be to maintain that level.

Dosage is how much of the program a participant receives. Depending on the program, dosage can be the amount of time, number of sessions, or number of activities in which an individual participates (Step 6—Plan and Step 7—Process Evaluation).

Evidence-based public health (EBPH) approaches for community emergency preparedness (CEP) are interventions, like the Resilient Older Adults in Disasters—Mentored and Prepared (ROAD-MAP) program, that use the best available research evidence, practitioner expertise, and other available resources. EBPH approaches

viii

include consideration of the characteristics, needs, and values of the target community (Step 3—Best Practices).

Fidelity is the faithfulness with which a program is implemented. This includes implementing a program without removing parts essential to the program’s effectiveness (i.e., core components). This is also sometimes called compliance, integrity, or adherence (Step 3—Best Practices, Step 4—Fit, Step 5—Capacities, and Step 7—Process Evaluation).

Fit is the overall compatibility between a program and the target population, the organization, and the target community (Step 4—Fit).

Goal is the overarching big picture of what an organization wants to achieve regarding improving the lives of its target population. Goals reflect the anticipated impact in the future (Step 2—Goals and Desired Outcomes).

Logic models are diagrams that link a goal, behaviors directly related to it, factors that influence those behaviors, intervention activities designed to change those factors, and inputs needed to conduct intervention activities (Step 2—Goals and Desired Outcomes).

Measures are tools used to collect data in a program evaluation. They may consist of individual questions or scales on a survey designed to obtain information about the behavior and related factors being examined (Step 8—Outcome Evaluation).

Needs and resources assessment is a systematic way to identify current conditions underlying the potential need for a program or intervention and to identify related community resources (Step 1—Needs and Resources Assessment).

Outcome evaluation is an activity to examine whether a program caused an improvement among its participants on specific areas of interest (e.g., an increase in the number of participants reporting storing an emergency supply kit) and by how much (Step 8—Outcome Evaluation and Step 9—Continuous Quality Improvement [CQI]).

Partnership and collaboration capacities refer to connections with other community partners who can help implement and support the program (Step 5—Capacities).

ix

Priority population is the group, often referred to as a target population, determined to be in most need of a CEP program (Step 1—Needs and Resources Assessment, Step 2—Goals and Desired Outcomes, Step 3—Best Practices, and Step 4—Fit).

Process evaluation assesses the degree to which a program is implemented as planned. It includes monitoring the program activities, who participated, and how often, as well as the strengths and weaknesses (quality) of the implementation (Step 6—Plan, Step 7—Process Evaluation, and Step 9—Continuous Quality Improvement [CQI]).

Readiness refers to the degree to which a community or agency is able to take action on an issue—i.e., put programming into place. Readiness can range from none at all (e.g., the group has not even started thinking about addressing an issue) to already having successful programs in place (Step 4—Fit).

Scale is a grouping of individual survey questions that address a single topic. Individual questions are often averaged together and interpreted as a group (Step 8—Outcome Evaluation).

Staff and volunteer capacities refer to staff and/or volunteers with appropriate credentials, training, experience, and commitment to the program (Step 5—Capacities).

Stakeholders are the individuals and organizations that have an interest in a program’s delivery and results. Stakeholders may include participants, their families, program staff and volunteers, funders, board members, and community organizations (Step 4—Fit).

Sustainability refers to the capacity to continue a program after initial funding has ended (Step 10—Sustainability).

Tool refers to the various worksheets and templates associated with each GTO step that prompt practitioners to make and record decisions (Steps 1–10).

Work plan is the organized, formal documentation of components and tasks necessary to implement a program, broken down by resources, personnel, delivery dates, and accomplishments; the work plan specifies who will do what, when, where, and how (Step 6—Plan).

x

Introduction

Getting To Outcomes® (GTO) is a user-friendly process for comprehensive planning, implementation guidance, and evaluation of programs and community initiatives. It is designed to help organizations run programs well and get desired outcomes, just as the guide’s name suggests. GTO is a ten-step process that guides the user through the key tasks needed to make a program a success. The GTO process is supported by training, technical assistance, and guides in several content areas, which offer tools and instructions to help users complete the ten GTO steps.

Getting To Outcomes at a Glance

xi

Purpose of the Guide

This guide is designed to lead the user through the ten steps of GTO and provide supplemental information and resources to improve planning, implementation, evaluation, improvement, and sustainability of a community emergency preparedness (CEP) program. This guide is a streamlined sequence of overviews, tools, and additional resources for each GTO step. Other GTO manuals for drug prevention, underage drinking prevention, positive youth development, homelessness services, home visiting, and teenage pregnancy prevention are available at http://www.rand.org/gto.

This guide is designed for individuals who are considering and/or planning for the implementation (and evaluation) of a CEP program. Emergency planners, personnel from community-based organizations, state and local health department staff responsible for CEP programs, and individuals assessing or evaluating CEP programs may also be interested in this guide.

Why Use GTO?

Research has shown that organizations that use GTO improve their programs and get better outcomes than organizations that do not use GTO (Chinman, Acosta, et al., 2016; Acosta et al., 2013; Chinman, Hunter, et al., 2008).

Why Use GTO to Improve CEP?

The number and severity of disasters is increasing. When a disaster hits, affected populations often seek help through emergency response systems, public health systems, and acute care hospitals. This surge of people seeking help can disrupt services and overwhelm systems. Having a seven-day supply of food, water, and medication and a written household evacuation plan—simple ways to improve household preparedness—can help prepare the population to cope with service disruption, thereby lessening the burden on systems and hospitals.

Traditional CEP programs have focused on (1) increasing awareness about the importance of being prepared and (2) educating community members about simple ways that they can make themselves and their neighbors better prepared for disaster. CEP programs are often focused on

http://www.rand.org/gto

xii

- educating or training households to create a family or personal preparedness plan (how to evacuate, where to meet up, how to cover medication needs) and a preparedness kit (e.g., food, water, batteries, radio, flashlight)

− improving neighbor-to-neighbor outreach to talk about preparedness and identify how neighbors may be able to help each other during a disaster

− educating or training individuals about how to maintain psychological health during a disaster (e.g., a psychological first aid program, including a stuffed animal or bar of chocolate in a household preparedness kit for comfort)

− coordinating volunteer efforts (such as Citizen Corps, Medical Reserve Corps, or a local neighborhood group that shovels during snow storms)

− organizational networking and preparedness planning (to build capability among community- and faith-based organizations that do not traditionally get involved in preparedness activities).

Preparedness efforts can also be tailored to specific populations that have certain needs that make them especially vulnerable to disasters. For example, older adults tend to have more health and functional needs than others, and preparedness efforts may need to take specific steps for that population.

The intended outcomes of CEP programs are twofold: 1. Improve individuals’ and households’

− awareness that a disaster can happen in their community − awareness of skills that they can use during a disaster − development of a personal or household preparedness plan or kit − awareness of the importance of and inclusion of psychological health items

in plans and kits − awareness of a neighbor or organization that they can count on during a

disaster − willingness to volunteer for coordinated preparedness efforts in their

community. 2. Improve community organizations’

− awareness that a disaster can happen in their community − awareness of emergency plans in the community (the role that their

organization plays in a coordinated response and where the organization can get help, if needed)

− use of preparedness planning

xiii

− willingness to participate in disaster planning efforts or community disaster exercises or drills.

GTO can help your organization assess the need for and the fit of a CEP program, identify a specific goal, and specify desired outcomes that relate to the goal. After that, GTO can help you choose an evidence-based public health (EBPH) approach and a plan for successful implementation. GTO will also help you plan and collect process and outcome evaluation data to improve and sustain the program. Conducting these activities will help you meet the needs of your community and demonstrate your effectiveness to funders, community members, and other stakeholders.

In this guide, we define community preparedness as the ability of communities (organizations and households in the community) to prepare for, withstand, and recover from manmade or natural disasters in both the short and long terms. To prepare communities for disaster, public health agencies, in partnership with emergency management, health care organizations (private and community-based), mental/behavioral health providers, and community and faith-based partners can conduct CEP programs.

Organization of the GTO CEP Guide

This guide contains 11 chapters—one for each GTO step, plus a summary. Each chapter contains

ü an overview of the GTO step—what it is, why it is important, and how to do it

ü an introduction to the tools contained within each step that explains why using the tools is important

ü detailed instructions for completing each tool within the step ü a completed example of each tool

The examples throughout this guide relate to a fictional location. The place is Townville, USA, located along the banks of a river that experiences frequent seasonal flooding. The town has a large population of older adults living throughout the community. In recent years, the floods have left many neighborhoods cut off for several days. Lack of access to shops and pharmacies has disproportionately affected older adults, many of whom require daily medications for chronic medical conditions. Hospitalizations and emergency medical assistance have resulted due to lack of compliance with medication regimes while older adults were cut off by the flooding. In addition, several severely dehydrated older adults were seen in emergency rooms during the disaster events. A public health team from the local health department (LHD) was working to try to understand more about the problems and figure out

xiv

whether there was some preparedness work that could be done to better educate and prepare this population for disasters that might leave them cut off from medical and retail services. The team includes the LHD staff (a program manager, a program coordinator, and a trainer) and managers of several older adult community centers (also referred to as senior centers) with which the LHD has collaborated in the past on older adult wellness interventions. The examples in this guide follow the LHD team’s work of using GTO to consider and then select, plan, and evaluate Resilient Older Adults in Disasters—Mentored and Prepared (ROAD-MAP), a real CEP program developed by the University of California, Los Angeles (UCLA) Center for Public Health and Disasters (CPHD). ROAD-MAP was selected for implementation in Townville after assessment of other options concluded that they were not appropriate. Only the ROAD-MAP assessment tools are included here as examples. ROAD-MAP is a CEP program that educates older adults in how to prepare emergency supplies, including an extra seven-day supply of medications and household supplies, and a disaster plan. The program is delivered as an in-person workshop with trained peer facilitators leading the 75-minute session. Resources are distributed to workshop participants to help them obtain extra medication and emergency household supplies after the workshop. For more information about ROAD-MAP, including the ROAD-MAP resource package, visit http://www.cphd.ucla.edu/tools-and-resources.

ü blank tools for you to complete in each chapter (Microsoft Word versions of each tool and the complete guide can also be downloaded without charge from the RAND website.)

ü tips and resources for use in completing each step

ü a summary checklist for doing each step and a summary of next steps.

The guide also includes an appendix containing ROAD-MAP process evaluation materials, a list of abbreviations, and a Glossary section at the front to help you better understand terms used throughout the chapters.

Tips for Using the GTO CEP Guide

• Completing the tools for each step takes time, thought, and consideration. Whentime and attention are put into completing these tools, they will help improve thequality of your program.

• The ten GTO steps and corresponding tools outlined in this guide are designedto be completed sequentially. The GTO process is most beneficial when program

http://www.cphd.ucla.edu/tools-and-resources

xv

facilitators and program managers/directors collaborate on their completion to ensure that key details are not missed.

• It is especially important to complete GTO Steps 1–6 before implementing yourprogram. GTO Steps 7 and 8—the evaluation steps—also have parts that areimportant to complete before you start running your program. Set aside a regulartime for your program facilitators and program manager to address the stepsleading up to, during, and immediately following implementation.

• If you plan to deliver your program more than once, you may want to use yourevaluation results in Step 9—Continuous Quality Improvement (CQI)—and revisitGTO Steps 1–6 before any subsequent implementation to see whether changesshould be made to improve the process and/or outcomes of the nextimplementation. You would then continue the evaluation and quality improvementsteps (Steps 7–9) with each implementation of the program. GTO prescribes thisongoing cycle for all future program implementations. Step 10 is also important tocomplete whenever ongoing implementation is planned.

GTO Step 1—Needs and Resources Assessment 1

Chapter One

GTO Step 1—Needs and Resources Assessment

What is GTO Step 1?

To create a fully informed plan for an effective program, it is essential to know what the problems are that need to be addressed (e.g., where the gaps in community preparedness are, who is most at risk, where there are limited resources to improve preparedness) and the resources that are available. A needs assessment is the process of gathering information about the current conditions of a targeted area, or the underlying need for a program. A resources assessment is the process of gathering information about the resources available to address a particular need or risk.

Why is GTO Step 1 important?

A needs and resources assessment of your community and target population can help you identify the most prevalent community risks, hazards and vulnerabilities, gaps, conditions that call for change, and what community resources are available to assist your efforts. Knowing current needs will help with setting realistic goals and desired outcomes.

How do I carry out GTO Step 1?

Complete this step by (1) gathering information on the problems or needs in your community, (2) identifying existing resources that address these needs, (3) prioritizing needs to select those you can address, and (4) specifying your target population.

GTO Step 1 contains three tools that will help you carry out a needs and resources assessment:

ü The Data Catalog Tool helps you determine which data source you plan to use and who will be responsible for collecting the data. If your public health department has conducted a hazards and vulnerabilities analysis, this may serve

WHAT DOES GTO STEP 1 DO?

This step helps you identify and document the need for a program and related existing community resources.

Step 1 Needs


as an important data source because it contains information about community risks and vulnerable populations. Local public health survey data (such as data collected for local monitoring of household preparedness, data on the locations of vulnerable populations, or data reported to the Centers for Disease Control and Prevention (CDC) for state or national emergency preparedness monitoring) may also be useful to understand community risks and vulnerable populations.

ü The Community Resource Assessment Tool helps you identify resources in your community that may support implementation of your program. You will save time and money and avoid duplicating efforts when you determine what is already working and where you can best contribute. Citizen Corps (if there is a program in your area) may have already identified local resources. In addition, in planning for how to improve preparedness capabilities, public health departments are required to assess their current state, which includes identifying resource elements (within their department). Planning, skills and training, and equipment and technology are the resources that CDC and subject-matter experts have determined are the most critical for being able to build and maintain the associated capabilities. You can draw upon your most recent current state assessment data for this tool.

ü The Triaging Among Needs Tool helps you identify the gaps in CEP that your program should be designed to target. It helps you identify priority needs that your organization has the capability to address.


Tip 1-1. Links to existing data sources to help you identify needs in your community

1. The Assistant Secretary for Preparedness and Response Technical Resources,Assistance Center, and Information Exchange (ASPR TRACIE), part of the U.S.Department of Health and Human Services (HHS), provides technical resources onhazard and vulnerability assessment: https://asprtracie.hhs.gov/technical-resources/3/Hazard-Vulnerability-Risk-Assessment/0

2. The HHS emPOWER Map is an interactive tool that highlights at-risk individuals usingelectricity-dependent life-maintaining and assistive durable medical equipment in theUnited States, sortable by natural hazards, state, county, and ZIP code:https://empowermap.hhs.gov/

3. The CDC Behavioral Risk Factor Surveillance System (BRFSS) provides state data fromtelephone surveys on health-related risk behaviors, chronic health conditions, and use ofpreventive services: https://www.cdc.gov/brfss/

4. The California Health Interview Survey (CHIS) is from the UCLA Center for Health PolicyResearch: http://healthpolicy.ucla.edu/CHIS/Pages/default.aspx

5. City-Data creates profiles for every city in the United States down to the neighborhoodlevel: http://www.city-data.com/

6. Healthy City provides data on California cities: http://www.healthycity.org/

7. American FactFinder is a U.S. Census Bureau source for data on communities:https://factfinder.census.gov/faces/nav/jsf/pages/index.xhtml

8. The Federal Emergency Management Agency (FEMA) National Household Survey,2015, is available at https://www.fema.gov/media-library-data/1467116407881-096f834e638f02fa8499941ce5c155bb/national_household_survey_2015_508_062716_508.pdf

9. The Vulnerable & At-Risk Populations Resource Guide assists state and local publichealth agencies, emergency management, hospitals, and other agencies with improvedlocal mapping options (custom mapping available): http://www.varpguide.com/

10. The CDC created Public Health Preparedness Capabilities: National Standards for Stateand Local Planning: https://www.cdc.gov/phpr/readiness/capabilities.htm

11. This toolkit from Ready.gov contains detailed community preparedness information:https://www.ready.gov/community-preparedness-toolkit

12. HealthData.gov provides many national- and state-level health statistics:https://www.healthdata.gov/

https://asprtracie.hhs.gov/technical-resources/3/Hazard-Vulnerability-Risk-Assessment/0

https://empowermap.hhs.gov/

https://www.cdc.gov/brfss/

http://healthpolicy.ucla.edu/CHIS/Pages/default.aspx

http://www.city-data.com/

http://www.healthycity.org/

https://factfinder.census.gov/faces/nav/jsf/pages/index.xhtml

https://www.fema.gov/media-library-data/1467116407881-096f834e638f02fa8499941ce5c155bb/national_household_survey_2015_508_062716_508.pdf

http://www.varpguide.com/

https://www.cdc.gov/phpr/readiness/capabilities.htm

https://www.ready.gov/community-preparedness-toolkit

https://www.healthdata.gov/




Tip 1-2. Collecting your own needs and resources assessment data

Most of the information you collect will likely come from existing data (e.g., the CDC, state and local public health departments). You may need to do some additional data collection on your own, however, to have a fuller picture of the needs in your community. This section provides a brief overview of the types of data that you might consider collecting. You might be able to implement certain data collection efforts on your own, but if you have funding it may be best to hire or consult with an experienced research organization, university, or outside technical assistance provider who can ensure that these efforts are done well and get you the information that you need. Some different types of data that might be useful to collect are discussed below.

Qualitative data: interviews, meetings/forums, and focus groups

Qualitative data are typically collected by asking open-ended questions, which encourage respondents to answer using their own words. For example, you might ask, “What services are currently available in your community to assist older adults in emergencies?” Qualitative data collection methods include the following:

• Key informant interviews: Interviews are conducted with individuals who are importantleaders (e.g., mayor, police chief, local pastor) or representatives in their communities.They are knowledgeable about the community and are likely to be aware of manyissues. Interviews can also be conducted with target population members (e.g.,residents in senior communities or members of senior social clubs) to learn more aboutthe service itself and about what other services may be needed (e.g., training).

• Community meetings/forums: Various community individuals are invited to a series ofmeetings and are asked about their opinions and perceptions of the needs in theircommunity. Although key leaders are often present, the meetings are held to obtaininformation from the general public.

• Focus groups: Focus groups may be particularly useful if you need to get informationquickly or when you want an opinion from an established group. They are considered anideal format for getting at underlying attitudes, feelings, beliefs, and behaviors of agroup. Besides being more efficient than interviews, focus groups allow for discussionsthat would not occur in one-on-one interactions and are effective in getting participantsto identify false or extreme views. In a focus group format, 6–12 individuals convene andanswer a predetermined set of open-ended questions posed by a facilitator. You canrecruit a variety of people for each focus group. Or, it is sometimes preferable toconvene participants for each group who have similar characteristics, such as parents,teachers, or law enforcement officials.


Showing that you care about what community members and other stakeholders think about these issues can build public good will and community buy-in, which may prove valuable later when you seek financial, political, and collaborative support.

Quantitative data: surveys

Quantitative means that the data include numbers. Surveys are a common way to systematically collect quantitative information from a number of individuals. Collecting your own survey data can be a demanding task, so you should check first to see whether you can use existing data before you embark on designing and implementing a survey.

Surveys often ask a standardized set of questions or offer statements (for example, “Families have a wide range of supports in this community to successfully take care of older adults”) to which respondents typically choose a response from a number of choices (for example, “strongly agree, agree, disagree, or strongly disagree”). Commonly, surveys are addressed to the following groups:

• Service providers: Service and treatment providers possess knowledge about the natureof problems in a community, what programs and resources are available, and who isand who is not being served.

• Clients or participants: Clients and program participants are excellent sources ofinformation on what needs are being met and what more should be done.

• A targeted population: Self-report surveys completed by those targeted by the initiative(e.g., public health workers) provide useful information on their attitudes and beliefs.Because assessing problem behaviors in a survey can be a sensitive topic, it is best touse questions that have already been written and tested. If you plan to administersurveys among your target population, it is best to consult with a person or organizationexperienced in administering surveys.

Online resources can help you identify high-quality survey questions that have already been tested. See the resources in Tip 1-1.


Instructions for completing the Data Catalog Tool

1. Begin by listing all the community problems or needs you intend to tackle andneed more information about.

2. For each problem or need listed, specify the source from which you will obtainthe data.

3. Note whether the data exist or you have to collect new data yourself—forexample, to determine the percentage of the older adult population that has aseven-day supply of prescription medication. The point is to gather sufficient datato defend the need for a program, inform goal-setting, and point you toward aspecific program to use. In the example, the Townville LHD team relied onexisting data collected in a survey of older adults.

4. Specify the person responsible for collecting the data and a due date foracquiring the information.


Example Data Catalog Tool

Completed by: Project team/manager Date: January____

Problem/Need Data Areas Source of Data Existing or New Data

Person Responsible/ Date Due

1. Older adults experiencemore adverse health outcomes during disasters.

R. Li, Advancing Behavioral and Social Research of the Elderly in Disasters, Committee on Populations, National Academies: Washington, D.C., 2009.

Existing Project manager/month 1

2. Older adults are morevulnerable during disasters due to higher levels of disability, chronic disease, and cognitive impairments.

N. Aldrich and W. F. Benson, “Disaster Preparedness and the Chronic Disease Needs of Vulnerable Older Adults,” Preventing Chronic Disease, Vol. 5, No. 1, 2008: http://www.cdc.gov/pcd//issues/2008/jan/07_0135.htm


3. Few older adults in theU.S. possess emergency plans.

Only 23.6 percent of respondents reported having any specific plan, 10.1 percent reported members of their household being registered for disaster help should it be needed, and 43.2 percent knew of a specific shelter location in their community, in case they had to abandon their homes.

T. M. Al-Rousan, L. M. Rubenstein, and R. B. Wallace, “Preparedness for Natural Disasters Among Older US Adults: A Nationwide Survey,” American Journal of Public Health, Vol. 104, No. 3, 2014, pp. 506–511.

National Health and Retirement Survey


http://www.cdc.gov/pcd//issues/2008/jan/07_0135.htm




4. Nationally, more than one-third of older adults (age 50 and up) do not possess even a three-day emergency kit.

T. M. Al-Rousan, L. M. Rubenstein, and R. B. Wallace, “Preparedness for Natural Disasters Among Older US Adults: A Nationwide Survey,” American Journal of Public Health, Vol. 104, No. 3, 2014, pp. 506–511.

National Health and Retirement Survey


5. Low perception ofpreparedness among older adults in Townville.

Only 35 percent of households in which all adults are age 65 or older in Townville feel “mostly” or “completely” prepared.

Household survey fielded by the Townville Public Health Department in 2011

Survey is being fielded again and updated results will be available by July.

Existing

New

Project manager/ month 1

As soon as results are available

6. A high prevalence of olderadults who take prescription medications do not possess emergency stockpiles.

Although 87 percent of 65+ adult households in Townville indicated that they take prescription medications, 25 percent indicated that they do not have a 30-day supply.

Household survey fielded by the Townville Public Health Department in 2011

Existing Project manager/ month 1


Data Catalog Tool

Completed by: _____________________ Date:_____________



1.

2.

3.

4.

5.


Instructions for completing the Community Resource Assessment Tool

1. Make as many copies of the tool as necessary for you and your co-workers tocomplete this step. Make more than one copy when you are documenting morethan three resources.

2. Identify local community resources that do or could help address the needs orgaps that you identified in the Data Catalog Tool.

3. Note the location or address of the resource or where it is delivered.4. Specify the targets served by the community resource. Note that the answers to

these questions may require informal surveys or interviews with providers of theresources you identified.

5. Enter the resource availability, including frequency or hours of operation, ifapplicable.

6. Describe who uses the program or resource now.7. Collect any information you can find on the outcomes the resource is producing.8. If the resource is having positive outcomes, try to identify what characteristics are

driving those positive outcomes (e.g., clear and simple educational materials onpreparedness, outreach through neighbor-to-neighbor networks).


Example Community Resource Assessment Tool

Completed by: Project team/manager Date: January ________

Resource 1 Resource 2 Resource 3 Name of resource and program/service/ capabilities

ABC Senior Center

Location XYZ Street Townville, USA

Target served Adults age 50 and over

Hours of operation M–F, 8 a.m.–4:30 p.m.; Sat, 10 a.m.–4 p.m.

Who uses it? Typically older adults from XYZ neighborhood

Community risk factors addressed

Offers activities and services tailored to the interests and needs of adults

Capabilities and community assets being built

Offers activities and services tailored to the interests and needs of adults

Any outcomes produced by the resource?

No outcomes being tracked

What’s working? Anecdotally, location is accessible to older adults and offers social and educational activities in formats appealing to older adults (e.g., big print, low tech)


Community Resource Assessment Tool

Completed by: ________________________ Date:________________

Resource 1 Resource 2 Resource 3 Name of resource and program/service/ capabilities

Location

Target served

Hours of operation

Who uses it?

Community risk factors addressed

Capabilities and community assets being built

Any outcomes produced by the resource?

What’s working?


Instructions for completing the Triaging Among Needs Tool Now that you have identified all needs, this tool will help you select the community needs not being met elsewhere and that can be reasonably tackled by your team.

The needs and resources data will inform this process. Begin by brainstorming a list of important community risks, hazards and vulnerabilities, gaps, or conditions that call for change, and then reduce the list as described below. 1. Make as many copies of this tool as needed for you and your co-workers to complete this task. List additional gaps you are concerned with on extra copies of the tool. 2. Decide how you will involve staff and other stakeholders in the filtering process. For example:

a. Distribute copies to your co-workers and have them fill it out on their own. Collect the completed copies and have someone collate the answers for each item. You and your co-workers should then meet to discuss the results and decide on the key priorities of your program.

b. Bring the co-workers together for a meeting to discuss and develop the priorities.

3. The left column, “Process,” provides prompts on how to carry out the filtering. 4. Start by identifying the most-important gaps in CEP that need to be addressed, according to your analysis of the problem data.

a. Examine the resulting lists, and cross out any that already are being addressed in other programming offered by your organization or other organizations in your community (and note which organization).

b. Cross out any that you lack the resources (time and budget) to confront. c. Now, cross out any others that you can’t change or whose changes you

can’t measure. d. Strike through any needs on your list that your organization simply chooses

not to address at this time. e. Copy those remaining after this sorting process into the space at row 7.

These are your priority needs. 5. Specify the target population of your program. You should note about how many people you will target and what characteristics define the group.


Example Triaging Among Needs Tool

Completed by: Project team/manager Date: January____

Process Needs: Important gaps in CEP to be addressed

(1) (2) 1. List the gaps in CEP that need to be

addressed.Lack of 7-day supply of medications

Lack of 7-day household emergency supplies and information

2. List capabilities needed to address the gaps. Trainers andcurriculum to inform how to obtain surplus medication supply

Trainers and curriculum to inform what to obtain and how to store emergency household supplies

3. Is this gap currently being addressedelsewhere?

No No

4. Is this a gap that we lack the resources (timeand budget) to confront?

No No

5. Is this a gap that we cannot change or whosechange we cannot measure?

No No

6. Is this a gap that we choose not to address atthis time?

No No

7. Highlight the remaining gaps with associatedcapabilities. These are your priority needs.

Clear and feasible instructions for acquiring surplus medications

Knowledge about which supplies should be on hand Knowledge about proper storage for supplies

8. Specify your target population, including theircharacteristics and the approximate numberfor which you have resources.

Older adults living independently who take prescription medications

Older adults living independently


Triaging Among Needs Tool

Completed by: ___________________________ Date:_________________

Process Needs: Important gaps in CEP to be addressed

(1) (2)

1. List the gaps in CEP that need to beaddressed.

2. List capabilities needed to address the gaps.

3. Is this gap currently being addressedelsewhere?

4. Is this gap that we lack the resources (timeand budget) to confront?

5. Is this a gap that we cannot change or whosechange we cannot measure?

6. Is this a gap that we choose not to address atthis time?

7. Highlight the remaining gaps with associatedcapabilities. These are your priority needs.

8. Specify your target population, including theircharacteristics and the approximate numberfor which you have resources.


Before moving on to Step 2

Now you’ll move on to using the information you’ve gathered and the priorities you’ve identified to develop specific goals and desired outcomes. The priorities from Step 1 and the goals and desired outcomes you develop in Step 2 will form the basis for selecting the programs and strategies you could implement, as well the outcomes you eventually plan to measure.

Note: It is okay if at this step you have realized that implementing the program you had in mind is not your community’s first priority or would be redundant with other efforts already under way. You can continue to use this guide and the GTO 10-step approach to address any alternative needs that emerged as more pressing during your needs and resources assessment.

Checklist Completion of Step 1

When you finish working on this step, you should have: p Completed the three Step 1 tools p Reviewed available data sources p Conducted a needs and resources assessment p Reviewed the findings of the needs and resources assessments p Selected priority needs that emerged from your assessments

GTO Step 2—Goals and Desired Outcomes 17

Chapter Two

GTO Step 2—Goals and Desired Outcomes

What is GTO Step 2?

In Step 1, you selected the top priorities to address and identified your target population. In Step 2, you will specify both broad goals and specific changes, called desired outcomes, that you want to achieve for your target population. A goal is a broad statement that represents the overall impact you would like to achieve through your program. A desired outcome is a way to make goals more specific. For example, it can target certain attitudes, skills, or actual behaviors in your target population.

Another key part of GTO Step 2 is creating a logic model for your program. A logic model

• is a visual map of the route from needs and resources to goals and desiredoutcomes to program activities to actual outcomes

• is a flow chart of building blocks that allows you to clarify assumptions about howeach step builds on the preceding until the desired outcome is reached

• lets you easily see whether there are any gaps in the logic of your program.


It is important to set a goal and desired outcomes to ensure that • everyone involved “is on the same page” with what you are trying to accomplish• you have benchmarks so you know when your program is working as planned• you are collecting the right evaluation data to assess progress toward your goals

and desired outcomes.It is important to have a logic model for your program to

• show the relationships between needs, goals and desired outcomes, programactivities, and results (outcomes)

Step 2 Goals


This step prompts you to develop a goal, specific desired outcomes, relevant program activities to reach the goal, and a logic model that displays all these elements.


• help you tell a compelling story (“create a value case”) of how your program isaddressing a problem your community cares about

• visually represent a road map for GTO steps.


Start with the results of your needs and resources assessment from GTO Step 1. From there, write at least one goal that addresses the problems identified. For each goal, specify a related desired outcome, using the SMART Desired Outcomes Tool to help create desired outcome statements that specify what you want to change, who you want to change, how much change you expect, and when the change will occur. Then use the Logic Model Tool to link the problem or community risk and the resources or capabilities to the goals. Ask whether your goals and desired outcomes address the problems identified. Write down different program activities, or even specific programs, that would help achieve the desired outcomes. You will finalize the decision about the specific program in later GTO steps. If you have a program in mind, make sure that you understand the community risks and problems it targets and outcomes it has achieved.


Tip 2-1. Ensuring that your desired outcomes are SMART

Specific • The desired outcomes should identify concrete changes that will take place.• Answers the question, "Does the desired outcomes statement clearly specify what will

change?"

Measurable • The desired outcomes should specify the amount of change.• Answers the question, "Does the desired outcomes statement state how much change is

expected?"

Achievable • The desired outcomes should make logical sense, given the needs and goals identified

and what a particular program can achieve.• Answers the question, "Does the desired outcome statement make sense in terms of

what the program is attempting to do or is known to accomplish?"

Realistic • The desired outcomes should specify how the change will be met with available

resources and plans for implementation.• Answers the question, "Is the desired outcome realistic, given available resources and

experience?"

Time-Based • The desired outcomes should specify the time within which the desired outcome will be

achieved.• Answers the question, "Does the desired outcome statement specify when desired

results will be achieved?"


Instructions for completing the SMART Desired Outcomes Tool

SMART stands for specific, measurable, achievable, realistic, and time-based. SMART desired outcomes are statements that present the specific changes you would like to see as a result of the program in concrete terms that can be measured. The SMART acronym will help ensure that your desired outcome statements are strong.

1. Make as many copies of this tool as necessary for you and your co-workers tocomplete this task.

2. Enter the first goal and desired outcome in the first column.3. Create a full desired outcomes statement in the SMART Desired Outcome

Statement column, ensuring that each letter in the SMART acronym is addressedand relates to the desired outcome you listed.

4. Once your statement has been written, check it by using the SMART Checklist box.Use this space to write each piece of the desired outcome statement thatcorresponds to each letter in the SMART acronym. See GTO Steps 7 and 8 and theappendix for ideas on possible measures and on how you might gather data toassess your desired outcome.


Example SMART Desired Outcomes Tool

Completed by: Project team/coordinator Date: January_____

Goal 1 SMART Desired Outcome Statement

Target population of older adults living independently increases possession of 7-day supply of prescription medications To increase the number of

program participants indicating that they possess a 7-day supply of their medications by 15% from baseline to follow-up (3-month period).

SMART Checklist S The number of program participants with a 7-

day supply of their prescription medications M 15% increase in the number of people with

surplus prescription medications A Measured via pre-/post-test

R Based on UCLA-developed ROAD-MAP CEP program

T 3 months between baseline and follow-up


Target population of older adults living independently increases possession of 7-day emergency household water and other supplies To increase the number of

program participants indicating that they possess a 7-day supply of water and other emergency household supplies by 20% from baseline to follow-up (3-month period).

SMART Checklist S The number of program participants with a 7-

day supply of water and other household emergency supplies

M 20% increase in the number of people with household water and other emergency supplies

A Measured via pre-/post-test R Based on UCLA-developed ROAD-MAP CEP

program T 3 months between baseline and follow-up


SMART Desired Outcomes Tool

Completed by: _____________________ Date:_____________


SMART Checklist S

M

A R

T Goal 2 SMART Desired Outcome Statement

SMART Checklist S

M

A R

T Goal 3 SMART Desired Outcome Statement

SMART Checklist S

M

A R

T


Instructions for completing the Logic Model Tool

1. Make as many copies of this tool as necessary for you and your co-workers tocomplete this task. You may want to develop rough drafts as you work and thentransfer the final details to a clean copy. Plan to work on this tool as you progressthrough the steps, not all at once.

2. Have copies of your completed Step 1 tools available for everyone working on theLogic Model Tool.

3. Have information about any programs you are considering using, such asdescriptions of their goals, curricula, or procedures and existing reports onevaluations.

4. Moving down each column from left to right, first identify the needs and resources inyour community that are relevant to your program.

5. Next, write in the goals and SMART desired outcomes you hope to achieve from theprogram(s) you are considering.

6. In the next box, write in the EBPH approach and/or programs you have identified toaddress your goals and desired outcomes based on community needs andresources. If you have not yet chosen your program, you will have the opportunity tothink through that choice further in GTO Step 3. When you make your pick, you canreturn to this tool and link specific parts of the chosen program to specific goals anddesired outcomes.

7. In the next box, define program activities that will make the program happen. DoingGTO Steps 4–6 and finally selecting a program will help identify the items for this list.

8. In GTO Steps 7 and 8, you will determine how you will assess the quality of theprogram delivery and how you will assess the success of the program on achievingyour goals and desired outcomes. You can complete these sections of the LogicModel Tool at that time.

* This is a GTO logic model that has been in use for over a decade. While it is not a traditional logic model, it serves to highlight the key decisions madeacross the GTO steps as one moves through the flow of creating and implementing a program.


What are the needs in your community that you are trying to address?

1. Lack of preparation forthe medical needs of seniors during and after a disaster

2. Lack of householdpreparedness for seniors

3. Greater capacity forsenior centers to address needs of population

What resources exist in your community to address the need?

Collaboration between LHD staff and local senior centers

What are the goals and desired outcomes of your program?

1. Increase the number ofpeople with a 7-day supply of their prescription medication

To increase the number of program participants indicating that they possess a 7-day supply of their medications by 15% from baseline to follow-up

Considering ROAD-MAP, developed by UCLA, a training workshop for older adults that teaches skills for acquiring surplus supply of prescription medications and emergency household supplies. Program includes video, handouts, and other resources to take home.

What are your specific program activities?

How will you assess the quality of your efforts?

How will you assess the outcomes of your efforts?

Example Logic Model Tool*

Completed by: Project team/manager Date: January ____ Program: ____________________

2. Increase the number ofpeople with 7-day household supplies (water, nonperishable foods, flashlight, etc.) To increase the number of program participants indicating that they possess a 7-day supply of water and other emergency household supplies by 20% from baseline to follow-up

What EBPH approach or program are you using to achieve these goals and desired outcomes?

* This is a GTO logic model that has been in use for over a decade. While it is not a traditional logic model, it serves to highlight the key decisions madeacross the GTO steps as one moves through the flow of creating and implementing a program.








Logic Model Tool*

Completed by: ________________________ Date: _____________ Program: ____________________




Now you are ready to take the information from Steps 1 and 2 and use it to help you develop more information about the program or potential programs you are considering. The next three GTO steps (3 through 5) lead you through selecting the best evidence-based approaches to achieve your goals and desired outcomes (Step 3). You will assess possible program choices and select the program that you conclude is most suitable for your local area and target populations (Step 4) and that you have the organizational capacity (Step 5) to actually implement.


When you finish working on this step, you should have:

p Established program goals that explicitly link to the findings from Step 1 p Identified specific desired outcomes for each goal that are linked to your goals using

the SMART framework p Completed the Step 2 Desired Outcomes Tool p Begun to construct a logic model

GTO Step 3—Best Practices 27

Chapter Three

GTO Step 3—Best Practices

What is GTO Step 3?

GTO Step 3 prompts you to consider using an EBPH approach to building CEP. This means including (1) the best available research evidence; (2) practitioner expertise and other available resources; and (3) the characteristics, needs, and values of the target community to make a decision about the best approach to take among the possible options. Step 1 helped you to consider practitioner expertise and community resources and Step 4 will help you examine fit with the characteristics, needs, and values of the target community. In this step, you will learn how to examine the research evidence on outcomes of CEP programs and begin to assess which might be most appropriate for your goals and desired outcomes.


Funders often require the use of evidence-based approaches. Healthy People 2020 calls for the use of “policies and practices that are driven by the best available evidence and knowledge” (https://www.healthypeople.gov/2020/About-Healthy-People). The Core Competencies for Public Health Professionals (http://www.phf.org/resourcestools/Documents/Core_Competencies_for_Public_Health_Professionals_2014June.pdf), from the Council on Linkages Between Academia and Public Health Practice, support an evidence-based approach to public health.

The reason for this trend is that evidence-based approaches are known to achieve outcomes (when used effectively). Using an evidence-based approach

• increases the likelihood of achieving goals and desired outcomes• promotes confidence among funders and stakeholders that you are using the

best approach possible


This step guides you through the best available research evidence on building CEP to help you select an EBPH approach.

Step 3 Best

Practices

https://www.healthypeople.gov/2020/About-Healthy-People

http://www.phf.org/resourcestools/Documents/Core_Competencies_for_Public_Health_Professionals_2014June.pdf



• usually comes with many features that newly created, untested approaches donot have, such as tools to track outcomes and previous results about whatoutcomes to expect.


Start by gathering information about the best research evidence available on specific EBPH approaches or interventions. You do not need to gather and synthesize all available evidence yourself—it can be overwhelming. You can identify the best available evidence by using tools that synthesize, interpret, and evaluate the research literature. For example, the primary purpose of systematic reviews is to critically analyze and summarize evidence from evaluations of specific approaches or interventions. One place to find systematic reviews on emergency preparedness and related topics is the Community Guide’s website (https://www.thecommunityguide.org/topic/emergency-preparedness). The Task Force on Community Preventive Services has systematically reviewed more than 200 interventions to produce evidence-based recommendations on population-level interventions on a variety of topics (e.g., adolescent health, social environment, vaccines), including emergency preparedness.

Not all EBPH approaches and programs will be reflected in systematic reviews. You can also find evidence in scientific articles and online by: (1) searching for review articles in publications databases, such as PubMed or Google Scholar, using keywords (for example, literature review, systematic review, CEP). Here are some recent examples:

• J. Levac, D. Toal-Sullivan, and T. L. O’Sullivan, “Household EmergencyPreparedness: A Literature Review,” Journal of Community Health, Vol. 37, No.3, 2012, pp. 725–733. This paper presents a synthesis of available literature onhousehold preparedness published over the past 15 years (62 papers).

• Kohn, S., J. L. Eaton, S. Feroz, A. A. Bainbridge, J. Hoolachan, and D. J.Barnett, “Personal Disaster Preparedness: An Integrative Review of theLiterature,” Disaster Medicine and Public Health Preparedness, Vol. 6, No. 3,2012, pp. 217–231. This paper contains a review of 36 papers describing thecurrent state of evidence concerning personal disaster preparedness.

(2) searching for intervention evaluation articles in publications databases, such as PubMed or Google Scholar, using keywords (for example, evaluation, CEP, intervention, or program). Here are a few recent examples of articles targeting preparedness among youth, Latino immigrant communities, and adults who are disabled and living independently in the community:

• P. Powell, M. Smith, and L. Black, “Involving Youth in CEP: Impacts of aMultistate Initiative,” Journal of Youth Development, Vol. 4, No. 4, 2016, p. 19.

https://www.thecommunityguide.org/topic/emergency-preparedness



This paper describes the evaluation of the Alert, Evacuate, and Shelter program, which identified and trained youth/adult teams to use geospatial technology to map shelter locations and evacuation routes.

• K. R. Ronan, K. Crellin, and D. M. Johnston, “Community Readiness for a NewTsunami Warning System: Quasi-Experimental and Benchmarking Evaluation ofa School Education Component,” Natural Hazards, Vol. 61, No. 3, 2012, pp.1411–1425. This paper describes the evaluation of a brief in-school educationalprogram that aimed to make children more knowledgeable and prepared forhazards generally but also in more specific relation to the rollout of a newtsunami warning system.

• D. Eisenman, D. Glik, L. Gonzalez, R. Maranon, Q. Zhou, C. Tseng, and S. Asch,“Improving Latino Disaster Preparedness Using Social Networks,” AmericanJournal of Preventive Medicine, Vol. 37, No. 6, 2009, pp. 512–517. This paperdescribes a randomized control trial testing a disaster preparedness programfielded in low-income Latino communities with lay health promoters (seehttp://cphd.ph.ucla.edu/sites/default/files/downloads/Improving%20Latino%20Preparedness_AJPM2009_0.pdf andhttp://www.cphd.ucla.edu/sites/default/files/downloads/Project%20PREP.pdf).

• D. Eisenman, A. Bazzano, D. Koniak-Griffin, M. A. Lewis, C. Tseng, K. Lamb,and D. Lehrer, “Peer-Mentored Preparedness (PM-Prep): A New DisasterPreparedness Program for Adults with Developmental Disabilities,” Intellectualand Developmental Disabilities, Vol. 52, No. 1, 2014, pp. 49–59. This paperdescribes a wait-list control design evaluation of a disaster preparednessprogram developed for adults with developmental disabilities who are livingindependently in the community(http://www.cphd.ucla.edu/sites/default/files/downloads/Eisenman.CR%20%282%29.pdf).

(3) searching the “gray” literature (literature published outside of a commercial publisher) using Internet search tools to find government or business reports. Here are some examples:

• The U.S. Department of Health and Human Services has emergencypreparedness fact sheets, tools, videos, and reports on a wide range ofemergency preparedness topics (including mental health, communication, andcultural and linguistic competency) and populations (including children, olderadults, and individuals with access or functional needs) on its Public HealthEmergency website:https://www.phe.gov/Preparedness/planning/abc/Pages/resources.aspx#tools

http://cphd.ph.ucla.edu/sites/default/files/downloads/Improving%20Latino%20Preparedness_AJPM2009_0.pdf

http://www.cphd.ucla.edu/sites/default/files/downloads/Project%20PREP.pdf

http://www.cphd.ucla.edu/sites/default/files/downloads/Eisenman.CR%20%282%29.pdf

http://www.cphd.ucla.edu/sites/default/files/downloads/Eisenman.CR%20%282%29.pdf

https://www.phe.gov/Preparedness/planning/abc/Pages/resources.aspx#tools

http://cphd.ph.ucla.edu/sites/default/files/downloads/Improving%20Latino%20Preparedness_AJPM2009_0.pdf


• The CDC has state-developed tools and resources to reach out to communitymembers about emergency preparedness on its website:https://www.cdc.gov/ncbddd/disabilityandhealth/emergency-tools.html

Next, use the Evidence Synthesis Tool to help you weigh how each specific approach you have identified in your review aligns with the goal and desired outcomes that you created in the Logic Model Tool and the SMART Desired Outcomes Tool in GTO Step 2, as well as the best available research evidence. Complete this tool for as many approaches, programs, or interventions as you would like to consider. Using information in the completed tool, complete GTO Steps 4 and 5 to narrow your choices to one.

Instructions for completing the Evidence Synthesis Tool

Make plenty of copies of the tool. You will need a copy of Section 2 of the tool for each EBPH approach under consideration. We provide only one example here—for ROAD-MAP, the program ultimately chosen by the Townville LHD team after consideration of two other options. ROAD-MAP was selected for implementation in Townville after assessment of other options concluded that they were not appropriate. Townville also used this tool to assess the National Family Caregiver Support Program’s Just in Case: Emergency Readiness for Older Adults and Caregivers but found, by using the Evidence Synthesis Tool, that it did not employ teaching methods to actively involve participants and offered only fact sheets (i.e., it did not provide necessary activities and materials for staff and participants to use). Another EBPH approach that was considered, a media campaign approach to older adult education, proved too costly to implement after doing the capacity assessment in GTO Step 5. You also may be considering among options. The completed tools for each option will help you make comparisons and sometimes difficult choices among different options to narrow to a final decision. 1. In Section 1:

a. In the first column, record the types of sources you reviewed fordetermining the best available evidence.

b. In the second and third columns, list the specific approaches or programssuggested by each source and briefly summarize the evidence presented.

2. In Section 2:a. Respond to each characteristic (yes or no) as to whether the goal,

influential factors, and behaviors targeted in the selected approachmatch those you identified in GTO Steps 1 and 2.

b. Complete the column on the right for each characteristic. This will help youstart thinking about issues of fit, which is addressed in GTO Step 4.

https://www.cdc.gov/ncbddd/disabilityandhealth/emergency-tools.html


Example Evidence Synthesis Tool

Completed by: Project team/manager Date: January______ Program: ROAD-MAP____

Section 1: Summary of Best Available Evidence

What sources did you explore to find the best available evidence?

Source of evidence Did you review this source? Yes/No

What approaches did it suggest using?

What is the evidence for those approaches?

1. Systematic review No

2. Journal articles Yes UCLA Community Preparedness ROAD-MAP: This program, designed to improve disaster preparedness among adults living independently in the community, consists of four two-hour classes co-taught by a health educator and peer mentors.

The UCLA Community Preparedness ROAD-MAP program was developed using strategies from two other programs evaluated via randomized designs. The evaluations found that earthquake safety knowledge and preparedness supplies significantly increased from prior to the intervention to after the intervention (Eisenman, Glik, et al., 2009).

3. Gray literature No


Section 2: Assessment of Evidence-Based Approach

For each specific EBPH approach or program identified in Section 1, complete the following assessment of how its features relate to community needs, influential factors, behaviors, goals, and/or desired outcomes. Approach being considered: UCLA Community Preparedness ROAD-MAP Program

Evidence-based approach features and how they relate to our needs, behaviors, goals, and/or desired outcomes

Does the program have these features? Yes/No

What would we need to change to make the program fit our needs?

1. Focuses clearly on our identified goals Yes

2. Focuses on the specific behaviors we haveidentified that lead to the goals

Yes

3. Addresses the determinants we selected inGTO Step 2 (e.g., knowledge, attitudes)

Yes

4. Provides necessary activities and materialsfor staff to use

Yes Additional pre-/post-testing materials will be needed to evaluate outcomes.

5. Employs teaching methods to activelyinvolve participants

Yes

6. Employs activities, instructional methods,and behavioral messages appropriate to our target population

Yes May have to adjust content to better suit the culture and language of the local community


Evidence Synthesis Tool

Completed by: _____________________ Date:___________ Program: _____________

Section 1: Summary of Best Available Evidence

What sources did you explore to find the best available evidence?

Source of evidence Did you review this source? Yes/No

What approaches did it suggest using?

What is the evidence for those approaches?

1. Systematic review

2. Journal articles

3. Gray literature


Section 2: Assessment of Evidence-Based Approach

For each specific EBPH approach or program identified in Section 1, complete the following assessment of how its features relate to community needs, influential factors, behaviors, goals, and/or desired outcomes.

Approach being considered:___________

Evidence Synthesis Tool

Evidence-based program features and how they relate to our needs, behaviors, goals, and/or desired outcomes

Does the program have these features? Yes/No

What would we need to change to make the program fit our needs?

1. Focuses clearly on our identified goals

2. Focuses on the specific behaviors we haveidentified that lead to the goals

3. Addresses the determinants (influentialfactors) we selected in GTO Step 2 (e.g.,knowledge, attitudes)

4. Provides necessary activities and materialsfor staff to use

5. Employs teaching methods to activelyinvolve participants

6. Employs activities, instructional methods,and behavioral messages appropriate to ourtarget population



You’ve figured out which approaches or programs are in line with your community’s most urgent needs and the best available evidence. Now you are ready to move on to the next step in the GTO process—making sure your selected approach(es) or program(s) “fit” (Step 4) well with your target population, community, and your organization and that you have the capacity (Step 5) to carry it/them (if multiple options are still on the table) out well. If you already have a program selected, Steps 4 and 5 can help you fine-tune your work in a way that helps maximize your resources and increase your chances of success.


When you finish working on this step, you should have: p Completed the Step 3 tool p Developed an understanding of the best available evidence in your topic area p Reviewed the best available approaches to find those that will help you achieve your

goals and desired outcomes p Selected one or more approaches to consider further

GTO Step 4—Fit 36

Chapter Four

GTO Step 4—Fit

What is GTO Step 4?

Fit means that you have a good and close match between the selected or potential approach or program and your own:

• Target population and their needs: For example, ifyour target population is Spanish-speaking olderadults, is the program you are considering intendedfor older adults who speak Spanish?

• Community: For example, if the program you areconsidering emphasizes stockpiling supplies in ahigh-poverty community or uses a hurricane-basedscenario in a community whose greatest risk isearthquakes, will the community be supportiveenough?

• Organization: For example, does the program youare considering fit your organization’s values? Mission? Schedule?

GTO Step 4 helps you evaluate the extent to which each program you are considering is appropriate (i.e., a fit) for your target population, community, and your organization. In this step, you will consider such things as culture, values and practices, organizational mission, and existing programs within the community. The purpose of assessing fit is to either avoid programs that do not fit well or improve fit by making acceptable adaptations. This step can help you narrow down your choices from the possible options you identified in Step 3.

Once you have selected a program, a key feature of GTO Step 4 is making changes to improve its fit, often called adaptation. These changes need to be made very carefully. Change the program too much, and you may not get the outcomes that the program has achieved before.


This step provides a structure to determine whether the program you identified during GTO Step 3 is appropriate for your target, community, and organization.

Step 4 Fit

GTO Step 4—Fit 37


You want to use the best program you can offer. Programs that do not fit your target population, organization, or community well are likely to face implementation challenges and, therefore, are less likely to achieve the desired outcomes. Assessing fit before using a program is important for several reasons:

• It increases the chances that a program will be accepted by and will be good forthe target population.

• It helps avoid duplication of efforts (do not start a program that overlaps with anexisting one).

• It helps avoid finding out later that the program failed because it was a mismatch(a poor fit) with your target population, your community, and/or your organization.

• When there are fit problems that cannot be resolved, a program should not beused.

• The fit assessment helps when selecting from several candidate programs(choose the one with the best fit).


To assess the fit of the programs you are considering, use the GTO Step 4 tool: • The Program Fit Assessment Tool will prompt you to consider how the

candidate programs identified in Step 3 fit with your target population,organization, and stakeholders.

As you complete this tool, you will want to decide what adaptations, if any, to make to improve the fit of the candidate program or programs still under consideration.

GTO Step 4—Fit 38

Tip 4.1. Types of program adaptations

Red-light adaptations, such as reducing or eliminating major activities or topics, may greatly weaken the program and generally would not be advised. For example, programs often provide opportunities to practice new skills as a critical step in changing behavior. Reducing or eliminating practice components may make the program less effective.

are more complex and may alter Yellow-light adaptations program content, so you should proceed with greater caution. They often require expert assistance from the program developer or someone experienced with using the program—e.g., the LHD trainer in Townville—to avoid weakening the content of the program.

Green-light adaptations are considered safe, easy changes that can make a program better connect with the audience (i.e., to fit the program to the culture and context). These adaptations do not change the core topics addressed by the program. They are generally minor changes, such as adjusting the location to one familiar to the participants. Tailoring minor elements to better reflect the target population can improve most programs, and you should feel comfortable making such adjustments. In sum, green-light adaptations do not change what makes a program effective (i.e., the core components).

GTO Step 4—Fit 39

Tip 4-2. Examples of program adaptations

Red-Light Adaptations

Shortening a program (e.g., deleting an activity or whole session) Reducing or eliminating activities that allow participants to personalize risk material Reducing or eliminating opportunities for skill practice Eliminating a focus on certain topics Contradicting, competing with, or diluting program goals Replacing interactive activities with lectures or individual work

Yellow-Light Adaptations

Changing the order of sessions or sequence of activities Adding activities to reinforce learning Adding activities to address additional topics Replacing or supplementing videos (with other videos or activities) Using other models or tools that teach the same skill Implementing the program with a population (e.g., ethnic or cultural group) for which there is less evidence Replacing activities

Green-Light Adaptations

Updating and/or customizing statistics and other information Adjusting the location of the program to one familiar and convenient for participants Adding debriefing or processing questions Making activities more interactive, appealing to different learning styles Customizing written documents (e.g., the use of wording more reflective of program participants served)

GTO Step 4—Fit 40

Instructions for completing the Program Fit Assessment Tool

1. Make as many copies of the tool as necessary for you and your co-workers tocomplete this tool for each of the remaining programs you are considering.

2. Starting with row 1 (target population needs), work through the questions in the fittool, laying out the program considerations and requirements and answering yes orno in the appropriate columns. You may need to talk to several different people toget the answers (e.g., members of target group, colleagues, co-workers).

3. If no adaptations are needed, you can do the program as is. If adaptations areneeded, enter your ideas in the column labeled, “What adaptations can be made toincrease the fit?”

4. If adaptations are needed, figure out whether they are green-light, yellow-light, orred-light adaptations. Definitions of each type of adaptation are provided in Tip 4.1,and examples of each type are listed in Tip 4.2.

GTO Step 4—Fit 41

Example Program Fit Assessment Tool

Completed by: Project team/coordinator Date: January__ Program Being Considered: ROAD-MAP Fit with the target population’s . . .

Considerations Fits? Yes/No

What adaptations can be made to increase the fit?

1. Needs Program must improve medical preparedness and household preparedness specifically for older adults, many of whom have chronic health issues or cognitive impairments

Yes None needed

2. Demographiccharacteristics

Program must be accessible to: - Older adults - English, Spanish, and Korean speakers - Low and high socioeconomic status groups who attend the senior centers

Some work needed

- Curriculum should be adapted to address barriers of low socioeconomic status older adults - Curriculum is available in English and Spanish but will need to be adapted into Korean

3. Other:Cognitive/visual abilities

Use of large fonts, spacing, and visuals to enhance comprehension

Yes None needed

Fit with the community’s . . .

4. Cultural norms Curriculum should be relevant to people with different cultural norms—e.g., using the names offood types familiar to participants at each of the senior centers

Yes Adapt names of food types in curriculum to those familiar to each of the cultural groups who attend the different senior centers

GTO Step 4—Fit 42

Fit with the community’s . . . Considerations Fits? Yes/No


5. Environment Program has to be accessible to older adults living in neighborhoods served by senior centers with transportation to and from the centers—e.g., during daytime hours

Yes None needed

6. Other: ___________ Use of large fonts, spacing, and visuals to enhance comprehension

Yes None needed

Fit with your organization’s . . .

7. Mission “Improve the quality of life for our residents by promoting a safe and healthy community in which to live, work and play”

Yes None needed

8. Priorities Prepare city residents to respond to disasters, collaborate with community organizations to enhance preparedness

Yes None needed

9. Leadership support LHD officers must support project and use of staff time to plan, implement, and evaluate the program

Yes None needed

10. Context/setting LHD staff must have time and support to deliver the program in the community

Yes None needed

11. Readiness for intervention Implementation within nextsix months

Some work needed

Need to work with busy schedule to find time to implement this program while carrying out other work duties

12. Other: ____________

GTO Step 4—Fit 43

Program Fit Assessment Tool

Completed by: _____________________ Date:______ Program Being Considered: _____________ Fit with the target population’s . . . Considerations

Fits? Yes/No


1. Needs

2. Demographiccharacteristics

3. Other

Fit with the community’s . . .

4. Cultural norms

5. Environment

6. Other:_____________

Fit with your organization’s . . .

7. Mission

8. Priorities

9. Leadership support

10. Context/setting

11. Readiness for intervention

12. Other:_____________

GTO Step 4—Fit 44


After reviewing your potential programs with fit in mind, you might have a clearer idea which programs that you selected in Step 3 are still good possibilities. If there are candidate programs that would have to be drastically adapted to fit, then you may want to eliminate them before going on to Step 5. If none of the potential programs have passed the “fit” test you conducted here in Step 4, you may need to go back to Step 3 and do some more investigation to find a new set of programs to consider. Knowing more about fit now may also help you to more quickly zero in on potential programs if you do circle back to Step 3 for more investigation.

In Step 5, we’ll show you how to examine the current capacities of the lead agency/organization/partners to make sure it/they can do a good job in implementing the selected program. Step 5 will be the final reviewing step before moving onto planning and implementing the selected program, as well as establishing evaluation criteria.


When you finish working on this step, you should have: p Completed the Step 4 tools for all programs under consideration p Developed an understanding of what fit means p Considered the most important aspects of your program to make sure there is a good

fit with your target population, your organization, and your community p Determined the right adaptation needed, if any, to improve the fit of your program(s) p Further narrowed your choice of programs to implement

GTO Step 5—Capacities 45

Chapter Five

GTO Step 5—Capacities

What is GTO Step 5?

This step will help ensure that you have the capacity necessary to deliver a program as it was intended. There are six types of capacities:

• program staff• other staff• board and other leadership• technical• fiscal and other resources• partnerships/collaborations.

By assessing capacity first, you can avoid program options that you do not have the capacity to implement and make planning easier. Completing the Capacity Assessment Tool for each program you are still considering will help further narrow down your choices and make you aware of additional capacity you may need, depending on the program you ultimately choose to implement.


Understanding your organization’s capacity is important because • you cannot meet the goals and desired outcomes of any program without

adequate capacity to deliver the program as intended• inadequate capacity, or resources, can cause added burden on staffers and

other programs, as well as poor program implementation.


This step provides a structure to determine whether the program(s) you identified during GTO Step 3 can be carried out effectively with the knowledge, skills, and resources of your organization and its partners.

Step 5 Capacities


How do I carry out Step 5?

With your program choices narrowed, you can consider several dimensions of your organization’s capacity, including staffing and leadership, technical, fiscal, collaborations, and other resource capacities, to implement the remaining candidate programs and to consider how you could meet any capacity needs identified in your assessment.

Instructions for completing the Capacity Assessment Tool

1. Make as many copies of the tool as necessary to complete this task. You willconsider six areas of capacity for each program you are considering:

• program staff, including trainers, outreach, and managers• other staff—e.g., administrative assistants, survey data collectors• board and other leadership• technical• fiscal and other resources• partnership/collaboration.

2. Go through each section in the tool and answer the questions to determinewhether your organization’s capacity is adequate, and then, as appropriate,explain your plan to increase capacity. You should be sure to add to the tool anyadditional specific capacities that are required to implement your program. Forexample, some programs require two staff to facilitate. If you were implementingsuch a program, you would want to list this as a needed capacity in the CapacityAssessment Tool. Include volunteer staff to the extent that you will rely on them.

3. If you discover that your organization lacks the necessary capacities to deliveryour program with fidelity by adhering to all its core components, it is important tobrainstorm ways to build capacity in that area. For example, if your trainersstruggled to explain making a preparedness plan in a prior program, you maywant to train them on how to deliver that content more concisely and allow themto practice this part. Or if program staff members lack up-to-date information onrelevant topics, you may want to share with them some background informationprior to implementing the program. If you determine that your organization cannotdeliver the program because of capacity challenges, it may be better to delayimplementation of the identified program while you take time to build thecapacities that may be lacking, or you may want to select another program.


Example Capacity Assessment Tool

Completed by: Project team/coordinator Date:_January____ Program: ROAD-MAP

Program Staff Capacities

Considerations OK? Plan to Increase Capacity

Do you have the number of staff recommended for the program?

Beyond the 3–4 program staff required to run the program, the ROAD-MAP requires 3 trained volunteers per site to facilitate the program at each site implementing the program.

Some work needed

We have existing agency staff to lead this program and train the volunteer trainers. We will recruit at least 3 volunteers at each of 4 participating senior centers to receive the ROAD-MAP training and then facilitate 2 ROAD-MAP workshops at their senior centers.

Do your staff meet the following program qualifications: Education level? Program staff should have

educational background in public health. Volunteers should have high school reading level.

Some work needed

Will focus recruitment efforts only on those with a high school education or above

Years of experience?

Program staff have experience with program implementation. Volunteers do not require prior experience before training.

Yes None needed

Communication skills?

Program staff and volunteers must speak English or Spanish coherently.

Some work needed

Will only recruit volunteer trainers who can deliver training in English or Spanish. Assess need for Korean translator.

Are your staff comfortable enough with the topic to effectively deliver the program with fidelity?

Facilitator needs to be familiar with all ROAD-MAP components and fidelity must be monitored.

Some work needed

Training will be provided on ROAD-MAP. We will interview volunteers after they are trained to assess their comfort level. Then, during the trained-volunteer–facilitated ROAD-MAP trainings at the senior centers, health department trainers will assess fidelity using the ROAD-MAP fidelity measure.


Program Staff Capacities


Have your staff received necessary training for working with the target group?

Facilitators need to be trained to integrate the healthy aging orientation of ROAD-MAP for work with older adults.

Yes None needed

Have your staff received sufficient training specific to the program?

Staff and volunteers will be trained in ROAD-MAP.

Some work needed

We will host 2 two-hour training sessions with 12–16 volunteers from 4 senior centers and provide additional support to those who need or request additional time.

Other Staff Capacities

Program Considerations/ Requirements

OK? Plan to Increase Capacity

What type of additional staff do you need to implement your program?

No additional staff beyond core program staff and peer volunteers.

Yes None needed

Do additional staff members have adequate qualifications to implement this program?

N/A

Have additional staff members received necessary training for their roles for this program?

N/A


Leadership Capacities



How committed is your organization leadership to the program?

Public health agency leadership should be supportive of program focus on household preparedness among older adults.

It is important to also engage leadership at the senior centers to enhance target recruitment.

Some work needed

Hold regular meetings with leadership at senior centers to make sure they are involved and supportive of the program.

Does leadership support the staff involved in this program?

ROAD-MAP does not specify what type of leadership support is needed for program staff, but it can be assumed that health agency leadership is on board with staff dedicating work hours to plan, implement, and evaluate this program.

Some work needed

Talk to managers at the LHD about the benefits of the program so that they see the value of dedicating 10% of the 3-person team to this project. And talk to leaders at the senior centers to gain their endorsement and help recruiting volunteers and program participants.

Are there clear channels of communication between all leaders involved in this program?

Staff leading program and senior center leadership (executive directors) should have clear communication regarding recruitment of older adults at sites.

Some work needed

A weekly conference call will be set up between program staff and program site leadership during the program’s recruitment period.

Do the leaders involve staff members in decisionmaking when appropriate?

N/A

Is facilitation of organizational meetings effective?

N/A


Technical Capacities



Are any special materials needed to deliver the program?

Handouts and a DVD Yes None needed

Do you need access to a computer or special computer programs to implement the program?

No Yes None needed

Does the program require other technical components?

Audiovisual equipment to play DVD during the classes

Some work needed

We will rent this equipment before the first class if needed at each senior center.

Fiscal and Resource Capacities



Printed materials (including curriculum and recruiting flyers)

Cost: $800 for 8 workshops at 4 sites attended by 20 older adults at each workshop.

Yes None needed

Transportation Cost: $150 (300 miles at $0.50/mile)

Yes None needed

Staff Cost: About $10,000 based on a 9-month time frame for the project

Some work needed

Hold discussions with leadership to allocate staff time to this project. Explore grants that can provide funding.

Number of volunteers

At least 3 volunteers per host senior center

Some work needed

Will recruit 12–16 volunteers from senior centers

Equipment Cost: $50 for TV monitor rental

Yes None needed

Amount of space Cost: $0 Yes None needed

Evaluation materials and efforts

Cost: Project evaluator for 50 hours—$2,250

Yes None needed

Other: refreshments at senior centers for ROAD-MAP class participants

Cost: $800 (160 participants at $5 per participant)

Yes None needed

Other: tote bags and pill containers

Cost: $960 (160 participants at $6 per participant)

Yes None needed

Total cost Cost: $15,010 Some work needed

Depends on items listed above


Collaboration/ Partnership Capacities


Which community partners are key to the success of the program?

Department of Aging Multipurpose Centers for older adults (also known as senior centers)

Some work needed

Program staff will conduct community outreach, building on existing relationships, to find 4 senior centers to host program.

Which of these already provide support for programming?

Department of Aging Multipurpose Centers for older adults (also known as senior centers)

Some work needed

Program staff must continue and expand community outreach to find 4 senior centers to host program.

What other stakeholders in your community might support the program if asked?

- Senior independent housing communities - Other community programs affiliated with Department of Aging Multipurpose Centers - Community and faith-based organizations - Public health nurses at health departments - Member organizations from the Senior Emergency Preparedness Action Committee - Community Academic Partnership for Research in Aging, consisting of local physicians, local pharmacists, and local businesses

Some work needed

Program staff will conduct community outreach to involve these other stakeholders to expand the program to other sites or improve its delivery.

What stakeholders in your community could hinder implementation?

None Yes None needed


Capacity Assessment Tool

Completed by: _______________ Date:______ Program: _______________________ Program Staff Capacities


Do you have the number of staff recommended for the program? Do your staff meet the following program qualifications: Education level?

Years of experience?

Communication skills?

Are your staff comfortable enough with the topic to effectively deliver the program with fidelity?

Have your staff received necessary training for working with the target group? Have your staff received sufficient training specific to the program?


Other Staff Capacities


What type of additional staff do you need to implement your program? Do additional staff members have adequate qualifications to implement this program? Have additional staff members received necessary training for their roles for this program?

Leadership Capacities


How committed is your organization leadership to the program? Does leadership support the staff involved in this program? Are there clear channels of communication between all leaders involved in this program? Do the leaders involve staff members in decisionmaking when appropriate? Is facilitation of organizational meetings effective?


Technical Capacities


Are any special materials needed to deliver the program? Do you need access to a computer or special computer programs to implement the program? Does the program require other technical components? Fiscal and Resource Capacities


Printed materials (including curriculum and recruiting flyers)

Cost:

Transportation Cost:

Staff Cost:

Number of volunteers Equipment Cost:

Amount of space Cost:

Evaluation materials and efforts

Cost:

Other:__________ Cost: Other:__________ Cost: Total cost Cost:


Collaboration Capacities


Which community partners are key to the success of the program? Which of these already provide support for programming? What other stakeholders in your community might support the program if asked? What stakeholders in your community could hinder implementation?



You’ve now reviewed one or more EBPH approaches and programs for their potential to meet your goals and desired outcomes, their fit with your community and target audience, and your capacity to implement them.

It is possible that none of the programs on your list were feasible, given the significance of some capacity gaps. This is because capacity gaps in people, in agencies, or in the general community can prevent good implementation. If this is the case, you can either circle back to Step 3 to find more suitable programs, or you might decide to take a break from this process while you work to develop the required capacities. Remember, capacity-building is a long-term process, but it can yield important gains for an organization or community. The capacity you build for a specific program may also be useful for other programs.

It is also possible that you are now left with more than one program that meets your needs, fits with your community and agency, and is possible given your capacity. If this is the case, one approach to finalizing program selection would be to convene a meeting of the stakeholders and present all the information gathered in Steps 1 through 5. The stakeholders can discuss the findings together and the pros and cons of eachprogram. By iterating with the stakeholders, you might be able to identify which of the handful of remaining candidate programs fit the best.

After selecting a program and determining that you have the capacities to implement the program well, you are in the position to update your Logic Model Tool and develop an implementation plan (Step 6). Plans for filling gaps in capacity will be addressed as part of this plan.


When you finish working on this step, you should have: p Completed the Step 5 tool p Developed an understanding of the key capacities you need to support your programming p Assessed whether you have the right levels of capacity needed to implement your potential

program p Determined which capacities need to be further developed so that you can move ahead

with your programming p Further narrowed your choice among potential programs to implement

GTO Step 6—Plan 57

Chapter Six

GTO Step 6—Plan

What is GTO Step 6?

Having completed Steps 1–5 and selected the intervention you plan to implement (in our example, the intervention is ROAD-MAP, a 75-minute workshop facilitated by two trained peer trainers), you are now ready to develop the operations of your program. In this step, you will use the Work Plan Tool to create a detailed plan for running the program, which includes

• a written list of all program activities, frompreparations through quality improvement

• a timeline showing who, what, where, when, and howactivities should be implemented.


GTO Step 6 is important because having a plan • ensures that no key program tasks are left out• improves teamwork and partner communication• identifies the need for changes as things begin to run counter to the plan• reduces lost time, wasted energy, and turmoil from staff turnover• explains the scope of the program to people with an interest in the program.

Information from your plan also informs a key part of your Logic Model Tool from Step 2.


With your program selected, add the major program activities, such as recruiting, trainings, and running the workshops, to your Logic Model Tool. Next, complete the Work Plan Tool in this step. You will also complete the Program Budget Tool to identify financial considerations associated with implementing and evaluating the program. This tool elaborates on the financial resources estimated in Step 5. You may want to create a rough draft of the Work Plan Tool as you gather the necessary information. When the


This step helps you make a detailed work plan for delivering and evaluating the program you identified in Step 3 and selected at the end of Steps 4 and 5.

Step 6

Plan


tool is complete, ensure that you have the required details in place to carry out the program activities you listed in the Logic Model Tool.

Instructions for completing the Work Plan Tool

1. Make as many copies of the tool as necessary for you and your co-workers tocomplete the task.

2. Gather the tools you developed in the five previous steps (assessments, logicmodel, outcome statements, fit and capacity assessments) so that you canreference these tools as you complete the Work Plan Tool.Starting on the left, under Tasks, work your way down, completingimplementation task details for your program. The tool is divided into severalcategories of tasks: administrative, policies and procedures, programpreparation, recruitment and retention, implementation, and program evaluation.If possible, list program tasks in the order of occurrence to help you plan themout. Note: Tasks that are important to carry out in most programs have beenincluded in the Work Plan Tool to help you map out specific implementationevents. You may delete any tasks that are not relevant to your chosen programor organization and add tasks in the extra rows that may be important but notmentioned in the tool. Also included are columns to add data and person(s)responsible for the identified tasks—these may be adjusted as appropriate foryour organization. A separate column provides space to indicate where you willobtain any resources you need to complete a task. Enter N/A if the column doesnot apply.Note about evaluation tasks: You may want to review Step 7 (ProcessEvaluation) and Step 8 (Outcome Evaluation) to gain an understanding of theevaluation tasks that need to be included in the work plan.

3. When the draft Work Plan Tool is complete, distribute it to everyone involved inthe implementation at your organization for feedback. Then finalize andredistribute the finished Work Plan Tool.

4. Regularly review the plan while you prepare and implement the program toensure that tasks have not been neglected. Fill in the “Date Done” column whenWork Plan activities are complete and update the tool as new tasks arise.


Example Work Plan Tool

Completed by: Project team/manager Date:Feb.–Sept._____ Program: ROAD-MAP Tasks: Administrative When Will It

Be Done? (Time Frame)

Who Is Responsible?

Where Will We Get Any Resources We Need?

Date Done

Prepare budget for program implementation and associated expenses

2/1–2/10 LHD project manager

N/A 2/10

Complete job description for volunteer trainers


N/A 2/5

Conduct outreach and complete memorandum of understanding (MOU) with 4 senior centers


N/A 3/15

Acquire program curriculum and materials, including evaluation survey, tote bags, pill boxes, and other workshop supplies

2/15–3/10 LHD project coordinator

Program developer, UCLA

3/10

Schedule ROAD-MAP training for peer volunteers


N/A 3/10

Tasks: Policies and Procedures

When Will It Be Done? (Time Frame)

Who Is Responsible?


Date Done

Develop recruitment plan to get older adults in the program


ROAD-MAP Resource Package

2/28

Obtain any required permissions from senior center management—e.g., for placement of ads in newsletter and posters in public spaces


LHD project manager

3/15




Who Is Responsible?


Date Done

Schedule ROAD-MAP program into regular activities of the senior centers (calendar for space utilization, piggyback on existing meetings, select convenient dates and times, reserve equipment for workshop activities, etc.)


N/A 3/31

Tasks: Program Preparation When Will It Be Done? (Time Frame)

Who Is Responsible?


Date Done

Identify or recruit peer volunteers (these should be older adults) to be trained in ROAD-MAP and to lead workshops


LHD project manager and managers at senior centers

3/10

Train 12–16 volunteers on ROAD-MAP and evaluation data collection

3/20, 3/25 Make-up/follow-up date 3/28

LHD trainer will lead; LHD project manager, coordinator, and evaluator will assist

ROAD-MAP Resource Package and workshop materials for trainees

3/28

Review all program videos and/or materials in curriculum kit

2/15–3/5 LHD trainer ROAD-MAP Resource Package

3/5

Prepare workshop materials in advance, including attendance log and pre-/post-surveys and survey transmittal form for each workshop

4/1–4/10 LHD project coordinator and volunteers

Print materials at LHD; arrange LHD space for assembly

4/10

Confirm ROAD-MAP location with senior centers

4/1 LHD project coordinator

N/A 4/1


Tasks: Program Preparation When Will It Be Done? (Time Frame)

Who Is Responsible?


Date Done

Purchase refreshments for workshops

Ahead of each workshop

Volunteers Approved budget to reimburse up to $100 per workshop

1 day ahead of each

If needed, acquire and test audio/video equipment at the program delivery site

4/10–5/10 LHD project coordinator will check with volunteers on need for this equipment and arrange for its delivery if needed

Rent TV monitor from Rent-a-Center

5/15

If needed, organize transportation to and from senior centers for ROAD-MAP facilitators and/or participants

4/1–5/1 LHD project coordinator and volunteers

Staff will drive their own car, carpool, or use community shuttle service

Week before each imple-menta-tion session (dates listed below)


Tasks: Recruitment (and Retention)


Who Is Responsible?


Date Done

Develop and test workshop recruitment plan

2/1–3/1 LHD project manager and leadership at senior centers

N/A 3/1

Implement workshop participant recruiting

3/15–4/15 LHD project coordinator, leadership at senior centers, and volunteers

Distribute flyers and posters, post notice on calendars and websites, and send email invitations and reminders

4/15

Tasks: Implementation When Will It Be Done? (Time Frame)

Who Is Responsible?


Date Done

Run 8 ROAD-MAP workshops at 4 senior centers

4/17–5/15

4/17 and 5/5 at center 1; 4/19 and 5/7 at center 2 4/21 and 5/9 at center 3 4/25 and 5/15 at center 4

ROAD-MAP trained peer volunteer workshop facilitators and LHD observers

ROAD-MAP Resource Package from UCLA and all materials for 20 participants per workshop

5/15


Tasks: Program Evaluation When Will It Be Done? (Time Frame)

Who Is Responsible?


Date Done

Design evaluation and complete GTO Steps 7 and 8 planner tools

2/20–3/15 LHD project evaluator

Program developer (UCLA)

3/15

Train volunteers on data collection forms and protocols

3/20, 3/25, 3/26 LHD evaluator Sample forms

3/26

Administer pre-survey At start of each workshop: 4/17–5/15

Volunteer workshop facilitators

Print surveys at LHD in advance

4/17–5/15 be-fore each ses-sion

Administer post-survey At conclusion of each workshop: 4/17–5/15


Print surveys at office in advance

4/17–5/15 after last ses-sion

Administer follow-up survey 3 months after each class

LHD evaluator Phone and email survey

8/20

Complete fidelity monitoring tool At each class: 4/17–5/15

LHD trainer and evaluator

N/A After each class ses-sion (4/17 to 5/15)

Complete attendance roster At start of each class: 4/17–5/15


Print attendance roster at office in advance

Each class ses-sion (4/17 to 5/15)



Who Is Responsible?


Date Done

Review incoming forms and enter data

4/17–5/15 LHD evaluator LHD observers will bring all evaluation materials back to LHD office

5/15

Analyze process evaluation data from attendance rosters, fidelity logs, and satisfaction surveys, and complete GTO Step 7 summary tool

5/20–6/20 LHD evaluator Microsoft Excel; all completed data forms

6/20

Analyze outcome evaluation data (including pre- and post-survey data and 3-month follow-up) and complete GTO Step 8 summary tool

7/30–9/2 LHD evaluator

LHD project manager

Microsoft Excel; all completed data forms

9/2

Present evaluation results and decide which changes are needed to improve the program’s performance and outcomes, using GTO Step 9 CQI process and tools.

9/10, 9/15 LHD staff and volunteers and invited senior center management

Meeting space at LHD

Microsoft PowerPoint or Word; handouts

Consider brief report to post on senior centers web sites if budget permits

9/15



Who Is Responsible?


Date Done

Adjust goals and outcomes and reassess fit and capacity in light of implementation; update Work Plan Tool on lessons learned from program implementation

9/15–9/20 LHD staff Blank GTO worksheets; completed GTO worksheets

9/20

Complete all GTO documentation, inventory supplies, and begin planning for next round of workshops

9/20–9/30 LHD project manager and coordinator

Storage space

9/30


Work Plan Tool

Completed by:_____________ Date:________________ Program:__________________ Tasks: Administrative When Will It

Be Done? (Time Frame)

Who Is Responsible?


Date Done

Prepare budget Complete job descriptions Complete memorandum of understanding (MOU) with host site(s), if any Acquire program curriculum and materials, including evaluation materials Schedule program training for program staff



Who Is Responsible?


Date Done

Develop recruitment plan for program

Tasks: Program Preparation


Who Is Responsible?


Date Done

Designate existing staff as program facilitator, hire new staff, or recruit volunteers, as needed Train facilitators on program Develop community support through outreach (if needed)


Tasks: Program Preparation


Who Is Responsible?


Date Done

Confirm program implementation location (on site or at a community-based organization) If needed, copy worksheets needed for program modules If needed, purchase participant incentives If needed, purchase snacks for implementation If needed, acquire and test audio/video equipment at the program delivery site If needed, organize transportation for staff and/or participants

Tasks: Recruitment (and Retention)


Who Is Responsible?


Date Done

Develop and test recruitment (and retention) plan and materials Notify eligible population about upcoming program Confirm dates, time, and space and send reminders

Tasks: Implementation When Will It Be Done? (Time Frame)

Who Is Responsible?


Date Done

Detailed schedule for implementing the program and collecting evaluation data


Tasks: Program Evaluation


Who Is Responsible?


Date Done

Design evaluation and complete GTO Steps 7 and 8 planner tools Collect data Enter and analyze data Review process evaluation data from relevant data collection tools and complete GTO Step 7 summary tool Review outcome evaluation data (including pre- and post-survey data) and complete GTO Step 8 summary tool Present results and decide which changes are needed to improve the program’s performance and outcomes, using GTO Step 9 CQI process and tools. Adjust goals and outcomes and reassess fit and capacity in light of implementation; update Work Plan Tool on lessons learned from program implementation Finalize documentation, inventory any supplies, and begin planning next round or next steps


Instructions for completing the Program Budget Tool

1. Make as many copies of the tool as necessary for you and your co-workers tocomplete the task.

2. Enter the resources required to implement your program in each of the categoriesshown in the tool:• Personnel: You may have existing staff to plan, implement, and evaluate the

program, or you may need to hire part-time staff or recruit volunteers. Anexample of how to calculate staffing expenses is provided. Other personnel costsunique to your organization should be included in this section.

• Program materials, equipment, and supplies: Expenses should include theprogram curriculum and any purchases needed to run the program (laptop orDVD player, projector, easels, flip chart paper for facilitating activities, markers,pencils, etc.).

• Other (e.g., travel, transportation): Expenses should include the cost of travelingto and from the site where the program is being conducted, travel costsassociated with sending staff to the curriculum training, etc.

Add extra lines or categories, if necessary. You may want to create a rough draft of the Program Budget Tool as you gather the information necessary to determine the costs of your program. Consult the Step 5 section that you completed on financial capacities.

3. Subtotal the costs by category.4. Enter a total cost on the line provided at the end of the tool.5. When the tool is complete, distribute it to everyone involved in program

implementation.6. Be sure to update your budget periodically to account for changing costs. You may

also need to complete different budgets for subsequent years. For example, theequipment costs shown in our sample may be one-time costs only.


Example Program Budget Tool

Completed by: Project team/manager Date: February_____ Program: ROAD-MAP

Item by Category Calculation Cost Estimate

Personnel

LHD project manager: 10% of work time over 9 months

10% time × $5,000 salary/month × 9 months

$4,500

LHD project coordinator: 10% of work time over 9 months

10% time × $4,200 salary/month × 9 months

$3,780

LHD trainer: 10% of work time over 4 months 10% time × $3,500 salary/month × 4 months

$1,400

Project evaluator (contractor) $45/hour × 50 hours $2,250 Volunteers No cost $0

Personnel Subtotal $11,930

Program Materials, Equipment, and Supplies

Printing, copying, and mailing curriculum materials for workshops

$100 per workshop of 20 × 8 workshops

$800

Renting audiovisual equipment, if needed $50 Tote bags, pill containers, and sealable bags for workshop participants

$6 per participant × 20 participants × 8 workshops

$960

Program Materials, Equipment, and Supplies Subtotal $1,810

Other (e.g., travel, transportation)

Mileage reimbursement for staff site visits to senior centers

10 visits for training and observations

$150

Refreshments at 8 workshops $5/participant × 20 participants/workshop × 8 workshops

$800

Other Subtotal $950

Total Cost of Program Sum of All Category Subtotals $14,690


Program Budget Tool

Completed by: _______________ Date:____________ Program: ___________________ Item by Category Calculation Cost

Estimate

Personnel

Personnel Subtotal

$

Program Materials, Equipment, and Supplies

Program Materials, Equipment, and Supplies Subtotal

$

Other (e.g., travel, transportation)

Other Subtotal

$

Total Cost of Program Sum of All Category Subtotals

$



You’ve now brought all of the GTO tasks you’ve finished up to this point into a solid work plan for implementation. Before launching your program, we recommend that you take some time to complete the planner tools from Step 7 (Process Evaluation) and Step 8 (Outcome Evaluation). Doing so will help you plan in greater detail the process and outcome evaluation before you launch your program. This will allow you to more effectively monitor the program while it is running.

You should also update your Logic Model Tool now to reflect your final program choice, your key program activities, and your evaluation plans. The example Logic Model Tool shown here has been completed for the selected program to be implemented, ROAD-MAP.


When you finish working on this step, you should have: p Finalized your program selection p Completed the Step 6 tools p Updated your Logic Model Tool p Considered and planned appropriate program adaptations p Identified program components and activities p Considered and selected participant recruitment strategies, if applicable p Completed a program budget p Developed a work plan for implementing your program


Completed by: Project team/manager Date: February ____ Program: ROAD-MAP_____


1. Lack of preparation forthe medical needs of seniors during and after a disaster

2. Lack of householdpreparedness for seniors

3. Greater capacity forsenior centers to address needs of population


Collaboration between LHD staff and local senior centers


1. Increase the number ofpeople with a 7-day supply of their prescription medication

To increase the number of program participants indicating that they possess a 7-day supply of their medications by 15% from baseline to follow-up

ROAD-MAP resource package and workshop

• Personal safety• Health information

and medicalsupplies

• Communication• Safety at home and

emergency kit


Recruitment of ROAD-MAP workshop participants

Recruitment and training of peer trainers

Peer trainers host 8 workshops for older adults at 4 senior centers and deliver ROAD-MAP protocol

Evaluate process and outcomes


Reach (# of trainers, # of trainings, and # of people attending classes)

Fidelity of training to curriculum using ROAD-MAP tool

User satisfaction using ROAD-MAP survey


Baseline and 3-month follow-up surveys of class participants to measure both desired outcomes (use example measures from GTO Tip 8.1)

2. Increase the number ofpeople with 7-day household supplies (water, nonperishable foods, flashlight, etc.)

To increase the number of program participants indicating that they possess a 7-day supply of water and other emergency household supplies by 20% from baseline to follow-up

Example Logic Model Tool


GTO Step 7—Process Evaluation 74

Chapter Seven

GTO Step 7—Process Evaluation

What is GTO Step 7?

This step involves evaluating how well the program ran: Did you implement the program according to your plan, and how well did it go? This step is called process evaluation because the collected data track the process of the program implementation, as opposed to the outcomes experienced by the participants (which are covered in GTO Step 8). Process evaluations typically track attendance of participants, program adherence, and how well you followed your work plan. They may also involve asking about satisfaction of program participants or about staff’s perception of how well the program was delivered. A process evaluation should be planned before the program begins and should continue while the program is running. Therefore, this step includes the Process Evaluation Planner Tool to plan for evaluation activities before the program starts and a Process Evaluation Summary Tool to interpret the results of your evaluation.


The process evaluation tells you how well plans are being put into action and helps routinely and systematically monitor areas important to making the program a success. Examples include

• recruitment of participants• individuals’ attendance or exposure• program adherence• participant satisfaction• staff perceptions.

The process evaluation will also tell you whether you need to make midcourse corrections (e.g., improve attendance because attendance is weak) or changes to your work plan for your next program implementation. Such data will provide you with

Step 7 Process

Evaluation


This step provides guidance on what to include in, and how to gather data for, a process evaluation, which tells you how well you delivered the program.


information that may be useful to funders and help you better understand your program’s outcomes.


To do Step 7, complete the GTO Process Evaluation Planner Tool, carry out the data collection and analysis called for in your evaluation plan, complete the Process Evaluation Summary Tool, and consider changes needed to improve the program in the future based on your process evaluation results. If your program is run continuously, you will need to identify a time when you can make a change to how you run your program going forward. Changes for program improvement will be addressed in Step 9.


Tip 7-1. Types of process evaluation information

You are likely to use a variety of methods for collecting your process evaluation data. Here’s some additional information about a few key methods mentioned in this chapter.

Participant data

What it is: Specific information about participants, such as age, gender, race/ethnicity, education level, household income, family size, and referral source

How to gather it: You have probably already gathered much of this kind of information in the course of planning for, establishing, or running your program. Often, these types of questions are asked as part of an intake to a service or an outcome assessment survey. Information can be gathered during an interview with each participant as well.

Why it is important: It tells you whether your program is serving the targeted population and whether program outreach efforts are working to engage the participants you planned to reach.

Focus groups

What they are: Focus groups are facilitator-led discussions on a specific topic with a group of no more than 6–12 participants brought together to share their opinions on that topic.

How to manage them: Generally, focus groups are led by 1–2 facilitators who ask the group a limited number of questions. Think of the structure of a focus group like a funnel—each major topic should start with broad questions and then get more specific. Be sure to record the focus group or have a designated note-taker. The data can be analyzed by looking for the themes that appear in the transcripts or notes. The following resources provide more information on focus groups:

• Community Tool Box—Conducting Focus Groups:http://ctb.ku.edu/en/table-of-contents/assessment/assessing-community-needs-and-resources/conduct-focus-groups/main

• Designing and Conducting Focus Group Interviews:http://www.eiu.edu/~ihec/Krueger-FocusGroupInterviews.pdf

Why they’re important: Focus groups are an excellent way to learn what people thought about your program and get suggestions about your program. Focus groups often yield qualitative (i.e., text) data, as opposed to surveys, which usually yield quantitative (i.e., numerical) data. Listening as people share and compare their different points of view provides a wealth of information—not just about what they think but also why they think the way they do.

http://ctb.ku.edu/en/table-of-contents/assessment/assessing-community-needs-and-resources/conduct-focus-groups/main

http://www.eiu.edu/~ihec/Krueger-FocusGroupInterviews.pdf


Satisfaction surveys

What they are: Information about whether the participants enjoyed the program, whether they got something out of it, and whether the program met their needs or expectations

How to do them: The easiest way is to administer brief paper or web-based surveys to participants as part of the program at the end of each session or activity. This is better than waiting until the end of the entire program because sometimes participants forget details from earlier sessions. However, the surveys should be administered so that respondents feel comfortable that their responses will be kept confidential (i.e., service providers do not administer and collect responses). Surveys can also be handed out at the end of a program with self-addressed, stamped envelopes so the participant can complete the survey and return it later. This method, however, adds expense (cost of postage), and often fewer surveys are returned. If you are using a packaged program, it may require you to use a certain questionnaire with the program participants. You could also implement a web-based survey and invite participants to complete it via email.

Why they’re important: They tell you whether the participants feel good about the program and can help you identify ways to improve participant satisfaction, which would be likely to improve retention in the program.

Staff perception data

What they are: Staff perceptions about what worked and didn’t work during the implementation of a program. You may also want to collect staff perceptions about training and supervision quality.

How to gather them: There are several methods for gathering data on staff perspectives, including • focus groups• surveys• interviews.In addition to what we’ve already mentioned about focus groups, an interview can be a good way to get detailed information about program implementation. While interviews with staff involve a similar type of questioning as a focus group, in an interview you are talking with one person at a time.

A program debriefing is a straightforward way for staff to quickly meet immediately after a program session has been conducted and answer two questions: 1. What went well in the session?2. What didn’t go so well, and how can we improve it next time?

Why they’re important: Program staff are often in an excellent position to comment on how well a program is being implemented and may have ideas for improvement.


Program adherence monitoring

What it is: Systematically tracking how closely each intervention activity was implemented as laid out in your final work plan. This includes how much of a program was administered (“dose”) and whether it was administered according to the program developer’s intentions.

How to do it: If you are using a packaged program, check with those responsible for disseminating the program to see whether they have an adherence guide, and make sure to obtain the scoring criteria. If an adherence instrument does not come with the program materials or you have developed your own program, look at adherence guides from other programs and create your own.

Why it is important: The closer you can come to implementing a program as it was intended, the better chance you have of achieving your goals and outcomes. SOURCE: Adapted from Hannah, McCarthy, and Chinman, 2011.


Instructions for completing the Process Evaluation Planner Tool

1. Make as many copies of the tool as necessary for you and your co-workers tocomplete this task.

2. Assign a person responsible for collecting the instruments, forms, andquestionnaires containing all the process information you will gather in the course ofthe program. The person who takes on this role needs to be especially organizedand reliable.

3. Your Logic Model Tool (GTO Step 2), Work Plan Tool (GTO Step 6), and manual orcurriculum for your selected program will help you complete the tools in this step.

4. Consider each process question listed (and any you wish to add), and note yourmeasures and other considerations for data needed in the column labeled“Considerations.” For example, for Question 1, you might enter age and gender ifthese are the characteristics you are interested in.

5. Enter the evaluation methods and data collection tools that you will use to addressthe following process evaluation questions:

• Program participant characteristics, such as age and gender, can begathered in the pre-survey or via attendance or sign-in sheets.

• Utilization by individual program participants can be calculated from yourattendance rosters. Rosters should be designed to capture the percentageof time that participants attend each session or module (100 percent, 75percent, 25 percent, etc.). Then you can also sum how many of thesessions each registered participant attended.

• Level of delivery achieved by program may be determined by outsideobservers or program staff completing monitoring logs, checklists ofrequired activities and core elements, or simple notes about the actualdelivery, compared with the agenda or program guide. See the appendixfor a copy of the ROAD-MAP participant satisfaction and demographicssurvey.

• Participant satisfaction may be determined through participant focus groupdiscussions, general observations, or a post-program evaluation surveythat asks open-ended questions. Some programs have their ownsatisfaction surveys you can adapt. See the appendix for a copy of theROAD-MAP participant satisfaction and demographics survey.

• Staff perception can be determined by asking staff questions about whatthey believed to be the successes, challenges, and opportunities relatedto the program’s implementation.


• Work plan adherence can be determined by reviewing the initial Step 6Work Plan to see how closely it was followed. This could include trackingthe timeliness of carrying out various tasks or the extent to which youserved the number or type of expected participants.

6. Enter the anticipated schedule for data collection and analysis (i.e., when thedata will be collected and the frequency of collection) and when the results will beavailable.

7. Enter the person(s) responsible for gathering and analyzing the data. Forexample, the program facilitator may take attendance, the facilitator’s supervisormay monitor adherence, and the supervisor may ask staff about their perceptionsof the program.


Example Process Evaluation Planner Tool

Completed by: Project team/evaluator Date: March _____ Program: ROAD-MAP Process Evaluation Questions Considerations Evaluation Methods and

Data Collection Tools Anticipated Schedule for Data Collection and Analysis

Person(s) Responsible

1. What were the characteristics ofprogram participants comparedwith those of the targetpopulation?

Age, gender, race/ethnicity The participants in the workshops will be asked to complete the ROAD-MAP Participant Survey. The survey includes demographic questions.

Collection: at each workshop Analysis: after the last scheduled workshop

Volunteer program staff

2. What was the participants’program utilization compared withthe program plan?

Comparison is how close attendance at ROAD-MAP workshops was to the target of 20 participants per workshop.

Sign-in sheets Collection: at the start of each workshop Analysis: after the last scheduled workshop

Volunteer program staff collect data, and evaluator enters and analyzes data

3. What level of delivery did theprogram achieve, and did allplanned components getdelivered?

Adherence to curriculum content Quality of communication Observer should provide feedback to facilitators after the first workshop

The ROAD-MAP fidelity assessment tool will be used. An observer of the program will sit in on the workshop and complete this tool.

Collection: during each workshop Analysis: after the last scheduled workshop

Evaluator


Process Evaluation Questions Considerations Evaluation Methods and Data Collection Tools

Anticipated Schedule for Data Collection and Analysis


4. How satisfied were theparticipants?

General satisfaction with program Satisfaction with information delivered (Is it interesting, informative, easy to understand, helpful?) Satisfaction with pamphlets and other handouts (Are they helpful?)

Participants will be asked to complete the 1-page post-workshop ROAD-MAP Participant Survey. The survey includes questions about satisfaction with the program.

Collection: at the conclusion of each workshop Analysis: after the last scheduled workshop

Volunteer program staff administers; evaluator enters and analyzes data

5. What was the staff’s (includingvolunteers) perception of theprogram?

Program implementation staff Partnering site staff

Interviews with volunteers Debriefing meeting of all program staff

Collection: interviews within 1 week of each facilitator’s second workshop; meeting within one week of last workshop Analysis: after the last scheduled workshop

Evaluator

6. How closely did the programfollow the GTO Step 6 WorkPlan?

Administrative tasks Program policy procedures Recruitment and retention tasks Implementation planning tasks Evaluation planning tasks

We will examine the GTO Step 6 Work Plan Tool to see whether the person in charge of each task accomplished it as planned and by the target due date.

At debrief team meeting

Evaluator

7. Other N/A


Process Evaluation Planner Tool

Completed by: ____________________________ Date:______ Program: __________________________ Process Evaluation Questions Considerations Evaluation Methods

and Data Collection Tools

Anticipated Schedule for Data Collection and Analysis


1. What were the characteristics of programparticipants compared with those of the targetpopulation?

2. What was the participants’ program utilizationcompared with the program plan?

3. What level of delivery did the program achieve,and did all planned components get delivered?

4. How satisfied were the participants?

5. What was the staff’s (including volunteers)perception of the program?

6. How closely did the program follow the GTOStep 6 Work Plan?

7. Other


Instructions for completing the Process Evaluation Summary Tool

1. Ask the person(s) you identified to collect and analyze the data in the ProcessEvaluation Planner Tool to provide the results for which they were responsible.

2. Enter the results that answer the evaluation questions in the Process EvaluationSummary Tool. Be sure that the questions in the Process Evaluation SummaryTool are the ones you included in your Process Evaluation Planner Tool.

• Program participant characteristics describe the demographics of theprogram participants from these data (e.g., number of participants, male orfemale, ethnicity, and age).

• Utilization of program can be calculated from attendance information. Youcould calculate the percentage of participants who have perfectattendance (number with perfect attendance divided by all whoparticipated), the overall attendance rate for the whole group (total numberof sessions attended by all divided by total number of sessions the groupcould have attended), or the overall attendance for each session of theprogram (number of participants that attended session divided by totalnumber of participants enrolled in program). If the program consists ofonly one session, calculate attendance as a percentage of the totalanticipated or targeted.

• Level of delivery achieved by program will depend on the measure youuse. For example, you might calculate the percentage of activities fullycompleted, partially completed, and not at all completed for each sessionor component.

• Participant satisfaction and staff perception of the process will alsodepend on the measure you are using. If using a measure that asks open-ended questions, look across the answers for general themes. If using asurvey with defined answer choices, calculate averages or frequencies ofthe questions.

• Work plan adherence describes the percentage of activities that youskipped or failed to deliver based on your work plan.


Example Process Evaluation Summary Tool Completed by: Project team/evaluator Date: August Program: ROAD-MAP

Process Evaluation Questions Process Evaluation Data and Results

1. What were the characteristics ofprogram participants compared withthose of the target population?

The target age range (65+) was 100% achieved. More than 70% of participants were female in all classes, which was a higher representation of the target 50/50 split but is representative of the community population. We targeted certain underrepresented minorities, such as African Americans and Koreans, but the majority of respondents were non-Hispanic white or Latino across sites.

2. What was the participants’ programutilization compared with the programplan?

Our target of 8 workshops was achieved. While there was variability between class sizes, the average per class (16 people) was close to our target of 20 participants per class.

3. What level of delivery did the programachieve, and did all planned componentsget delivered?

In all 8 workshops, the trainers achieved or excelled at delivering the majority of the program content, methods, and implementation using the teaching protocol. In 2 of the classes, some of the content modules were skipped due to time constraints.

4. How satisfied were the participants? More than 90% of participants rated the information “very” informative and interesting, worth their time, and easy to understand. 96% of participants rated the provided materials as “very” helpful. 94% of participants indicated that they “very much” thought the information will help them make the right decisions. 95% of participants indicated that they definitely would recommend the program’s presentation to a loved one or friend.



5. What was the staff’s (includingvolunteers) perception of the program?

Peer volunteer workshop facilitators thought the program went well but that they could benefit from more training. Some peer volunteer workshop facilitators also wanted more time in the session to teach materials. The team was generally satisfied with the program but thought it could be provided in more languages and with greater organization from the presenters.

6. How closely did the program follow theGTO Step 6 Work Plan?

All team members who were responsible for each task were on top of them, but there were several delays in due dates, such as for completing an MOU with the site and designing evaluation materials.

7. Other


Process Evaluation Summary Tool

Completed by: _____________________ Date: ______ Program: ____________


1. What were the characteristics ofprogram participants compared withthose of the target population?

2. What was the participants’ programutilization compared with the programplan?

3. What level of delivery did the programachieve, and did all planned componentsget delivered?

4. How satisfied were the participants?

5. What was the staff’s (includingvolunteers) perception of the program?

6. How closely did the program follow theGTO Step 6 Work Plan?

7. Other



Once you’ve finished your process evaluation plan, you are ready to move on to Step 8, in which you’ll plan your outcome evaluation to examine whether you are achieving the changes you seek among individuals receiving your program.


When you finish working on this step, you should have: p Completed the Step 7 tools p Developed a clear process evaluation plan prior to launching your program, including a

plan for o Tracking the number of participants and their attendanceo Monitoring your program adherence

p Determined how well you followed your work plan p Analyzed your process evaluation data after running your program p Developed a plan for making mid-course corrections if necessary

GTO Step 8—Outcome Evaluation 89

Chapter Eight

GTO Step 8—Outcome Evaluation

What is GTO Step 8?

GTO Step 8 involves evaluating how well the program achieved the intended outcomes. Did the participants in the program change on the desired outcomes, such as knowledge, attitudes, and behaviors? This step is called outcome evaluation because the collected data track the desired outcomes of the program, as opposed to the process of program delivery (GTO Step 7). The outcome evaluation should be planned before the program begins and should have specific time points for data collection, such as before and after the program has gone through a complete cycle.

GTO Step 8 contains two tools: ü The Outcome Evaluation Planner Tool will help you plan your outcome

evaluation. ü The Outcome Evaluation Summary Tool will help you interpret the results

of your outcome evaluation.

When these are complete, you will be ready to undertake program improvement using GTO Step 9.


The purpose of Step 8 is to understand whether you have met the goals and desired outcomes established in GTO Step 2. Combined with the results of your process evaluation (GTO Step 7), this step will begin identifying program areas for improvement to help address those missed outcomes in an effort to improve the program while maintaining achieved ones. Outcome evaluation results can help you demonstrate the effectiveness of your program to your funders and other stakeholders.

Step 8 Outcome

Evaluation


This step helps with planning an outcome evaluation and using the results from it. An outcome evaluation reveals how well you met the goals and desired outcomes you set for the program in Step 2.



In GTO Step 8, you need an evaluation design and a data collection and analysis plan, including a measurement tool (e.g., a pre-/post-survey), a target population to be measured (e.g., all the participants in the program), a timeline for when to collect the data (e.g., from the pre-/post-survey), a plan for entering the collected data (usually into a spreadsheet), and a plan for analysis to determine whether outcomes were achieved (e.g., the change from the pre-survey to the post-survey). Outcome evaluations can be complex and costly and are often intimidating for program staff. This guide is meant to assist with simple outcome evaluations. If you want to carry out more-complicated outcome evaluations, you may need to get help from a trained program evaluator.

A design is a term for the type of evaluation you will conduct. The type of design guides when you collect data and from which groups. For example, a simple and inexpensive design uses a questionnaire to collect data from program participants just before a program begins and after a complete program is completed (often called a pre-/post-). This design might be appropriate to assess competency as an outcome of a train-the-trainer program. Another type of design, called the pre-/post- with comparison group, compares program participants with a similar group not receiving the program during the same time period. This way, you can be sure that any changes taking place in the participants getting the program from pre- to post- were real and did not happen to all nonparticipants (i.e., if both groups improve the same amount, then the program did not have an effect). This improves confidence that differences were due to the program and not to something else. That is why this design is a stronger way to evaluate whether the program led to changes in knowledge, attitudes, or behaviors over time. However, this design is more complicated, so you may want to consult a program evaluator. Finally, sometimes you may only be interested in how participants did in a program at the end. Surveying participants only at the end is called a post-only design. It is the easiest to do, but it is the weakest type of evaluation because you have no information about how much change occurred before the program started and it only includes participants who completed the program.

In outcome surveys, individual survey questions are often grouped together into topical categories called scales. For example, a knowledge scale may include several questions assessing different types of knowledge. The question responses can be combined to form a scale. Then, the analysis of these data can also be done easily by scoring each scale, calculating the average for the group surveyed, and then comparing the pre- and post- scores.


Tip 8-1. Examples of survey questions for different outcome areas

Outcome Area Survey Question(s) Response Options Perceived level of preparedness

How well prepared do you feel your household is to handle a large-scale disaster or emergency?

Well prepared, somewhat prepared, or not at all prepared

Preparedness kit Water supply: Does your household have a 7-day supply of water for everyone who lives there? (1 gallon per person per day) Food supply: Does your household have a 7-day supply of nonperishable food for everyone who lives there? (Food that does not require refrigeration or cooking) Medication supply: Does your household have a 7-day supply of prescription medication for each person who takes prescription medications? Battery-operated radio: Does your household have a working battery-operated radio and working batteries for your use if the electricity is out? Flashlight with batteries: Does your household have a working flashlight and working batteries for your use if the electricity is out? Personal comfort item: Does your preparedness kit have a personal comfort item for each household member, such as chocolate, a picture of loved ones, or a stuffed animal?

Yes, no, don’t know/unsure

Communication plans

Communication with family: In a large-scale disaster, what would be your main method or way of communicating with relatives and friends?

Communication with family: 1 Regular home telephones 2 Cell phones 3 Email 4 Pager 5 2-way radios 6 Other 7 Don’t know/unsure


Outcome Area Survey Question(s) Response Options Communication with authorities: What would be your main method or way of getting information from authorities in a large-scale disaster or emergency?

Communication with authorities:

1 Television 2 Radio 3 Internet 4 Print media 5 Neighbors

6 Other 7 Don’t know/unsure

Evacuation plans

Mandatory evacuation compliance: If public authorities announced a mandatory evacuation from your community due to a large-scale disaster or emergency, would you evacuate?

Yes, no, don’t know/unsure Reason if noncompliance: What would be the main reason you might not evacuate if asked to do so? a) Lack of transportationb) Lack of trust in publicofficials c) Concern about leavingproperty behind d) Concern about personalsafety e) Concern about familysafety f) Concern about leavingpets g) Concern about trafficjams and inability to get out h) Health problems (couldnot be moved) i) Other

Organizational ties

Formal support: Do you currently belong to a community organization (such as a school, church or other faith community, or a volunteer organization) that you can depend on in a disaster?



Outcome Area Survey Question(s) Response Options In the past year, have you

• received any specific preparedness trainings,such as first aid, CPR [cardiopulmonaryresuscitation], mental health first aid, orcommunity preparedness?

• participated in an emergency drill or exercise?• conducted outreach to neighbors or friends to talk

about preparedness?• volunteered in a community activity related to

disaster preparedness (e.g., outreach event,attended a presentation)?

• identified an organization you can rely on during adisaster?


In addition, the Final Report of the 2007 Public Health Response to Emergency Threats Survey (PHRETS), conducted in Los Angeles County by the David Geffen School of Medicine at UCLA, includes all the PHRETS questions on individual, workplace, school, and daycare preparedness and shelter in place, evacuation, and communications plans. This report is available at http://cphd.ph.ucla.edu/sites/default/files/downloads/PHRETS%202007%20Final%20Report_1.pdf

http://cphd.ph.ucla.edu/sites/default/files/downloads/PHRETS%202007%20Final%20Report_1.pdf


Tip 8-2. Data collection methods for measuring desired outcomes

Methods Pros Cons Cost Surveys Self-

administered surveys

Anonymous Inexpensive Easy to analyze Standardized Easy to compare with other data

Could be biased if respondents do not understand the questions or answer honestly May not have very many responses; some respondents may not answer all of the questions

Low to moderate

Telephone surveys

Easy to analyze Standardized Easy to compare with other data

Same as above, but those without phones may not respond Others may ignore calls

Moderate to high, depending on number of surveys to complete

Face-to-face structured surveys

Same as self- administered, but you can clarify responses

Same as self-administered but requires more time and staff time

High

Recorded interviews

Objective Quick Does not require new participants

Can be difficult to interpret Data are often incomplete

Low

Open-ended interactions

Open-ended face-to-face interviews

Gather in-depth, detailed info Info can be used to generate survey questions

Takes much time and expertise to conduct and analyze Potential for interview bias

Low to moderate if done in house Cost can be high if hiring outside interviewers and/or transcribers

Open-ended questions on a written survey

Can add more in-depth, detailed info to a structured survey

People often do not answer them May be difficult to interpret the meaning of written statements

Low

Focus groups

Can quickly get info about attitudes, perceptions, and social norms Info can be used to generate survey questions

Cannot get individual-level data from focus group Can be difficult to run Hard to generalize themes to larger group May be hard to gather 6–8 persons at same time Sensitive topics may be difficult to address in a focus group

Low if done in house Cost can be high if hiring a professional Usually incentives are offered to get participants

Other Observation (of children, parents, program staff)

Can provide detailed information about a program, a family, etc.

Observer can be biased Can be a lengthy process

Low to moderate if done by staff or volunteers

SOURCE: Adapted from Hannah, McCarthy, and Chinman, 2011.


Tip 8-3. Reporting evaluation results for different audiences

Obviously the most important reason we evaluate what we’re doing is because we want to know whether we’re having an impact. However, sharing our results in simple, meaningful ways can have other useful impacts as well. Keep in mind that different groups of stakeholders may be interested in different types of information. The general public may be less interested in lots of data than funders or local policymakers are. In this tip we have included some different ways that information might be reported for different audiences.

Stakeholder Information of interest Example of reporting method

Funder Whether the program is working

Detailed report with executive summary of findings Grant application (if applicable)

Community members Whether the program is working How the program can be improved How the program is impacting community members

Executive summary of findings and accompanying presentation, flyer, web page

Agency staff Whether the program is working How the program can be improved

Detailed report with executive summary of findings


Instructions for completing the Outcome Evaluation Planner Tool

This tool will help you plan how to carry out your outcome evaluation. While this tool allows you to create your own outcome evaluation survey items, we recommend that, whenever possible, you choose measures that already exist and have been used to evaluate programs like yours. Some programs have their own outcomes survey. With this tool, you can also choose your design (i.e., pre-/post-, pre-/post- with comparison group).

1. Make as many copies of the tool as necessary so that you have a row for each ofyour program’s outcomes.

2. Review the desired outcomes statement from the SMART Desired Outcomes Toolyou completed in GTO Step 2, and copy each desired outcome into the first column.

3. Check the appropriate box in the Evaluation Design column to indicate your choiceof evaluation design for each outcome.

4. Next, identify the scales and/or existing or new questions that you will use tomeasure each of your desired outcomes statements. See resources in this guide,literature, and manuals for programs like yours for examples.

5. Select a measure that can be used to assess each desired outcome. Enter this inthe next column.

6. In the next column, indicate from where you are pulling the scale or questions (forexample, your program’s survey).

7. In the last column, enter “All” if you are using all the items in the scale, or enter thenumber of items from a scale that you will use.

8. With this tool completed, you can construct your outcome survey questionnaire. Addany additional questions, such as demographics or level of participation orsatisfaction, that you also decide to measure.


Example Outcome Evaluation Planner Tool

Desired Outcome

Evaluation Design

Scale Name/Questions Source of Scale/ Questions

Items to Include

To increase the number of program participants indicating that they possess a 7-day household emergency water supply by 20% from baseline to follow-up (3-month period)

� Pre-/post- with comparison group

þ Pre-/post-

� Post- only

Behavioral Risk Factor Surveillance System (BRFSS) General Preparedness Module Q2: Does your household have a [7] day supply of water for everyone who lives there? A [7] day supply of water is 1 gallon of water per person per day. 1 Yes 2 No 7 Don’t know/Not sure 9 Refused

BRFSS, 2012 Question 2 from BRFSS General Preparedness Module

To increase the number of program participants who regularly take prescription medication who possess a 7-day extra supply by 15% from baseline to follow-up (3-month period)


þ Pre-/post-

� Post- only

California Health Interview Survey (CHIS) Emergency Preparedness QA09_EM1: Do you take any medicine daily that a doctor prescribed? YES.........................1 NO...........................2 QA09_EM2: Do you have at least an extra [one] week supply of all the prescription drugs you take every day? YES...........................1 NO............................2 DON'T KNOW......... –8 IF NO: Could you get an extra [one] week supply of all your prescription drugs? YES..........................1 NO...........................2 DON'T KNOW....... –8 IF NO: What is the main reason you would not be able to get an extra supply of your prescription drugs? __________________________ DON'T KNOW............ –8

CHIS, 2009 Questions EM1 & EM2

Completed by: Project team/evaluator Date: April_____ Program: ROAD-MAP


Outcome Evaluation Planner Tool

Desired Outcome

Evaluation Design

Scale Name/Questions

Source of Scale/Questions

Items to Include


� Pre-/post-

� Post- only � Pre-/post- with comparison group

� Pre-/post-


� Pre-/post-


� Pre-/post-


� Pre-/post-

� Post- only

Completed by: ___________________ Date: ______ Program: ________________


Instructions for Completing the Outcome Evaluation Summary Tool

This tool helps interpret your survey data to see how much change you achieved on the desired outcomes. With this tool you can summarize your pre- and post- scores for your program participants and a comparison group (if you have one).

1. Make as many copies of the tool as you need.2. Copy over your measures (scales of questions) from the Outcome Evaluation

Planning Tool.3. Enter the results from your survey instruments in the remaining columns.4. If you have pre-program data, calculate the pre-program averages for theparticipants in two parts:

o First, apply the scoring rule on each scale for each participant.o Second, calculate averages across all participants for each scale or item.

For each scale, add the scale scores for each participant together, thendivide by the number of participants. Place this final number into the Pre-Program Score column of the tool in the space labeled “Program.” Do thesame for single items.

5. Repeat the same procedure to generate post-program averages, if you have post-program data. 6. If you have data for a comparison group, you will need to calculate pre- and post- averages for each scale and enter them into the tool in the space labeled “Comparison” (below the participants’ scores) or write in “Not applicable” (N/A). 7. For each scale, calculate the percentage change from the pre- to post- averages:

o Subtract the pre-program average from the post-program average.o Divide the result by the pre-program average.o Convert to a percentage (you can do this by multiplying by 100).

8. If you used a comparison or control group, calculate the percentage change for thatgroup as well (for each scale), and enter it in the appropriate column. 9. Briefly summarize the meaning of each result in the Interpretation column. Forexample, if there is a 50-percent increase in knowledge among the program participants but only a 10-percent increase in the comparison group, you might interpret this greater positive change as a result of the program.


Example Outcome Evaluation Summary Tool

Completed by: Project team/evaluator Date: August____ Program: ROAD-MAP Item/Scale Name Pre-Program

Score Post-Program Score

Percentage Change [(post- minus pre-) divided by pre-]

Interpretation

Does your household have a 7-day supply of water for everyone who lives there? A 7-day supply of water is 1 gallon of water per person per day. 1 Yes 2 No 7 Don’t know/Not sure

Program: 56%

Program: 75%

33.9% The number of program participants who have a 7-day household water supply increased by 33.9% from pre-program evaluation to follow-up.

Comparison: N/A

Comparison: N/A

Peer-Mentored Preparedness (PM-Prep) Preparedness Index (Q1, Q6, and Q7)

Program: 35%

Program: 70%

100% The number of program participants with a household communication plan increased by 100% from pre-program evaluation to follow-up.

Comparison: N/A

Comparison: N/A


Item/Scale Name Pre-Program Score

Post-Program Score


Interpretation

Do you take any medicine daily that a doctor prescribed? YES.........................1 NO...........................2 DON'T KNOW........ –8

Program: 85%

Program: 85%

N/A This is not an outcome measure, but rather an introductory question before asking the two outcome questions.

Comparison: N/A

Comparison: N/A

Do you have at least an extra [one] week supply of all the prescription drugs you take every day? YES............................1 NO..............................2 DON'T KNOW......... –8

Program: 14%

Program: 16%

14.2% The number of program participants who regularly take prescription medication who have a 1-week extra supply increased by 14.2% from pre-program evaluation to follow-up.

NOTES: Program: Scores for the group of participants who received the program. Comparison: Scores for the group of participants who did not receive the program.


Outcome Evaluation Summary Tool

Item/Scale Name Pre-Program Score

Post-Program Score


Interpretation

Program: Program:

Comparison: Comparison:

Program: Program:


Program: Program:


Program: Program:


Program: Program:


Program: Program:


NOTES: Program: Scores for the group of participants who received the program. Comparison: Scores for the group of participants who did not receive the program.

Completed by: _______________ Date:________ Program: __________________



You should have some idea at this point whether you have actually achieved your desired outcomes. The final two steps in this process will help you reflect on what you’ve done, fine-tune your work before you conduct your program again, and bring together a set of ideas about how to sustain your work.


When you finish working on this step, you should have: p Completed the Step 8 tools p Identified the questions you want the evaluation to answer p Chosen the measures you want to collect p Developed methods to use in the outcome evaluation p Developed and finalized a plan to put those methods into place p Conducted the outcome evaluation (collected your data) p Analyzed data and interpreted your findings p Reported your results

GTO Step 9—Continuous Quality Improvement 104

Chapter Nine

GTO Step 9—Continuous Quality Improvement (CQI)

What is GTO Step 9?

Step 9 will help you both review your past decisions and use your process and outcome evaluation data to determine what worked well, where there is room for improvement, and what changes may be needed the next time you run the program.

GTO Step 9 contains one tool: ü The CQI Review Tool will help you create a snapshot of your program’s

successes and shortcomings and prompt you to identify necessary improvements.


CQI takes advantage of what you have learned over time from your evaluation to improve the program for the future without starting over from the beginning. It puts the investment made in evaluation to work by using the results to make changes and understand their effects for the next time you run the program. It helps all staff involved to keep your program fresh and a good fit for your participants, your organization, and your community. Although it began in manufacturing, CQI is becoming a part of routine operations in many health and social services organizations and can be applied across all programs, representing an emphasis by the organization on the quality of its services.


The CQI Review Tool will prompt you to summarize your evaluation data and work back through GTO Steps 1 to 8 as you assess what went well and what should be improved. You will evaluate whether you met the goal and desired outcomes you created in GTO Step 2 using the results from your process and outcome evaluations. This will prepare


This step provides a framework for using process and outcome evaluation data to make program improvements.

Step 9

CQI


you to decide whether and how to revise your goals and desired outcomes, reassess fit and capacity, and revise your work plan for the next program implementation. The next step is to assess whether the changes you make will prove to be effective.

Instructions for completing CQI Review Tool

1. Make as many copies of the tool as necessary for you and your co-workers tocomplete this task. Try to include as many stakeholders as possible in this review.

2. Assign a person responsible for collecting the completed GTO tools, including thenotes containing all of the process and outcome evaluation data gathered in thecourse of the program. You also will need your program materials (e.g., manual) tohelp you complete this tool.

3. Complete the first section: Priorities for Action.o Use materials gathered and generated in GTO Step 1 (Needs and

Resources Assessment) to enter your program’s information in theTargeted Need column. This could come from section 7 in the GTO Step 1Triaging Among Needs Tool.

o Using the SMART Desired Outcomes Tool from GTO Step 2 (Goals andDesired Outcomes), enter the desired outcome statement associated witheach need.

4. Use outcome evaluation data.o From your interpretation of the results in your Outcome Evaluation

Summary Tool (GTO Step 8), check the impact on each of the SMARTdesired outcomes (reached, missed, or exceeded). Finally, determinewhether any further action is needed (yes or no). Further action may beneeded if you did not reach your desired outcome or if you believe there isroom for improvement.

5. Complete the second section: Process Evaluation.o Using your GTO Step 7 Process Evaluation Summary Tool, complete the

fields describing program dates and target population. The lettered fieldsare asking for (A) total target population, (B) total number of participantswho attended at least one session, (C) total number of participants whoattended every session, and (D) total number of participants included inthe process evaluation. You can then calculate the percentage of yourtarget population that actually attended (D divided by A) and then thepercentage of actual participants included in the evaluation (D divided byB).


6. Complete the third section: Planning Program Improvements.o Once you know more about what has worked and not worked, you can

make decisions about changes to make before the program isimplemented again. The questions in this section prompt you to reviewpast GTO steps. If your answers suggest that changes are needed, youmight need to rework tools from previous steps. For example, if you needto change your goal or desired outcomes, you may need to make changesto the scales in your Outcome Evaluation Survey Tool. If you decide youneed to make changes in any GTO step, go back and update the relevantGTO tool. Answer each of the questions honestly, and, where needed,create strategies for improvement for your next implementation.

o For more information on making a small change and understandingwhether it makes a difference, review Promoting Success: A Getting ToOutcomes Guide to Implementing Continuous Quality Improvement forCommunity Service Organizations, by Sarah B. Hunter, Pat Ebener,Matthew Chinman, Allison J. Ober, and Christina Y. Huang, Santa Monica,Calif.: RAND Corporation, TL-179-NIDA, 2015(http://www.rand.org/pubs/tools/TL179.html). It provides more-specificguidance on conducting continuous improvement.

http://www.rand.org/pubs/tools/TL179.html


Example CQI Review Tool

1. Priorities for ActionTargeted Need

Desired Outcome Outcome Evaluation Result Action Needed? Yes/No

More than 1/3 of older adults (age 50+) do not possess a 7-day supply emergency kit.

To increase the number of program participants indicating that they possess a 7-day household emergency water supply by 20% from baseline to follow-up (3-month period).

Progress on desired outcome: þ Reached ¨ Missed ¨ Exceeded

No

A high prevalence of older adults who take prescription medications do not possess emergency stockpiles.

To increase the number of program participants indicating that they possess a 7-day supply of their medications by 15% from baseline to follow-up (3-month period).

Progress on desired outcome: ¨ Reached þ Missed ¨ Exceeded

Yes

Completed by: Project team/manager Date: September Program: ROAD-MAP


2. Process EvaluationDates and Participation Targets Program dates: Eight ROAD-MAP workshops run between April 17 and May 15 A. Total target population: 20 per class, 8 classes = 160 participants

Target population characteristics: Age 65+ 50/50 split of male and female participants Mixed racial/ethnic composition with at least 10% African American and 10% Cambodian

Program Adherence/Delivery B. Total program participants who attended at least one session: 115 C. Total who attended every session: N/A (program delivers entire curriculum in 1 session) What level of program adherence did you achieve (offer activities according to program requirements), and what evidence do you have to document this level of adherence?

A program evaluator sat in on workshops and filled out a fidelity assessment tool based on the instructor’s delivery of the program content, methods, and implementation. The adherence was generally good, with trainers receiving score of “achieved” or “excelled” for the majority of the program content, methods, and implementation. Only 2 of the workshops had missing content.

Process Evaluation Results Divide the Total at D by the Total at A

Divide the Total at D by the Total at B

D. Total participants in evaluation: 115 in pre-/post- and 100 in 3-month follow-up

% of target: 72% (D ÷ A × 100)

% of actual: 100% for pre-/post-, 87% for 3-month follow-up (D ÷ B × 100)

Evaluation participants (check all that apply): þFacilitators or staff þ Participants (all) ¨ Participants (some) ¨ Others ____________________

How well does the evaluation represent the population of participants in the program? (check one):

¨ Not at all well ¨ Somewhat well þ Very well


3. Planning Program Improvements

Step-by-Step Review Response Changes for the Next Time?

Did doing the program demonstrate the need for it? (GTO Step 1)

How relevant and current to our participants are the needs data? Does an evaluation show participation, satisfaction, and results?

The data are relevant to senior needs in the United States. Participant pre-program data also show need.

Continue to monitor pre-program data for needs of the target population.

Do we need to change goals and desired outcomes or potential participants? (GTO Step 2)

Target different conditions or behaviors? Reset benchmarks up or down?

We don’t need to change the current goals and desired outcomes because they are important targets for getting individuals prepared for disasters and they address needs that are specific to older adults.

We will consider adding a third goal related to emergency communication plans. Primarily, we need to devote more time to the prescription medications supply goal.

Should we consider another program? (GTO Step 3)

Or are there other improvements we need to make?

ROAD-MAP remains a relevant program to adapt for CEP aimed at older adults.

N/A

Does the program still philosophically and logistically fit our organization, community, and participants? (GTO Step 4)

If not, why not? What adaptations could be made? Were any adaptions made? How did that go?

Yes N/A

Do we have the resources and capacities to do the program well? (GTO Step 5)

Has there been a shift in resources? Are new staff capacities needed?

Changes in residents will require that we involve/train new peer volunteers to help execute the program.

Introduce staff to program by going over the program materials and GTO tool worksheets we completed to originally plan the program.



How well did we plan? (GTO Step 6) Any suggestions for improvement? Anything missing?

In general, the program was implemented as planned. However, certain elements of the timeline did not reflect the plan.

Create more-realistic timelines—e.g., explorelengthening workshop by 15 minutes or increase training on time management.

How well did we implement the program? (GTO Step 7)

Did we implement the program with adherence—i.e, were the core components delivered? What are the main conclusions from the process evaluation?

In all 8 classes, the trainers achieved or excelled at delivering the program core components, but some of the content was skipped in 2 of the 8 sessions due to time constraints.

Add an extra 15 minutes to the class so that the complete curriculum can be covered across all sites.

How effectively did the program help us reach our desired outcomes? (GTO Step 8)

What are the main conclusions from the outcomes evaluation?

The desired outcome of water and household supplies was reached. The prescription medication supply target was missed, though it did increase somewhat.

Review qualitative data on medications supply challenges and emphasize how to overcome these in the next round of classes, and devote more time to this topic. Perhaps annotating film content, adding a question-and-answer section, and adding a handout would help with this outcome.


CQI Review Tool

Completed by: _______________ Date: ____________ Program: ___________________

1. Priorities for ActionTargeted Need

Desired Outcome Outcome Evaluation Result Action Needed? Yes/No

Progress on desired outcome: ¨ Reached ¨ Missed ¨ Exceeded Progress on desired outcome: ¨ Reached ¨ Missed ¨ Exceeded Progress on desired outcome: ¨ Reached ¨ Missed ¨ Exceeded Progress on desired outcome: ¨ Reached ¨ Missed ¨ Exceeded


2. Process EvaluationDates and Participation Targets Program dates: _____ A. Total target population: _____

Target population characteristics:

Program Adherence/Delivery B. Total program participants who attended at least one session: _______ C. Total who attended every session: ________ What level of program adherence did you achieve (offer activities according to program requirements), and what evidence do you have to document this level of adherence? Process Evaluation Results Divide the Total

at D by the Total at A

Divide the Total at D by the Total at B

D. Total participants in evaluation: _____ % of target: _____ (D ÷ A × 100)

% of actual: _____ (D ÷ B × 100)

Evaluation participants (check all that apply): ¨ Facilitators or staff ¨ Participants (all) ¨ Participants (some) ¨ Others ____________________

How well does the evaluation represent the population served? (check one): ¨ Not at all well ¨ Somewhat well ¨ Very well


3. Planning Program Improvements


Did doing the program demonstrate the need for it? (GTO Step 1)

How relevant and current to our participants are the needs data? Does an evaluation show participation, satisfaction, and results?

Do we need to change goals and desired outcomes or potential participants? (GTO Step 2)

Target different conditions or behaviors? Reset benchmarks up or down?

Should we consider another program? (GTO Step 3)

Or are there other improvements we need to make?

Does the program still philosophically and logistically fit our organization, community, and participants? (GTO Step 4)

If not, why not? What adaptations could be made? Were any adaptions made? How did that go?

Do we have the resources and capacities to do the program well? (GTO Step 5)

Has there been a shift in resources? Are new staff capacities needed?

How well did we plan? (GTO Step 6) Any suggestions for improvement? Anything missing?

How well did we implement the program? (GTO Step 7)

Did we implement the program with adherence—i.e., were the core components delivered? What are the main conclusions from the process evaluation?

How effectively did the program help us reach our desired outcomes? (GTO Step 8)

What are the main conclusions from the outcomes evaluation?



In each chapter, we’ve tried to suggest ways you could begin thinking creatively about how to implement and sustain your program at each stage of its development. In Step 10, we’ll summarize these and other ideas for sustaining the successes of your program.


When you finish working on this step, you should have: p Completed the Step 9 tool p Documented successful program activities p Assessed which program activities did not work well overall or for specific groups p Identified areas for improvement p Created strategies for improvement p Increased buy-in within your organization by soliciting and acting on the suggestions of

program staff

GTO Step 10—Sustainability 115

Chapter Ten

GTO Step 10—Sustainability

What is GTO Step 10? This step will help you think through whether and how you can continue to deliver your program over time so that you reach the goal and desired outcomes established during GTO Step 2. Sustainability involves a deliberate effort to integrate the core elements of the program into the routine of your organization. This may include difficult discussions with staff and community members about what to change and/or discontinue if programming is not meeting established goals.

GTO Step 10 contains one tool—the Sustainability Review Tool. However, you will rely heavily on many of the tools from GTO Steps 1–9 to guide your discussions about sustainability efforts and the completion of this tool.

You will consider two important questions during this step:

1. What is working that should be sustained?2. How do we sustain activities that should continue?


First, if the original problem still exists, and your program shows that it achieves outcomes, then there is still a need for your program. By sustaining the program, the organization and the future participants will continue to get benefits from the large investment in starting the program. Sustaining effective programs maintains the positive feelings that your successful program generated among staff, community, and funders and adds to your reputation for delivering quality programming using an EBPH approach.


This step guides you through some questions to consider when making decisions about whether your organization should continue a program.

Step 10

Sustainability


Like planning to deliver program modules, sustainability should also be thoughtfully planned. The following are suggestions for improving sustainability that could be included in a sustainability plan.

• Program documentation: Make sure that all aspects of your program aredocumented so that key knowledge does not leave the program in the event of staffturnover.

• Program effectiveness: By creating and maintaining high visibility of programs thatare showing positive results (through publicizing the activities and positive evaluationresults of your program)‚ you can establish a reputation for effectiveness andincrease your program’s likelihood of being sustained.

• Program financing: Programs that rely completely on a single source of funds aremore vulnerable than those with a diversified funding base. Taking the followingactions can improve your chances of sustaining your program: (1) plan initially foreventual funding cutbacks, (2) cultivate additional resources while the program isongoing (e.g.‚ in-kind costs or low fees for services), and (3) adopt anentrepreneurial spirit in seeking additional support.

• Training: Programs that incorporate and train more people with ongoing jobs in anorganization are more likely to have lasting effects. These employees can continueto provide programming, train others, and form a constituency to support theprogram. The more people there are who are trained to conduct the program in anorganization, the more likely it is that the organization will be able to continue theprogram, even in the face of staff turnover. Keep in mind that if the only people whooperated the program were those fully funded by the program grant or contract, noone would be left to carry on any of its useful components when the grant or contractfunding was exhausted or those staffers left.

• Institutional strength: The strength of the institution implementing the program isrelated to sustainability. Institutional strengths include goal consistency between theinstitution and the program, strong leadership and high skill levels, and mature andstable organizations.

• Integration with existing programs or services: Programs that are stand-alone, orself-contained, are less likely to be sustained than programs that are well integratedwith the host organization(s). In other words, if the program does not interact andintegrate with other programs and services, the program will be easier to cut when


the initial funding ends. Therefore, program personnel should work to integrate their programs rather than to isolate and guard them.

• Fit within your community: Your program should demonstrate value over preexistingprograms. This will enhance your potential for sustainability of the program.

• Program champions: Program sustainability sometimes can depend on generatinggoodwill for the program’s continuation. Goodwill often depends on obtaining aninfluential program advocate or “champion.” This person can be internal to theorganization (e.g., a high-ranking member of the organization) or external.

• Direct oversight: Simply put, a program will more likely continue when a staffer isdirectly assigned to manage it and knows that his or her supervisor will be askingabout its progress.


Like Step 9, Step 10 involves a global or comprehensive review of (1) what you have done to date and (2) what you will do in the future to promote the program’s sustainability. In this section, you will address these questions, record your answers, and indicate the next steps needed to sustain your program.


Tip 10-1. Looking at the GTO steps with an eye toward sustainability

Each of the GTO steps provides a lens through which to assess different elements of sustainability. Here are some suggestions to guide your thinking on this important topic.

Getting started

Continue to build relationships. Whether you are starting something new or refining an existing program, relationships are always important to your success. Get buy-in all along the way from a diverse group of participants.

GTO Step 1

Ensure that the selected program is based on real needs in the community. As needs change, assess whether and how your program can meet those changing needs. Identify what sorts of resources you might need to sustain the success of the program.

GTO Step 2

Choose goals and objectives that are meaningful and important to program participants and your other stakeholders. Working toward goals that your stakeholders care about will help you gather support from your stakeholders to sustain the program.

GTO Step 3

Ground your efforts in what works. This will increase staff competence and confidence and help you deliver a strong program.

GTO Step 4

Take time to continually assess fit. The more congruent your program is with existing needs, resources, and characteristics in your community, the easier it will be to gain support for it.

GTO Step 5

Develop important capacities in an ongoing way. Training is important to ensure that your staff and volunteers know how to deliver a program. Ongoing training ensures that new staff are always up to date on your program and operations.

GTO Step 6

A good work plan tells your story. Developing and using a clear work plan optimizes your use of time, energy, and resources. It brings together all your research, assessments, goals, outcomes, and evaluation plans, which will help you track your work, communicate what you are doing, and more easily attain the goals of an effectively implemented program.

GTO Step 7

Process is important. Identifying strengths, weaknesses, and areas for improvement will increase your overall effectiveness, which helps build confidence in your program.

GTO Step 8

Positive outcomes are crucial. The centerpiece of sustainability is achieving positive outcomes. Clearly demonstrate the effectiveness of what you’ve done and tie it to your vision, goals, and the needs in your community. Involve the participants. Collect stories, especially from those who have completed the program and feel it worked well for them.

GTO Step 9

Revitalize your work. Looking for ways to continuously improve what you are doing keeps your work fresh and current and strengthens your overall program.

GTO Step 10

Plan for sustainability. You won’t know where you are going on this important topic if you do not describe your goals and figure out how you’ll know when you get there.

SOURCE: Hannah, McCarthy, and Chinman, 2011.


Instructions for completing the Sustainability Review Tool

1. Make as many copies of the tool as necessary for you and your co-workers tocomplete this task. The tool can be completed individually or as a group with oneperson as the recorder.

2. Assign a person responsible for collecting the completed GTO tools, including notescontaining the process and outcome evaluation data gathered in the course of theprogram. You will also want any written guidance (i.e., the manual) that came withyour chosen program to help you complete this tool.

3. Follow the questions and the guidance provided in each row. By answering eachquestion, you will address what you have done to date and how you want to dothings in the future.


Example Sustainability Review Tool: Past Work

Completed by: Project team/manager Date: September Program: ROAD-MAP Questions (Use your GTO tools as you ask

these) Answers

(Yes or no and explain) Next Steps

Wha

t we

have

don

e in

the

past

Does the need for the program continue? Consider whether the need for the program has changed or remains the same and whether there are any new concerns.

Yes, older adults, especially those from other senior centers in Townville, still face enhanced needs for disaster preparedness, especially with regard to medication stockpiling.

Explain the need or enter “N/A”

Seek funding to make the program an annual initiative of the LHD and older adult partners.

Are our results good enough to continue doing the program? Look at your results and determine the program’s impact on the participants.

Yes, one desired outcome was met, and we saw a small increase that fell well short of our goal for the other.

Expand to other sites to continue to accomplish desired outcomes. Make improvements (see CQI plan) to reach prescription medication supply target.

What particular result can we use to justify the program? Any goal or desired outcome that you achieved may be a good “result” to share with stakeholders to justify the program. Any dramatic improvement shown by your data is definitely a result you may want to share. Look at the Goals and CQI tools to see what desired outcomes were reached or exceeded.

Increase in number of participants with 7-day supply of water and other essential household items, and prescription medication supply improvement

Continue to share our results with LHD managers and other potential funders to support applications for additional funding to make the training an annual project.

What should we change about the way we do the program? Using the data reports and the CQI tool from GTO Step 9, think about the process—recruitment, enrollment, attendance, logistics, etc.—and consider whether one or more of these activities could be strengthened or changed to be a better fit for your site and staff.

Need better recruitment strategies for different groups Need to lengthen classes to cover all materials

Consult community leaders of targeted racial/ethnic groups to enhance recruitment strategies. Schedule longer classes at sites.


Questions (Use your GTO tools as you ask these)

Answers (Yes or no and explain)

Next Steps

Who knows the program and supports keeping it going here? Consider which individuals at the site are champions of this program—i.e., influential people who really like the program—and are enthusiastic about it, including leadership. Should somebody else be brought on? Who is going to take the lead?

The LHD program manager, coordinator, and trainer all have experience running the program. The executive directors and other leadership and volunteers at senior center sites also know and support it.

The LHD trainer is going on an extended leave, so we need to engage a replacement trainer who is well qualified to work with older adults.

Are the GTO tools completed and available to use in the future? You need access to the completed GTO tools. This could mean getting electronic and paper copies from the person who completed them.

No We need to scan all completed tools so that we have a digital file that can easily be shared.


Example Sustainability Review Tool: Future Work

Completed by: Project team/manager Date: September Program: ROAD-MAP

Sustainability Questions Answers Next Steps

Wha

t we

will

do

in th

e fu

ture

Where will the GTO tools, the program evaluation, and the program manual and materials be kept? Decide who will have access to them and consider how this is the same or different from other program materials at your site. Where do you currently keep them?

They will be kept in a program binder at our public health office.

Need to scan all hard copies to also have a digital file with all these materials. Email to all on the team.

Who will be in charge of making the program happen? Also, think about who is trained to be the facilitator, how that decision was made, and what supervision would be necessary.

The program manager will remain in charge as long as she is working at the health department.

If current position changes, we will need to familiarize the replacement program manager.

Who else is in favor of and needs to be involved in keeping the program going? Think about the staff, leadership, board members, and the community of youth.

LHD leadership Executive directors and other leadership at senior center sites

Need to continually meet to ensure ongoing support of program.

Who will do the evaluation and pre-/post- surveys, track attendance, and monitor adherence? When (how often) and to whom will the results be reported? Think about who could lead these activities (1 person or more—staff or an outside group). Think about how to organize the results and who needs to see them.

The team evaluator Engage evaluator again as soon as funding is secured for next round.

How much funding, if any, do we need for running the program? Are there resources other than funds that are needed to run the program well (e.g., for recruitment, good attendance, supplies)?

The first iteration of program cost $14,000, but we can save on personnel in the future as we continue to use our trainers to deliver the program.

Revise program budget and submit to LHD management to ensure that funding is available to keep the program running.



When will we run the program again? And when will we revise the Step 6 Work Plan? Consider the different times of year or days and times of the week and what worked best in the past. Think about lead time needed to look back at the Work Plan and revise it for the future.

We are considering running the program again in a year.

Recruit additional sites earlier and start participant notification earlier.

How can we keep staff trained in the program? The more staff are trained, the more likely you will be able to continue the program. Look back at the program materials and what is required of facilitators. Consider who could be trained and who would be responsible for doing the training.

Keep a health educator on board so that he or she can train additional volunteers to teach classes.

Ask current trainer to help us find a replacement before she goes on leave.


Sustainability Review Tool: Past Work

Completed by: _______________ Date: ____________ Program: ____________________ Questions (Use your GTO tools as you ask

these) Answers

(Yes or no and explain) Next Steps

Wha

t we

have

don

e in

the

past

Does the need for the program continue? Consider whether the need for the program has changed or remains the same and whether there are any new concerns.

Explain the need or enter “N/A”

Are our results good enough to continue doing the program? Look at your results and determine the program’s impact on the participants.

What particular result can we use to justify the program? Any goal or desired outcome that you achieved may be a good “result” to share with stakeholders to justify the program. Any dramatic improvement shown by your data is definitely a result you may want to share. Look at the Goals and CQI tools to see what desired outcomes were reached or exceeded. What should we change about the way we do the program? Using the data reports and the CQI tool from GTO Step 9, think about the process—recruitment, enrollment, attendance, logistics, etc.—and consider whether one or more of these activities could be strengthened or changed to be a better fit for your site and staff. Who knows the program and supports keeping it going here? Consider which individuals at the site are champions of this program—i.e., influential people who really like the program—and are enthusiastic about it, including leadership. Should somebody else be brought on? Who is going to take the lead? Are the GTO tools completed and available to use in the future? You need access to the completed GTO tools. This could mean getting electronic and paper copies from the person who completed them.


Sustainability Review Tool: Future Work

Completed by: _______________ Date: ____________ Program: ____________________


Wha

t we

will

do

in th

e fu

ture

Where will the GTO tools, the program evaluation, and the program manual and materials be kept? Decide who will have access to them and consider how this is the same or different from other program materials at your site. Where do you currently keep them? Who will be in charge of making the program happen? Also, think about who is trained to be the facilitator, how that decision was made, and what supervision would be necessary. Who else is in favor of and needs to be involved in keeping the program going? Think about the staff, leadership, board members, and the community of youth. Who will do the evaluation and pre-/post- surveys, track attendance, and monitor adherence? When (how often) and to whom will the results be reported? Think about who could lead these activities (1 person or more—staff or an outside group). Think about how to organize the results and who needs to see them. How much funding, if any, do we need for running the program? Are there resources other than funds that are needed to run the program well (e.g., for recruitment, good attendance, supplies)? When will we run the program again? And when will we revise the Step 6 Work Plan? Consider the different times of year or days and times of the week and what worked best in the past. Think about lead time needed to look back at the Work Plan and revise it for the future. How can we keep staff trained in the program? The more staff are trained, the more likely you will be able to continue the program. Look back at the program materials and what is required of facilitators. Consider who could be trained and who would be responsible for doing the training.



When you finish working on this step, you should have: p Completed the Step 10 tool p Reviewed how each of the previous steps helps with sustainability p Identified at least one, if not more, respected program champions p Developed a sustainability plan

127

Chapter Eleven

Summary

You have completed the Getting To Outcomes® Guide for Community Emergency Preparedness. We hope you found this guide useful as you planned for successful implementation of a program or initiative in your community.

By working through this guide, you have learned that the ten steps of GTO are interrelated; much of the content in each step is used to inform the activities for subsequent steps. If you skipped any of the steps, consider a review of those steps and complete the associated tools to ensure that you have a solid plan for implementing the program.

• GTO Step 1—Needs and Resources Assessment:Determine needs and resources in your community.

• GTO Step 2—Goal and Desired Outcomes: Identify at least one goal and anassociated desired outcome.

• GTO Step 3—Best Practices: Choose an EBPH approach or program to implementthat reflects best practices.

• GTO Step 4—Fit: Ensure appropriateness of the EBPH approach or program for yourtarget population, community, and organization.

• GTO Step 5—Capacities: Ensure that abilities and resources are in place toimplement the program effectively.

• GTO Step 6—Plan: Develop a comprehensive plan for implementing the program.

• GTO Step 7—Process Evaluation: Plan and conduct a process evaluation.

OBJECTIVE

This chapter briefly summarizes the ten steps of GTO and suggests that you complete any steps you may have skipped earlier.

128

• GTO Step 8—Outcome Evaluation: Plan and conduct an outcome evaluation.

• GTO Step 9—Continuous Quality Improvement (CQI): Review evaluation data andpast GTO steps and make program changes as needed.

• GTO Step 10—Sustainability: Develop a plan for securing continued support.

Although fully completing the activities and tools provided in this guide will make your program more successful the first time, GTO is a continuous process. The next implementation will be enhanced by an ongoing review of the past implementation, using the process outlined in Step 9. Ultimately, the GTO process leads to strong program outcomes and will benefit the staff and participants in your program.

129

Appendix

ROAD-MAP Process Evaluation Workshop Observation Form

Instructions: This evaluation tool is designed to assist the principal investigators by summarizing observations of the disaster preparedness class. The form documents the completion of required intervention activities (content area) and the volunteer’s communication skills.

Rate the volunteer’s implementation of each content area listed below and his/her demonstration of communication skills during the session.

• Use a score of 1 (“Not Achieved”) to indicate that the goal was not achieved at all—i.e., every element listed for the item was skipped or implemented incorrectly.

• Use a score of 2 (“Partially Achieved”) to indicate that the goal was only partiallyachieved—i.e., at least one but not all the elements listed were implemented correctly.

• Use a score of 3 (“Achieved”) to indicate that the goal was achieved—i.e., all of theelements listed were implemented in a correct and satisfactory manner.

• Use a score of 4 (“Excelled”) to indicate that the goal was exceeded—i.e., all of theelements listed were implemented in at least a satisfactory manner, and one or moreitems were implemented in an exceptional manner.

Date: ___/___/___

Name of senior center: ______________________________

Number of participants: _____

Language of presentation: English □ Spanish □

Duration of session: _______________ min./hrs.

130

Content Skills

Did the volunteer:

Not Achieved

Partially Achieved

Achieved Excelled

1. Hand out “Be Ready” packets? 1 2 3 4 2. Discuss creating a disasterpreparedness tool kit?

1 2 3 4

3. Discuss supplies needed duringa disaster (e.g., nonperishable food items, flashlight)?

1 2 3 4

4. Explain how to develop acurrent medications list?

1 2 3 4

5. Provide a list of skills on how toprotect oneself during a disaster?

1 2 3 4

6. Explain how to create acommunication plan in case of a disaster?

1 2 3 4

Communication Skills

Did the volunteer:

Not Achieved

Partially Achieved

Achieved Excelled

1. Demonstrate professionalism? 1 2 3 4 2. Establish a rapport with seniors(e.g., introduce himself or herself and the purpose of the class)?

1 2 3 4

3. Listen effectively (allow seniorsto speak, avoid needless interruptions)?

1 2 3 4

4. Identify and clarifymisconceptions?

1 2 3 4

5. Solicit feedback from seniors toensure understanding of the material?

1 2 3 4

131

ROAD-MAP Program Satisfaction Survey

LET US KNOW WHAT YOU THINK! Today’s date: _________________

What is your gender? Male □ Female □

What is your age? Under 65 □ 65–75 □ 76–85 □ Over 85 □

What do you consider your race/ethnicity? Hispanic/Latino □ White □

African American □ Asian/Pacific Islander □

Other □ (specify) ______________

Did a spouse, partner, or caregiver attend this class with you? Yes □ No □

We want to know how useful the “Be Ready” presentation was to you. Please mark the box that best corresponds with your answer.

How would you rate this presentation? Very Much Somewhat Not at All

Was this presentation informative and interesting?

J K L

Was the presentation worth your time? J K L

Was the person presenting the information easy to understand?

J K L

Do you think the “Be Ready” pamphlet will be helpful to you?

J K L

Do you think the information you learned during this presentation will help you make the right decisions about how to prepare for a disaster?

J K L

Do you think preparing for a disaster will help you?

J K L

CONTINUES ON BACK

132

Which of the following best describes your thoughts about preparing for a disaster? Please circle the number that best represents your response.

Please choose one statement only.

I have heard about preparing for a disaster but never thought about doing it myself.

1

I am trying to decide whether I should prepare for a disaster. 2

I have decided I do not want to prepare for a disaster. 3

I have decided I do want to prepare for a disaster, but I haven’t started to do this yet.

4

I have decided I do want to prepare for a disaster, but I have not finished doing so.

5

I have decided I do want to prepare for a disaster, and I have finished doing so.

6

Would you recommend the “Be Ready” presentation to a friend or loved one? Definitely Yes □ Maybe □ Probably Not □

What suggestions or comments do you have for improving this program?

____________________________________________________________________

Thank you very much! This evaluation will help us improve future disaster preparedness classes for older adults in this county.

133

Bibliography Acosta, J., M. Chinman, P. Ebener, P. S. Malone, S. Paddock, A. Phillips, P. Scales, and M. Slaughter, “An Intervention to Improve Program Implementation: Findings from a Two-Year Cluster Randomized Trial of Assets—Getting To Outcomes,” Implementation Science, Vol. 8, No. 87, 2013, PMC3751245.

Aldrich, N., and W. F. Benson, “Disaster Preparedness and the Chronic Disease Needs of Vulnerable Older Adults,” Prevention of Chronic Disease, Vol. 5, No. 1, 2008. As of July 25, 2017: http://www.cdc.gov/pcd//issues/2008/jan/07_0135.htm

Al-Rousan, T. M., L. M. Rubenstein, and R. B. Wallace, “Preparedness for Natural Disasters Among Older US Adults: A Nationwide Survey,” American Journal of Public Health, Vol. 104, No. 3, 2014, pp. 506–511.

American FactFinder, homepage, undated. As of July 25, 2017: https://factfinder.census.gov/faces/nav/jsf/pages/index.xhtml

Assistant Secretary for Preparedness and Response Technical Resources, Assistance Center, and Information Exchange (ASPR TRACIE), “Topic Collection: Hazard Vulnerability/Risk Assessment,” last updated July 26, 2017. As of July 26, 2017: https://asprtracie.hhs.gov/technical-resources/3/Hazard-Vulnerability-Risk-Assessment/0

Brunkard, J., G. Namulanda, and R. Ratard, “Hurricane Katrina Deaths, Louisiana, 2005,” Disaster Medicine and Public Health Preparedness, Vol. 2, No. 4, 2008, pp. 215–223.

CDC—See Centers for Disease Control and Prevention.

Centers for Disease Control and Prevention, “Tools and Resources,” last updated September 26, 2016. As of July 25, 2017: https://www.cdc.gov/ncbddd/disabilityandhealth/emergency-tools.html

Centers for Disease Control and Prevention, “Public Health Preparedness Capabilities: National Standards for State and Local Planning,” last updated April 26, 2017b. As of July 26, 2017: https://www.cdc.gov/phpr/readiness/capabilities.htm

Centers for Disease Control and Prevention, “Behavioral Risk Factor Surveillance System,” last updated May 8, 2017c. As of July 25, 2017: https://www.cdc.gov/brfss/

Chinman, M., J. Acosta, P. Ebener, P. S. Malone, and M. Slaughter, “Can Implementation-Support Help Community-Based Settings Better Deliver Evidence-Based Sexual Health Promotion Programs? A Randomized Trial of Getting To Outcomes®,” Implementation Science, Vol. 11, No. 78, 2016, DOI 10.1186/s13012-016-0446-y, PMC4886421.

Chinman, M., S. Hunter, P. Ebener, S. Paddock, L. Stillman, P. Imm, and A. Wandersman, “The Getting To Outcomes Demonstration and Evaluation: An Illustration of the Prevention Support System,” American Journal of Community Psychology, Vol. 41, 2008, pp. 206–224, PMC2964843.

City-Data, homepage, 2017. As of July 26, 2017: http://www.city-data.com/

http://www.cdc.gov/pcd//issues/2008/jan/07_0135.htm

https://factfinder.census.gov/faces/nav/jsf/pages/index.xhtml


https://www.cdc.gov/ncbddd/disabilityandhealth/emergency-tools.html

https://www.cdc.gov/phpr/readiness/capabilities.htm

https://www.cdc.gov/brfss/

http://www.city-data.com/

134

The Community Guide, “Emergency Preparedness,” undated. As of July 25, 2017: https://www.thecommunityguide.org/topic/emergency-preparedness

Community Tool Box, “Section 6. Conducting Focus Groups,” 2017. As of July 25, 2017: http://ctb.ku.edu/en/table-of-contents/assessment/ assessing-community-needs-and-resources/conduct-focus-groups/main

Council on Linkages Between Academia and Public Health Practice, Core Competencies for Public Health Professionals, June 26, 2014. As of July 25, 2017: http://www.phf.org/resourcestools/Documents/ Core_Competencies_for_Public_Health_Professionals_2014June.pdf

David Geffen School of Medicine at UCLA, 2007 Public Health Response to Emergent Health Threats Survey (PHRETS II), 2007. As of July 25, 2017: http://cphd.ph.ucla.edu/sites/default/files/downloads/PHRETS%202007%20Final%20Report_1.pdf

Eisenman, D., A. Bazzano, D. Koniak-Griffin, M. A. Lewis, C. Tseng, K. Lamb, and D. Lehrer, “Peer-Mentored Preparedness (PM-Prep): A New Disaster Preparedness Program for Adults with Developmental Disabilities,” Intellectual and Developmental Disabilities, Vol. 52, No. 1, 2014, pp. 49–59.

Eisenman, D., D. Glik, L. Gonzalez, R. Maranon, Q. Zhou, C. Tseng, and S. Asch, “Improving Latino Disaster Preparedness Using Social Networks,” American Journal of Preventive Medicine, Vol. 37, No. 6, 2009, pp. 512–517.

Federal Emergency Management Agency, National Household Survey, 2015, undated. As of July 25, 2017: https://www.fema.gov/media-library-data/1467116407881-096f834e638f02fa8499941ce5c155bb/ national_household_survey_2015_508_062716_508.pdf

Fernandez, L. S., D. Byard, C. C. Lin, S. Benson, and J. A. Barbera, “Frail Elderly as Disaster Victims: Emergency Management Strategies,” Prehospital and Disaster Medicine, Vol. 17, No. 2, 2002, pp. 67–74.

Hannah, G., S. McCarthy, and M. Chinman, Getting To Outcomes in Services for Homeless Veterans: 10 Steps for Achieving Accountability, Philadelphia, Pa.: National Center on Homelessness Among Veterans, 2011.

HealthData.gov, homepage, undated. As of July 26, 2017: https://www.healthdata.gov/

Healthy City, homepage, undated. As of July 26, 2017: http://www.healthycity.org/

Healthy People, homepage, 2017. As of July 25, 2017: https://www.healthypeople.gov

Healthy People, “About Healthy People,” 2017. As of July 25, 2017: https://www.healthypeople.gov/2020/About-Healthy-People

HHS emPOWER Map 2.0, home page, last updated December 30, 2016. As of July 26, 2017: https://empowermap.hhs.gov/


http://ctb.ku.edu/en/table-of-contents/assessment/assessing-community-needs-and-resources/conduct-focus-groups/main


http://cphd.ph.ucla.edu/sites/default/files/downloads/PHRETS%202007%20Final%20Report_1.pdf


https://www.healthdata.gov/

http://www.healthycity.org/

https://www.healthypeople.gov

https://www.healthypeople.gov/2020/About-Healthy-People

https://empowermap.hhs.gov/

135

Hunter, Sarah B., Pat Ebener, Matthew Chinman, Allison J. Ober, and Christina Y. Huang, Promoting Success: A Getting To Outcomes Guide to Implementing Continuous Quality Improvement for Community Service Organizations, Santa Monica, Calif.: RAND Corporation, TL-179-NIDA, 2015. As of July 25, 2017: http://www.rand.org/pubs/tools/TL179.html

Kohn, S., J. L. Eaton, S. Feroz, A. A. Bainbridge, J. Hoolachan, and D. J. Barnett, “Personal Disaster Preparedness: An Integrative Review of the Literature,” Disaster Medicine and Public Health Preparedness, Vol. 6, No. 3, 2012, pp. 217–231.

Krueger, R. A., Designing and Conducting Focus Group Interviews, St. Paul, Minn.: University of Minnesota, October 2002. As of July 25, 2017: http://www.eiu.edu/~ihec/Krueger-FocusGroupInterviews.pdf

Levac, J., D. Toal-Sullivan, and T. L. O’Sullivan, “Household Emergency Preparedness: A Literature Review,” Journal of Community Health, Vol. 37, No. 3, 2012, pp. 725–733.

Li, R., Advancing Behavioral and Social Research of the Elderly in Disasters, Washington, D.C.: Committee on Populations, National Academies, 2009.

McGuire, L. C., E. S. Ford, and C. A. Okoro, “Natural Disasters and Older US Adults with Disabilities: Implications for Evacuation,” Disasters, Vol. 31, No. 1, 2007, pp. 49–56.

National Family Caregiver Support Program, Just in Case: Emergency Readiness for Older Adults and Caregivers, undated. As of July 25, 2017: https://www.acl.gov/sites/default/files/programs/2016-10/Just_in_Case030706_links.pdf

Powell, P., M. Smith, and L. Black, “Involving Youth in CEP: Impacts of a Multistate Initiative,” Journal of Youth Development, Vol. 4, No. 4, 2016, p. 19.

Public Health Emergency, “Tools,” last reviewed May 16, 2017. As of July 25, 2017: https://www.phe.gov/Preparedness/planning/abc/Pages/resources.aspx#tools

Ready.gov, “Community Preparedness Toolkit,” undated. As of July 26, 2017: https://www.ready.gov/community-preparedness-toolkit

Ronan, K. R., K. Crellin, and D. M. Johnston, “Community Readiness for a New Tsunami Warning System: Quasi-Experimental and Benchmarking Evaluation of a School Education Component, Natural Hazards, Vol. 61, No. 3, 2012, pp. 1411–1425.

UCLA Center for Health Policy Research, “California Health Interview Survey,” 2012. As of July 25, 2017: http://healthpolicy.ucla.edu/CHIS/Pages/default.aspx

UCLA Center for Public Health and Disasters, “Tools and Resources,” undated. As of July 25, 2017: http://www.cphd.ucla.edu/tools-and-resources

Vulnerable & At-Risk Populations Resource Guide, homepage, undated. As of July 26, 2017: http://www.varpguide.com/

Whitman, S., G. Good, E. R. Donoghue, N. Benbow, W. Shou, and S. Mou, “Mortality in Chicago Attributed to the July 1995 Heat Wave,” American Journal of Public Health, Vol. 87, No. 9, 1997, pp. 1515–1518.

http://www.rand.org/pubs/tools/TL179.html

http://www.eiu.edu/~ihec/Krueger-FocusGroupInterviews.pdf

https://www.acl.gov/sites/default/files/programs/2016-10/Just_in_Case030706_links.pdf

https://www.phe.gov/Preparedness/planning/abc/Pages/resources.aspx#tools

https://www.ready.gov/community-preparedness-toolkit

http://healthpolicy.ucla.edu/CHIS/Pages/default.aspx

http://www.cphd.ucla.edu/tools-and-resources

http://www.varpguide.com/