Presented by:

Carolyn Rider & Amanda Linares Research & Evaluation Section

Webinar Session One

April 8, 2013

Impact/Outcome Evaluation

Training for Local Health

Departments

The Network IOE Team

Carolyn Rider, MA• IOE project coordination• Main point of contact for technical assistance: planning and

reporting; data entry templates; evaluation materials and resources; evaluation requirements

• Teleconferences• Compendium of Surveys

Amanda Linares, MS• IOE data management and analysis• Compilation of submitted IOE data and USDA report generation• Teleconferences• Compendium of Surveys

You may be referred to other staff, but your first point of contact is Carolyn.Rider@cdph.ca.gov or IOEval@cdph.ca.gov

Today’s Agenda

• Evaluation Overview

• IOE’s Role in the Network

• Overview of IOE Requirements

• Resources for Conducting IOE

• Required and Optional Surveys

Evaluation Overview

How it works…

• Objective– Identify successful interventions and potential best

practices and provide direction for program improvement, refinement, and redirection of effort

• Participants– Local Health Departments (LHDs) receiving >$350,000

or choosing Objective 11 as an optional objective• Evaluation Support

– Research & Evaluation Section (RES) provides training; quality surveys, evaluation tools and resources; and year-round one-on-one evaluation guidance

How it works…

• State level analysis of compiled LHD findings enables the Network to assess overall project accomplishment

• Individual and compiled LHD findings are shared with USDA annually

How it works…

• Some aspects of evaluations are standardized – Minimum sample sizes– Standardized survey tools– Pre-test/post-test design

• Other evaluation components are designed locally, for example:– Optional survey modules can be added– Control groups

Why Evaluate?

• Your organization– How can we improve our intervention?– What works?

• USDA and other funders– What are we getting for the money we are

spending on these programs?

• Public– There is a high level of interest in improving

diets and physical activity and reducing the prevalence of overweight and obesity

Impact Evaluation

• Includes a control or comparison group

• Strongest designs include randomization

• Provides evidence that outcomes are a result of the intervention

• Excludes alternative interpretations

• Only a few of our IOE projects conduct Impact Evaluation

Outcome Evaluation

• No control or comparison group

• Determines if changes occur in conjunction with an intervention

• Cannot exclude alternative explanations, i.e. does not prove that the observed outcomes are attributable to the intervention

• Most of our IOE projects are outcome evaluation

Impact vs. Outcome Evaluation

Impact Outcome

Pre/Post-test Survey Design X X

Control Group X

Evidence that intervention causes outcomes X

Evidence that outcomes occur in conjunction with intervention X

Questions on …

Evaluation Overview?

Impact Outcome Evaluation’s Role in the

Network

IOE’s Role in the Network

• What is the history of Impact Outcome Evaluation?

• Why do we need to use standardized surveys?

• What happens to my data and final report after I turn it in?

IOE History

• In FFY 04 the Network asked contractors receiving over $500,000 in Federal Share to conduct IOE to proactively demonstrate fiscal responsibility– 12 contractors participated

• In FFY 05 the Network lowered the participation threshold to $350,000– In FFY 07 there was peak participation of 48

contractors– In FFY 12, 40 contractors participated

• In FFY 13, 59 contractors, including 17 new LHDs, are required to participate in IOE

• In FFY 14, we anticipate all but 10 LHDs will have budgets requiring them to participate

Using Standardized Surveys to Evaluate Network Priority Outcomes

When individual LHD results are compiled statewide, we can assess the following:

• Increased consumption of healthy food – especially fruit and vegetables

• Increased consumption of healthy beverages – water and milk

• Decreased consumption of less healthy food and sugar-sweetened beverages

• Increased physical activity

• Increased food security (adults only)

Elementary and High School IOE Had Good FV Findings in FFY 12

  N Pre-test* Post-test* Difference* p-value

Elementary/Middle School

Total Consumption 8,658 3.57 3.95 0.38 <0.001

Fruit 8,674 1.89 2.12 0.23 <0.001

Vegetable 8,666 1.68 1.83 0.15 <0.001

High School  

Total Consumption 884 5.78 6.28 0.50 <0.001

Fruit 885 2.08 2.26 0.18 0.002

Vegetable 885 3.70 4.02 0.32 0.001

Elementary/Middle School = 30 contractors; High School = 2 contractors

* Times per day

Adult FV Findings Were Also Quite Strong

  N Pre-test* Post-test* Difference* p-value

Intervention          

Total Consumption 804 2.54 3.22 0.68 <0.001

Fruit 807 1.22 1.59 0.37 <0.001

Vegetable 808 1.32 1.63 0.31 <0.001

Control          

Total Consumption 72 2.55 2.27 -0.28 0.032

Fruit 72 1.28 1.06 -0.22 0.008

Vegetable 79 1.26 1.21 -0.05 0.472

Adults= 10 contractors

* Cups per day

Youth Showed Positive Effects for Drinks and Sweets in FFY 12

Elementary School

  N Pre-test Post-test Difference p-value

Elementary/Middle School        

Water 8,522 3.5 3.58 0.08 <0.001

Punch, Sports Drinks, Fruit-Flavored Drinks 8,594 0.9 0.84 -0.06 <0.001

Soda 8,628 0.58 0.57 -0.01 0.225

Sweets 8,639 0.81 0.7 -0.11 <0.001

High School    

Water 893 3.47 3.4 -0.07 0.175

Punch, Sports Drinks, Fruit-Flavored Drinks 882 0.92 0.74 -0.18 <0.001

Soda 885 0.73 0.6 -0.13 <0.001

Sweets 882 0.68 0.57 -0.11 <0.001

Elementary/Middle School = 30 contractors; High School = 2 contractors

Adults Showed Positive Effects In Addition to FV for FFY 12

  N Pre-test Post-test Difference p-value

Adult          

Drink Fruit Drinks, Sports Drinks, Punch 510 2.01 1.82 -0.19 <0.001

Drink Soda 503 1.80 1.64 -0.16 <0.001

Drink Milk 496 2.72 2.85 0.13 0.001

Drink or Use Milk on Cereal Past Week 507 0.79 0.83 0.04 0.007

Take Skin off Chicken 518 2.93 2.99 0.06 0.223

Eat Fish Past Week 493 0.58 0.65 0.07 0.002

Use Food Labels 488 2.40 2.84 0.44 <0.001

Run Out of Food by End of Month 485 1.91 1.83 -0.08 0.028

Rate Eating Habits 477 5.72 6.51 0.79 <0.001

Youth Showed Gains in Physical Activity

  N Pre-test Post-test Difference p-value

Elementary/Middle School      

Physical Activity ≥60 Min 8,554 3.47 3.99 0.52 <0.001

Play Outdoors ≥30 Min 8,526 3.31 3.85 0.54 <0.001

High School          

Physical Activity ≥60 Min 851 3.51 3.81 0.30 <0.001

Play Outdoors ≥30 Min 848 3.06 3.25 0.19 0.032

Elementary/Middle School = 30 contractors; High School = 2 contractors

After the data and final reports are in…• Pre-post data from each set of surveys are combined;

then the data are cleaned and basic analysis is performed

• A report based on the aggregated data is written for the USDA final report, due near the end of each calendar year

• Section 1, or the first 21 questions, of your individual final reports are given to USDA as part of the yearly final report

• We do advanced analysis on the combined pre-post dataset– This year, analysis from FFY 12 will be presented at

the Society for Nutrition Education and Behavior Conference in August, and hopefully the American Public Health Association Annual Meeting in November

Questions on …

Impact Outcome Evaluation’s Role in the Network?

Requirements and Guidelines

for Conducting IOE

Local Health Department Scope of Work (SOW)

• GOAL 1: – The target population (Supplemental Nutrition Assistance Program-

Education (SNAP-Ed)/Nutrition Education and Obesity Prevention (NEOP) participants and those eligible up to 185% Federal Poverty Level (FPL)) is empowered and enabled to select healthy foods and beverages and increase physical activity through nutrition education, social marketing and environmental supports.

• Objective 11: – (Impact/Outcome) Annually, conduct and report on Impact Outcome

Evaluation (IOE) to assess change in healthy food and beverage consumption and related factors such as perceived benefits of eating healthier foods and beverages, perceived control, self-efficacy, readiness to consume healthier foods and beverages, and perceived diet quality among a group of at least 100 participants.

Guidelines for IOE

To help LHDs receiving over $350,000 in USDA funds to complete IOE SOW requirements, RES has developed a set of guidelines that provide additional information. This webinar reviews these guidelines.

LHDs can also access the guidelines on the IOE website:

http://www.cdph.ca.gov/programs/cpns/Pages/ImpactEvaluation.aspx

Guidelines for IOE

• The Network requires local health departments that receive over $350,000 to conduct impact/outcome evaluation (IOE) using the following guidelines.

• The guidelines were developed to generate sound information through rigorous evaluation that can be used to improve nutrition education activities or develop new ones.

• Technical assistance is available throughout the year to ensure the guidelines are applied to produce valid evaluation results as well as to fit unique situations.

IOE Guidelines

Contractors are expected to include an IOE objective in their SOW.

IOE Guidelines

Contractors are expected to measure change in fruit and vegetable consumption and factors that influence it or beverage consumption and factors that influence it using a pre-test/post-test survey design.

IOE Guidelines

Contractors are expected to use an age-appropriate tool selected from the following:

Children in 3rd-8th grades: Network Youth Survey

Adolescents in 9th-12th grades: Network High School Survey or Beverage and Snack Questionnaire 2 (BSQ2) pending approval of Rethink Your Drink (RYD)

Adults: Food Behavior Checklist (FBC), Fruit and Vegetable Checklist (FVC), or Adult RYD Survey pending approval of RYD

IOE Guidelines

• Contractors are expected to evaluate an intervention that has face-to-face contact with the same individuals for at least 30 minutes, on five or more different occasions.

• This may include contacts made by sub-contractors or collaborating agencies if it can be demonstrated that the individual has contact at the other site.

IOE Guidelines

Contractors are expected to submit results for at least 100 matched surveys, i.e., a pre-test (collected before the intervention) and post-test (collected after the intervention) for the same individual.

IOE Guidelines

Contractors that have previously conducted IOE are expected to do a more rigorous evaluation each year. This may include: • adding a comparison group • increasing sample size• Many other possibilities…

IOE Guidelines

• Contractors are responsible for collecting, entering, and analyzing their data and are expected to submit data and a report by July 31st of each year.

• Data entry templates (DETs) are provided by the Network for all required surveys* to assist with data entry and analysis.

*DETs are still under development for the new beverage surveys (BSQ2 and RYD Surveys).

IOE Guidelines

• Contractors are expected to submit a draft evaluation plan with their final report by July 31st of each year.

• This plan should make specific any general language written into the SOW and incorporate findings from the prior year’s evaluation.

Questions on …

Requirements and Guidelines for Conducting IOE?

Resources Available for LHDs Conducting IOE

Evaluation Tools & Resources Available to LHDs

• Trainings and Webinars• Website• Guidelines for LHDs• Compendium of Surveys, including mandatory

and optional surveys• Survey administration protocols• Templates for creating evaluation plans and

reports• Data Entry Templates, Codesheets & Instruction

FAQ• Planning Teleconferences• One-on-one technical assistance

IOE Website

http://www.cdph.ca.gov/programs/cpns/Pages/ImpactEvaluation.aspx

• IOE Plan and Final Report Templates• Data entry code sheets• 2010-2012 Final IOE Reports• Conference presentations drawn from Network IOE• Other misc research and evaluation resources

Planned Website Updates:

Survey Administration Protocols

• Administration Protocols should be printed from the Compendium of surveys:– Network Youth Survey– Network High School Survey

• Instruction Guides are shipped with surveys when they are ordered from the Network:– Food Behavior Checklist– Fruit and Vegetable Checklist

• For other surveys, please check the Compendium of Surveys for protocol information.

IOE Plan Template

Plan templates are provided by RES to assist LHDs to develop a sound evaluation plan.

During FFY 2013, this plan was due on 11/30. In subsequent years, it is due 7/31, along with the Final Report.

IOE Report Template

Final Report templates are also provided for LHDs to summarize intervention activities and evaluation results in a standard format.

Final Reports are due annually on July 31, along with supporting data files and the next year’s draft plan.

We will discuss this template in depth on Wednesday.

Data Entry Templates

LHDs use data entry templates (DETs) created for our standard surveys to enter data and produce a basic, standardized analysis

This DET has data entered for the Network High School Survey. We will discuss DETs in depth on Wednesday.

Technical Assistance: Teleconferences and One-on-one TA

• An annual teleconference is scheduled for each LHD after we receive your Final Report and Plan (due July 31) and before beginning the next year’s evaluation. – Who is there?

• All necessary staff from your LHD• Any subcontractors essential to your project• Network Program Manager• Network Research & Evaluation representative

– What will happen?• Discuss your Final Report from the previous year and your

plan for next year; troubleshoot intervention and evaluation

• One-on-one TA is available year-round by phone or email. Contact Carolyn Rider.

Questions on …

Resources Available for LHDs Conducting IOE?

Required and Optional Surveys

Required and Optional Surveys• Mandatory Survey Instruments

– Youth • Network Youth Survey

– High School • Network High School Survey or Beverage and Snack

Questionnaire 2 (BSQ2)– Adult

• Food Behavior Checklist, Fruit and Vegetable Checklist, or Adult Rethink Your Drink Survey

• Optional Survey Instruments– Network Compendium of Surveys is a comprehensive

collection of Network-approved survey instruments

Compendium of Surveys

• Updated regularly to provide contractors with the best measures we can find to evaluate their programs

• Instrument topics include knowledge, self-efficacy, social norms, social support, food security, access to/availability of healthy food and physical activity opportunities, and many more

• A new edition is currently on the Network website

Compendium of Surveys

Questions on …

Required and Optional Surveys?

For IOE Training Day 2…

IOE Training Day 2

• Be sure to download the Example IOE Intervention file AND a Youth Final Report Template– We will use the details from this example

intervention to get started filling out a Final Report Template together

– This will be a good opportunity to troubleshoot any questions that may be confusing or unclear

Wednesday’s Agenda

• Engaging Stakeholders in the Evaluation Process

• Using Network Templates to Enter and Analyze Survey Data

• Interpreting Data Analysis Results

• Preparing an IOE Final Report• Sharing and Using Evaluation

Results for Program Improvement

Questions on …

Today’s Webinar?