Improving Final Report Writing in California’s Tobacco Control Projects

Jeanette Treiber, PhD

Robin Kipke, MS

UC Davis

Background

CDPH funds approx. 100 local programs for tobacco control (mainly for policy work)

CDPH also funds evaluation capacity building Center (TCEC)

3 yr funding cycles require evaluation plan and reporting

CDPH provides reporting guidelines Reporting guidelines are not well adhered to 2007 reports received relatively low scores

The Problem

Evaluators do not think that their work is being used (Torres et al. 1997)

“Evaluators’ most long-standing concern about the lack of use of their work has to do with the dustgathering qualities of unread final reports.” (Preskill and Russ-Eft 2005)

Utilization focused evaluation relies in part on good reporting

Objectives

Build reporting capacity of local California tobacco control projects

Identify ECB practices that can help in the improvement of report writing

Reasons Weak evaluation reporting Reporting procedures not followed Aggregate data hard to generate from reports Questionable use as formative evaluation tools

Scoring Criteria (Excerpt)

Evaluation Design Section

Type of design (experimental, quasi-experimental, or non-experimental)

0 1 2

Reason for selecting the design used 0 1 2

# of times data are collected, when data are collected (pre-tests, during intervention, and/or post-tests), # of groups compared (if any), and whether activities varied by group

0 1 2

Any limitations of the design as a way to assess the intervention process and/or outcome

0 1 2

Capacity Building Methods & Measures

Regional one-day training events throughout California

Webinar One-day training satisfaction survey Compare scores of:

Overall mean 2007 vs 2010 Paired t-test of comparable sub-group of 2007

vs. 2010 reports Score improvement rate of those that attended

vs. those that did not attend training events

Results

Attendance: One-day training: 79 projects; 103 participants* Webinar: 21 projects One-day training + webinar: 20 projects

Training Agenda

Why Final Evaluation Reports Matter Anatomy of a Final Evaluation Report (FER)

a review each section, scoring criteria, common pitfalls hands-on exercise: score and discuss a sample FER

Writing the Results Section a review of several examples hands-on exercises taken from actual progress reports

Making evaluation reports work for multiple audiences How to repackage information in the FERs

Office Hours Individual consultation about your Final Evaluation

Report or evaluation activities

Webinar Agenda

Q & A session on commonly asked questions Available guidance/resources Identifying primary report audience The writing process How to handle changes to evaluation plan How to report negative results, lost data Reporting on multi-site evaluation Privacy issues Report needs involvement of project director

Training Satisfaction Survey Results (partial)

“The training will help me write a high quality Final Evaluation Report”

74% strongly agreed 23% agreed

“The presentations were informative” 71% strongly agreed 24% agreed

“The interactive exercises were helpful.” 71% strongly agreed 23% agreed

Training Satisfaction (continued)

Learning tool most appreciated:

Hands-on exercise / role play of scoring report

Hands-on eval. training

is most useful

(Trevisan 2004;

Patton 2005)

Overall Score Comparison

Pre/Post Score Comparison

Intervention Pre mean score in %

Post mean score in %

Percent point change

None 61.9 63.8 + 1.9

One-day training

65.6 78.3 + 12.7

One-day training and webinar

62.2 70.6 + 8.4

Limitations

Possible scorer effect Limited measure of training methodology

effect Insufficient sample size to measure webinar

effect

Lessons Learned

One-day training events have good potential of success in improving report writing

Hands-on exercises are preferred methods of learning

Training satisfaction surveys provide limited evidence of ECD effect

It may take repeated or longer training to produce better results

Final Evaluation Report scoring tool is a useful measure of capacity building

Next Steps/Conclusions

Study characteristics of those who attended versus those who did not

Develop measures for webinar training Develop measures for various training

methods and activities Develop measures for capacity building in

general

References

CTPH (California Department of Public Health). 2.2007.Tell Your Story. Guidelines for Preparing a Complete, High Quality Final Evaluation Report. http://www.cdph.ca.gov/programs/tobacco/Documents/CTCPTellYourStory.pdf

Preskill Hallie and Darlene Russ-Eft (2005). Building Evalution Capacity: Thousand Oaks (Sage Publications)

Patton, Michael Quinn. (2005). Teaching Evaluation Using the Case Method. New Directions for Evaluation. Volume 2005, Issue 105, pages 5–14, Spring

Trevisan, Michael S. (2004). Practical Training in Evaluation: A Review of the Literature. American Journal of Evaluation 25;

Torres, Rosalie, Hallie Preskill and Mary Piontek. (1997). Communicating and Reporting Practices and Concerns of Internal and External Evaluators. Evaluation practice, Vol. 18, No. 2, 105-125.