SOUTH DAKOTA’SPartnership for Success

EvaluationBrooke Blaalid, MSWEvaluation ManagerB Consulting, LLC

November 21, 2014Chamberlain, SD

B Consulting, LLC 1

Partnership for Success still follows the SPF process.

5 Step planning process known to support positive youth development, reduce risk-taking behaviors, build on assets, and prevent problem behaviors.

B Consulting, LLC 2

Conduct Needs Assessment

Build Capacity

Develop a Comprehensive Strategic Plan

Implement EBPs

Monitor and Evaluate Program Effectiveness

Remember: Inherent in the SPF process is sustainability and cultural competency.

Reasons for You to Do Evaluation

B Consulting, LLC 3

Need to demonstrate effectiveness to funders. Accountability.

Need to comply with agency regulations or requirements.

Need to demonstrate effectiveness to clients/participants.

Need information to make good decisions about future programs/directions.

Need accurate information to tell us how to improve agency practices

and services.

Need to demonstrate effectiveness to self – feel that you have accomplished something.

Purposes of Evaluation

B Consulting, LLC 4

1. Assessment of merit and worth: the development of warranted judgments, at the individual or societal level, of the value of a policy or program.

2. Program and organizational improvements: the effort to use information to directly modify and enhance program operations.

3. Oversight and compliance: the assessment of the extent to which a program follows the directives of statutes, regulations, contracts or other formal expectations.

4. Knowledge development: the discovery or testing of general theories, propositions, and hypotheses in the context of policies and programs.

From Evaluation: an Integrated Approach. Mark, Henry & Julnes, 2000.

Evaluation Types

B Consulting, LLC 5

1. Monitoring is checking records and doing site visits to ensure that the intervention included in the contractual work plan is being implemented with reasonable fidelity. This is DADA’s role and not that of the evaluation team nor the PRCs.

2. Process evaluation documents the program implemented by tracking the input or activities. It defines the independent variable.

3. Outcome evaluation measures the program’s effects that you expect to achieve after the program is implemented.

Program actually implementedInputs

Outcomes – targeted behavior

Common Evaluation Steps1. Definition of the goals. Sometimes referred to as a

hypothesis, evaluation questions or purpose of the evaluation.

2. Design of the evaluation. How will you know if the goals are reached?

3. Instrumentation or method of data collection4. Data collection5. Analysis of the data collected6. Report on the evaluation

B Consulting, LLC 6

Goals and Measureable Objectives Goals describe desired changes in an individual behavior, infrastructure or environment in broad terms.

Example: Increase the effectiveness of prevention in the community by improving coordination and collaboration among providers Example: Reduce the impact of drinking and driving in my community

Objectives are the specific changes desired and are specific, achievable, time limited and measurable. They are often separated into outcome objectives and process objectives.

Process objectives are often stated as activities or effort (input) which are expected to cause the changes stated in the outcome objectives.

The Police Dept. will increase their underage drinking patrols on State Street by 25% starting in January 2015 as indicated by Departmental staffing rosters. The Community Coalition will ….

B Consulting, LLC 7

Goals and Measureable Objectives

Measurable outcome objectives include 4 parts: Change of a specific scope will occur – the incidence of alcohol related arrests will decrease by 5%. Target population – 12-20 year olds in Community, South Dakota Timeframe – between October 1, 2014 and June 30, 2019 Measure or indicator – as indicated by

Percentage of alcohol related arrestsOutcome objectives can be

Long-term – reflect a quantifiable degree of behavior change Intermediate – reflect a quantifiable change in underlying conditions Immediate – reflect a quantifiable change in knowledge, skills and abilities.

B Consulting, LLC 8

SPF PFS Process Evaluation QuestionsHow well was the SPF PFS implemented at the state and tribe/coalition levels? This question will be addressed through CSAP reporting tool data, community survey, NOMs, Community Outcomes, records, participant observations and the Coalitions Capacity Survey.

We will use qualitative and quantitative data relevant to five sub-questions:

Did the implementation of the SPF match the plan

What deviations from the plan occurred

What led to the deviations

What impact did the deviations have on the intervention and evaluation

Who provided what services to whom in what context and at what cost

B Consulting, LLC 9

Primary Evaluation Questions

B Consulting, LLC will conduct the outcome evaluation to answer three primary questions: Was alcohol use and its related problems, especially

those in the NOMs, prevented or reduced? Did SD achieve the outcome objectives? Was prevention capacity and infrastructure at the

state and coalition levels improved?

B Consulting, LLC 10

B Consulting, LLC

11

What are the NOMs and why should we care about them?

NOMs = National Outcome Measures Government Performance and Results Act (GRPRA) is a

Congressional mandate to collect certain data for federally funded programs including the SPF PFS program.

Must use approved NOMs measures for community and program-level surveys.

With approval from CSAP grantees may substitute data available from local sources for specific NOMs.

B Consulting, LLC 12

What are the NOMs and why shouldwe care about them? (continued)

4 NOMs that coalitions must collect • 30 day use• Perceived risk of harm• Binge drinking in the past 30 days• Alcohol related arrests

The following data will be collected via the replacement for the PMRT.) Increased access to services: number of persons served, age, gender, race

and ethnicity. Increased retention of substance abuse programs

• Number of evidence-based programs implemented• Youth exposed to prevention messages.

Technical Assistance One on-site visit from B Consulting, LLC staff during the development of the strategic plan. Shared, regional TA available throughout the 5 years of your cooperative agreement with SD Prevention Program. TA available via telephone and e-mail at all times. Assistance with CSAP Data Reporting Tool Guidance on unique evaluation components for your program (s). Assistance with development of SD-PLI instruments and others as needed.

B Consulting, LLC 13

PFS Requirements• An updated MRT data system will be launched soon but is not yet

available• Community Surveys SAMSHA is requiring a 70% respondent rate.

The State will get technical assistance from the funder to deal with this.

• 5 year Evaluation Plan– Final Evaluation Report

• 6 mandatory outcome measures: – Alcohol and Prescription Drug use: consumption, intervening variables, and

consequences. – Community Outcome Measures are reported yearly on November 1st.

• Coalition Capacity Survey Checklist• South Dakota Participant-Level Survey (SD PLI) for school-based

EBPs.

B Consulting, LLC 14

Evaluation Plans

• 5 year evaluation plan• Must upload evaluation plan and final

evaluation report into the MRT• *New Requirement from Feds: must submit

a final evaluation report*• Coalitions will need to complete 5 year

evaluation plan and final report• Will be developing evaluation plans with

coalitions after the state evaluation plan is approved by Federal Project Officer.

B Consulting, LLC 15

SD-PLI

• SD-PLI will consist of only Pre and Post– NO FOLLOW-UP

• SD-PLI is a requirement by the State of South Dakota, not PFS

• SD-PLI should be web-based• SD-PLI data will need to be collected by local

evaluator.• SD-PLI will be developed by B Consulting, LLC;

however, we will only analyzing the data for over-all state evaluation

• Prescription Drug use questions will be added due to new requirement.

B Consulting, LLC 16

SD-PLI

• State is not requiring that student surveys be matched from pretest to post-test. There are two main reasons for this:– 1) The statistical advantage of using matched

surveys is lost without a comparison group.– 2) You need to consider CONFIDENTIALITY and

HUMAN SUBJECT PROCEDURES AND PROTECTIONS• Do you have any suggestions on how we can

gather previous prescription drug use data?• What data is currently available to use in

regards to prescription drug use?

B Consulting, LLC 17