Problem/Need and Theory of Change/Logic Model

AmeriCorps State and National External Review

Daniel Barutta and Sarah Yue, Program Officers

2

Learning Objectives

¨ Understand how CNCS defines Problem/Need, Theory of change, and Logic Model

¨ Know how to assess Problem/Need, Theory of Change, and Logic Model criteria when reviewing an AmeriCorps application

¨ Practice assessing these criteria with sample application narratives and logic models

Theory of Change

¨ Cause and effect relationship between a community problem or need, a program’s intervention, and the desired outcomes

¨ Supported by data and evidence¨ Theory of Change elements:

Community Problem/Need

Specific Intervention

Intended Outcome

Evidence• Guides choice of intervention

• Supports cause-effect relationship

Statistics documenting the

need

3

Example: Riverton Literacy Corps

Community Problem/Need

Specific Intervention

IntendedOutcome

Evidence: Research on building block skills leading to reading proficiency. Research on design, frequency,

duration of tutoring sessions.

Statistics on the number of students at below grade level in program’s service area. Data explaining why this problem exists and why it is significant.

Children reading below grade level

in 3rd grade

Students are able to read at 3rd grade level (as measured by 3rd grade reading exam)

Individualized tutoring 3 times/week for 20 min on five “building block” literacy skills through reading, writing and verbal communication activities

4

5

Logic Model

¨ A visual way of depicting the program’s Theory of Change The community problem/need that exists Resources available to operate the program Planned activities (interventions) Outcomes the program intends to achieve

¨ Should have a logical “flow” from left to right

¨ Should tell a coherent story about what the program will do and what it plans to accomplish

Reading a Logic Model

6

CNCS Logic Model

¨ Follows a prescribed template:PROBLEM INPUTS ACTIVITIES OUTPUTS OUTCOMES

Short-TermOUTCOMESMedium-Term

OUTCOMESLong-Term

The community problem that the program activities (interventions) are designed to address.

Resources that are necessary to deliver the program activities (interventions), including the number of locations/sites and number/type of AmeriCorps members.

The core activities that define the intervention or program model that members will implement or deliver, including duration, dosage and target population.

Direct products from program activities.

Changes in knowledge, skills, attitudes and opinions. These outcomes, if applicable to the program design, will almost always be measurable during the grant year.

Changes in behavior or action. Depending on program design, these outcomes may or may not be measurable during the grant year.

Changes in condition or status in life. Depending on program design, these outcomes may or may not be measurable during the grant year. Some programs, such as environmental or capacity-building programs, may measures changes in condition over a period as short as one year.

7

8

CNCS Logic Model (continued)

¨ Programs are not required to have all three types of outcomes

¨ Not all outcomes need to be measurable during the grant period or directly connected to performance measures

¨ It’s OK if outcomes are in the “wrong” boxes as long as the elements are logically aligned

¨ One type of outcome is not "better" than other types (i.e., long term outcomes are not necessarily more valuable than short-term ones, and changes in condition are not inherently better than changes in behavior)

9

Expectations for High-Quality Applications

¨ Strong Theory of Change with logical alignment between problem/need, intervention, and outcome

¨ Clearly articulated problem/need backed up by specific, relevant, current data

¨ High-quality intervention supported by solid evidence showing that it is likely to achieve meaningful outcomes

¨ AmeriCorps member service that provides a significant "value-add" in addressing the problem/need

10

Review Criteria for External Review

Problem/Need¨ The applicant clearly describes how the

community problem/need will be addressed by the program.

¨ The applicant clearly describes how the community problem/need is prevalent and severe in communities where members will serve and the need has been well documented with relevant data.

Review Criteria for External Review (cont.)

Theory of Change and Logic Model ¨ The applicant clearly describes the proposed

intervention including the roles of AmeriCorps members and (if applicable) the roles of leveraged volunteers.

¨ The applicant clearly describes how the intervention is likely to lead to the outcomes identified in the applicant’s theory of change.

¨ The applicant clearly describes how the AmeriCorps members will produce significant and unique contributions to existing efforts to address the stated problem.

¨ All elements of the logic model are logically aligned.

11

Logistics for Review

¨ Sections of the application that should be reviewed to assess the criteria: Problem/Need narrative Theory of Change and Logic Model narrative Logic Model chart (located at end of

application) Evidence Base narrative

¨ You do not need to read or consider the Performance Measures section of the application.

12

13

Logistics for Review (continued)

¨ Level of evidence will be assessed outside of external review, but the Evidence Base section should still be read and considered in addressing the review criteria

¨ 3-page limit for logic model; do not review beyond this point

¨ Information in logic model chart and narrative is complementary; details do not have to be repeated both places, but should not contradict each other

Scoring Rubric

4-point scale*:¨ Exceeds the criterion¨ Meets the criterion¨ Partially meets the criterion¨ Does not meet the criterion* One criterion (“All elements of the logic model are logically aligned”) will be assessed on a three-point scale without an “Exceeds the criterion” option

Refer to the Scoring Rubric document on the Reviewer Resource Webpage to read an overview of each rating.

14

Criterion Descriptions

¨ The Reviewer Resource Webpage contains a chart that lists specific descriptions of the “Exceeds the criterion” and “Does not meet the criterion” ratings for each individual review criterion

¨ The “Meets the criterion” and “Partially meets the criterion” ratings will fall between these two extremes

¨ Please use both reference documents (the Scoring Rubric and the Criterion Descriptions) throughout your review process 15

16

Practice: Sample Application

¨ Open the Sample Application (located on the Reviewer Resource Webpage) Read the Problem/Need narrative, Theory of

Change and Logic Model narrative, Evidence Base narrative, and Logic Model chart

Using the Scoring Rubric and Criterion Descriptions, score the Problem/Need and Theory of Change and Logic Model criteria for the Sample Application (you can record your scores on a blank IRF)

Write down your justification for each score Pause the training until you have finished

17

Practice: Sample Application

¨ The following slides provide CNCS’s assessment of what the score should be for each criterion, plus written justifications for each score.

¨ The justifications are presented in the style of appropriate reviewer comments, but reviewers are NOT required to provide this many comments. Reviewers should limit their comments to significant strengths and weaknesses only.

¨ It’s OK for your scores to differ slightly from CNCS’s scores. If your scores differ significantly, please re-read the Scoring Rubric and Criterion Descriptions and re-review the application.

Problem/Need – Criterion 1

¨ Criterion: The applicant clearly describes how the community problem/need will be addressed by the program.

¨ Score: Meets the criterion¨ Justification:

Strengths: The applicant clearly defines the two needs in the target

community that the program will address: environmental restoration and improved academic performance in science for economically disadvantaged youth.

The proposed member activities (planning and conducting environmental restoration projects, and developing and conducting environmental education service learning projects for sixth and seventh grade students) are logically connected to the need defined by the applicant.

18

Problem/Need – Criterion 1

¨ Justification (continued): Strengths (continued):

The proposed intervention is well suited to address the causes of declining environmental quality – reduced funding to support land maintenance and a lack of knowledge of how to increase and maintain volunteers.

Weaknesses: It is unclear whether the environmental

education service learning projects will address the cause(s) of low academic performance in science among economically disadvantaged youth.

19

Problem/Need – Criterion 2

¨ Criterion: The applicant clearly describes how the community need/problem is prevalent and severe in communities where members will serve and the need has been well documented with relevant data.

¨ Score: Meets the criterion¨ Justification:

Strengths: The applicant cites specific, relevant, and

reasonably current (within 6 years) data to substantiate the environmental challenges in the target region including invasive species, erosion of stream banks, and increased chemical load in the water system.

20

Problem/Need – Criterion 2

¨ Justification (continued): Strengths (continued):

The applicant provides subsidized lunch data indicating a significant level of economic disadvantage in the target schools. The applicant also cites convincing statistics indicating that economically disadvantaged students in the state have low achievement scores in science.

Weaknesses: The academic achievement data cited by the

applicant is statewide rather than specific to the target counties or schools.

21

Theory of Change and Logic Model – Criterion 1

¨ Criterion: The applicant clearly describes the proposed intervention including the roles of AmeriCorps members and (if applicable) the roles of leveraged volunteers.

¨ Score: Meets the criterion¨ Justification:

Strengths: The application clearly describes the

proposed activities of the AmeriCorps members and how they fit into the overall program design and goals.

The applicant explains how volunteers will be leveraged by the program to contribute to the desired outcomes.

22

Theory of Change and Logic Model – Criterion 1

¨ Justification (continued): Weaknesses:

It is not clear to what extent the proposed outcome targets will be accomplished by members vs. by leveraged volunteers. [Contextual note: either way is fine as long as it is clearly explained in the application.]

23

24

Theory of Change and Logic Model – Criterion 2

¨ Criterion: The applicant clearly describes how the intervention is likely to lead to the outcomes identified in the applicant’s theory of change.

¨ Score: Meets the criterion¨ Justification:

Strengths: The proposed member activities are logically

connected to the desired outcomes of improved environmental quality and increased student academic achievement in science.

Previous performance measure data indicates that the program has been effective in improving environmental quality in the target region.

Theory of Change and Logic Model – Criterion 2

¨ Justification (continued): Strengths (continued):

The applicant cites a recent, relevant study on a similar student demographic that demonstrates that high-quality service learning activities resulted in increased student academic achievement. The applicant has modeled their own program design after this study, with additional thoughtful modifications that are responsive to requests from the local schools.

Weaknesses: Details are lacking regarding what the service-

learning projects will be or how they will be implemented, making it hard to assess whether the program's effectiveness is likely to match that of the study results. 25

Theory of Change and Logic Model – Criterion 2

¨ Justification (continued): Weaknesses (continued):

The applicant does not cite sufficient evidence that the program activities will lead to the proposed mid- and long-term outcomes (volunteers increasing their understanding of the importance of conservation, and restoration and conservation remaining sustainable).

[Contextual note: the fact that the evidence cited by the program was non-experimental will be considered as part of the Level of Evidence determination, which is outside of the scope of external review. It should not be assessed here.]

26

Theory of Change and Logic Model – Criterion 3

¨ Criterion: The applicant clearly describes how the AmeriCorps members will produce significant and unique contributions to existing efforts to address the stated problem.

¨ Score: Meets the criterion¨ Justification:

Strengths: AmeriCorps members will be providing environmental

restoration services that state and local governments and nonprofits lack the resources to accomplish. Members will also be building the capacity of local organizations to recruit and support volunteers to help meet these needs.

The service-learning activities developed by AmeriCorps members will be new to the target schools and will enhance the existing curriculum.

27

Theory of Change and Logic Model – Criterion 3

¨ Justification (continued): Weaknesses:

The applicant does not describe in sufficient detail how the AmeriCorps members' roles will differ from those of existing staff at the public and private parks where they will serve.

28

Theory of Change and Logic Model – Criterion 4

¨ Criterion: All elements of the logic model are logically aligned. ¨ Score: Partially meets the criterion [Note: this criterion is

scored on a three-point scale.]¨ Justification:

Strengths: The needs, interventions, outputs, and outcomes in the

logic model are directly connected to each other, and the "flow" of the logic model generally makes sense.

Weaknesses: The logic model is lacking some important details, such as the

inputs for two of the three rows, which negatively affect its alignment. Some of the descriptions provided in the boxes, such as the "Activities" description in the third row, are not sufficiently detailed, and the application narrative does sufficiently compensate for these missing details.

29

Theory of Change and Logic Model – Criterion 4

¨ Justification (continued): Weaknesses (continued):

The row in the logic model about preparing members for careers in environmental stewardship is not directly connected to the program's theory of change as described in the narrative. [Contextual note: some programs do include member development as a primary goal of the program, particular programs that recruit disconnected youth as members. However, this particular application does not appear to be a program of that type.]

30

Theory of Change and Logic Model – Criterion 4

¨ Justification (continued): [Contextual note: it is OK that the program

listed changes in condition – such as improvement of trails and parkland - as short-term outcomes, because this is consistent with the program's theory of change. It is also OK that the program doesn't list all three types of outcomes for all of their proposed interventions; programs were not required to have all three kinds of outcomes.]

31

Recap: Learning Objectives

Understand how CNCS defines Problem/Need, Theory of Change, and Logic Model

Know how to assess Problem/Need, Theory of Change, and Logic Model criteria when reviewing an AmeriCorps application

Practice assessing these criteria with sample application narratives and logic models

After completing this training, you will take an assessment that will give you more practice in scoring sample application narratives.

32

33

Assessment Questions

¨ To check for understanding and to verify that you have completed this orientation session, please complete the assessment at the following link: https://www.surveymonkey.com/s/NeedToCLogicModel

(Please reference the PowerPoint on the Reviewer Resource Webpage to access the

link)