Practical Ways for Evaluating Programs to Collect
Accountability Information
by
Dr. John G. RichardsonAgricultural Programs Accountability Manager
College of Agriculture and Life SciencesDepartment of Agricultural and Extension Education
Box 7607 North Carolina State UniversityRaleigh, North Carolina 27695
USAEmail: [email protected]
Phone: (919) 515-6080Fax: (919) 513-3935
Program Priorities Focus on Performance
“You will see even more so in this budget than in the past, a focus on those [domestic programs] that are indeed federal priorities
and delivering results”
Those programs that the administration thinks are performing poorly “are more likely to be reduced in this budget,” (Bolton
statements in an interview, 2-3-05)
Joshua Bolten, White House Budget Director
Reported by Alan Fram, The Associated Press, in an article entitled “Domestic Programs
Squeezed”, The News & Observer, Friday, February 4, 2005. p. 4A.
Impacts
It is Critical That We Understand Impacts
Impact = Learning + Behavior Change + Results
Learning – is the extent to which participants change attitudes, improve knowledge, and/or increase skills by participating in an educational program.
Behavior Change – is the application of what was learned in the educational program. It is the adoption of practice based on knowledge gained and skills developed.
Results – are evidence of participation, attitude change, knowledge gained, skill developed, and/or changes in behavior. What happened as a result of the change in behavior.
Impact may be expressed as a change in economic value or efficiency, environmental quality, societal or individual well-being.
Program Measures
Level 7 – End Results [IMPACT]Level 6 – Practice ChangeLevel 5 – KASA (Knowledge,
Attitude, Skill, Aspiration)Level 4 – ReactionsLevel 3 – People InvolvementLevel 2 – ActivitiesLevel 1 – Inputs
Observation
This means of gathering
accountability information is the
deliberate act of looking at some
object, event, group, individual
actions or behavior to obtain
program assessment information.
Interview
The direct contact of an individual
with another for the purpose of
obtaining information.
Testimonial
This program assessment tool is the
act of an individual or group
expressing, through verbal or
written means, their direct reaction
to, or experiences resulting from,
Extension’s educational program
efforts.
Anecdotal Information
Information that may be difficult to
verify or quantify, but can provide
insights into program outcomes and
successes.
Existing Records
(records/data analysis) –
This program assessment method is
the use of information or records
that already exist or can be obtained
with minimal planning.
Content Analysis
A way of assessing effects of change
through analysis of materials, levels of
nutrients, moisture levels, or other
factors through actual measurement of
components.
Questionnaire
A survey instrument is used to
obtain specific reactions or
input from all members of an
audience or a sampling of the
targeted audience.
Unobtrusive Measures
A means of evaluating that can provide
specific information to the evaluator
without any obtrusive presence needed
in order to obtain information.
Determining Program Economic Benefit Values
Numbers Count!“Whenever some quantification is done
—no matter how speculative or limited
—the number tends to get into the public domain and the qualifications tend to get forgotten…The number is the thing.”
(U.S. Congress Report by the Subcommittee on Oversight and Investigation, 1980, 7, 10)
Key Valuing Tips
Must be believable Use common sense Be reasonable and realistic Use credible sources Seek multiple sources of values information (Web, Experts) Ask clientele Use observation skills Anecdotal evidence may be sufficient Use comparative values if needed Historical values or commonly accepted values may be used Extrapolation may be used May use estimates of values lost Shadow pricing may be used (reduced crime rates, education) Think beyond the box Can not readily value everything Beware of unintended messages Some outcomes may take longer than others Jgr 10-04
Determining Extension Program Economic Benefit Value
Value
Added
Savings
Reduced Costs
IncreasedIncome
Increased
Productivity
ExpectedValue
Willingness to Pay
IndirectValues
Non- marketBenefits
How WeAre
Better Off
Multiplier Effect
Alternative Opportunity
Costs of Capital
Valuing
Extension
Programs
URL: http://www.ces.ncsu.edu/AboutCES/Factsheets/EconomicBenefitValues.htm
Reduced costs
This means for estimating program value has been used extensively.
Estimate gains (dollars saved) resulting from participants changing their behaviors/practices as a result of the organization’s educational programs.
An example: the value of soil sampling education can be the dollars saved when application rates are based on analysis recommendations rather than indiscriminately applying higher rates.
Increased income
Extension programs can: help participants gain new skills that enhance their job prospects, lead participants to adopt innovative practices that increase yields, or encourage other decisions that cause participants to increase their
income.
One can estimate income gained as a result of adopting new or alternative enterprises that enable the more efficient use of available resources
The program's value can be the difference in participants' income levels before and after the program.
Increased Savings
There are a number of Extension programs that focus on savings enhancement
The increased savings realized by program participants can be considered the estimated value of the program
Increased Productivity
This valuation method is commonly used in non- agricultural businesses as well as in agriculture.
Productivity is the ratio of output per unit of input
Therefore, practices, processes, or innovations that either increases output or reduces the units of inputs would increase productivity.
Examples:
Farming more acres with the same number of employees Larger or more efficient equipment often increases the productivity of a
farm operation Increased crop yield per acre
The difference between the value gained resulting from adopting a new practice, and the costs required to implement the new practice, would be a good estimate of the value of educational program causing such efficiency gains.
Value added
Value added is the increased value from using a product in a new way or modifying the product in a way that generates more profits for the business.
An example: a farmer uses the grain produced on the farm as feed for hislivestock rather than selling the grain for cash. This may significantly improve the returns per bushel of grain produced.
The value of the program that resulted in this alternative use of the grain can be estimated to be the difference in returns produced by feeding the grain to the livestock compared to the price of the same bushel of grain if sold
Expected values
This method of program valuation, requires one to have knowledge of the end product or result that is expected from the educational program.
For example, the value of a program to help people start home-based business may be the projected income generated by such a new business.
The expected income can be estimated based on the known history of similar ventures.
Alternative Opportunity Cost of Capital
Some Extension programs address alternative allocation of resources within one’s business to increase total return.
The increase in total returns can be used as an estimate of the program value.
For example, a farmer has a choice to
1. invest in machinery which would result in a major
cost savings ,or
2. to invest the same funds in a regular savings acct.
The value of the program to assist the farmer in making this decision could be estimated to be the difference between the net income realized from the cost savings of the machinery investment and the interest earned had the money be invested in the savings account.
Willingness to Pay
The willingness of people to pay for some item or service may be used as an estimate of the economic benefit of providing that product or service.
Consumer’s perception of improved quality of products has been successfully parlayed into additional income through improved product packaging or presentation.
People are often willing to state how much they would be willing to pay for certain improved packaging, conveniences, and services.
Such stated willingness-to-pay could serve as the basis for estimating the value of educational programs exploring the costs and benefits of activities such as improved packaging and other market development, or services such as recreational opportunities (trail development, agri-tourism, or similar recreational ventures).
Multiplier effect
This term is often used in economic development circles.
When new money is introduced into a community as a result of economic activity, it is likely that there will be greater demand for other goods and services in the community.
This may result in more new businesses.
The "multiplier" then, is the number of times that the initial dollar of economic activity causes additional dollars to be generated in the community.
The estimated value of the Extension program that stimulated the initial increase in economic activity could be the total value of additional goods and services being generated.
How we are better off?
The insurance industry continually places values on peoples’ lives and serious injuries.
Changes in habits, lifestyles, practices, infrastructure, and physical and social environment, impacts our well-being.
Removal of hazards, improvements in physical surroundings, and changed behaviors that results in reduced illnesses or increased life expectancy, all add to the value of life.
Adopting practices that result in safer driving, improved eating habits, improved water quality and safety, increased seat belt use, safer pesticide handling, and numerous other behaviors are examples of changes that impact the protection of human life.
The value of such changes in behavior can reasonably be quantified using statistical values ascribed to human life, injury, and health, and thus can be used to estimate the economic value of educational programs.
Non-market benefits (cost effectiveness)
Extension programs produce change in human behavior. Easy to identify Difficult to value due to their non-market nature.
Examples of non-market benefits: (not bought or sold) changes (increases) in a person’s education level, changes in one’s aspirations or attitudes, improvement in one’s quality of life, self-esteem, or major leadership skills and
abilities.
Human development and social change have been studied and values projected for these changes.
These projections can be used to value of such Extension programs. For example, US Census Bureau estimates values of different education levels in
lifetime earnings
EDUCATION PAYS OFF
$0 $1 $2 $3 $4
Less than high school
High school
Some college
Associate's
Bachelor's
Master's
Professional
Doctoral
million million million million
2.5 million
2.1 million
1.6 million
1.5 million
1.2 million
1.0 million
4.4 million
$3.4 million
New data from the Census Bureau shows that people with higher education levels earn more money over a lifetime.
Lifetime earnings estimates(Full-time workers ages 25 to 64)
Source: Census Bureau THE NEWS & OBSERVER
The Associated PressSunday, July 21, 2002
Indirect values Indirect values are difficult to assess.
Indirect benefits are affected by factors beyond the scope the educational program.
Valuation of indirect program results is questionable at best, if attempted by persons not recognized as valuation experts.
Example: the value of refrigeration equipment sold to provide better protection for food in a restaurant
The primary program goal: food safety through proper food preparation and handling practices by food service workers.
Motivation to buy new equipment may be: a result of Food Handlers Extension Certification Program
HOWEVERIn reality, the equipment may have simply been:
A replacement of old equipment, Merely a step toward expanding the restaurant’s business, Or, simply the result of a visit by a very good refrigeration equipment salesperson.
FACTORS TO ANSWER IN PROVIDING PROGRAM
IMPACT/ACCOUNTABILITY INFORMATION TO IDENTIFIED
AUDIENCES
•What?
•Who?
•When?
•How?
Accountability Key
The Key is to provide the right accountability information to the right people at the right time in
the right format