Evaluation Research Applying Research Skills for Everyday Practical Purposes

Evaluation ResearchEvaluation ResearchApplying Research Skills for Applying Research Skills for Everyday Practical PurposesEveryday Practical Purposes

Evaluation ResearchEvaluation Research

WeWe’’re talking re talking ““applied researchapplied research”” now. now. Ideas that should come to mind:Ideas that should come to mind:

– Applied is kind of like engineeringApplied is kind of like engineering– Evaluation is like inspection of structureEvaluation is like inspection of structure– Field testing a productField testing a product– This is a This is a ““high demandhigh demand”” area area

Evaluation ResearchEvaluation Research Evaluation research is appealing because it helps Evaluation research is appealing because it helps

us scientifically analyze whether organizations us scientifically analyze whether organizations are doing what they intend to do.are doing what they intend to do.

Motive:Motive:

– If you were considering donating to the Red Cross for If you were considering donating to the Red Cross for Hurricane relief, what questions would you have for the Hurricane relief, what questions would you have for the Red Cross?Red Cross?

– If you were a CEO, what questions would you have of If you were a CEO, what questions would you have of your employees regarding your company?your employees regarding your company?

– If you were a judge attempting to make your probation If you were a judge attempting to make your probation department help prevent juvenile delinquency, what department help prevent juvenile delinquency, what questions would you have of probation?questions would you have of probation?

Evaluation ResearchEvaluation Research Evaluation research begins with an understanding Evaluation research begins with an understanding

of Organizations.of Organizations. There are stakeholders who are invested in the There are stakeholders who are invested in the

organization because they want to provide organization because they want to provide something to the world or because they are something to the world or because they are getting something out of the organization.getting something out of the organization.– PhilanthropistsPhilanthropists– Board MembersBoard Members– PoliticiansPoliticians– ContractorsContractors– ManagersManagers– EmployeesEmployees

They are typically hierarchically arranged and form They are typically hierarchically arranged and form the support behind an organization or program.the support behind an organization or program.

Evaluation ResearchEvaluation Research So who is likely to call for an Evaluation?So who is likely to call for an Evaluation? Why?Why? Who is likely to work with (or against) the researcher doing Who is likely to work with (or against) the researcher doing

the evaluation?the evaluation?

Stakeholders who control funding are most likely to request Stakeholders who control funding are most likely to request evaluations. evaluations. – Reasons others wouldnReasons others wouldn’’t?t?

The most ideological of supporters and those getting The most ideological of supporters and those getting income and prestige from programs are more likely to income and prestige from programs are more likely to interfere.interfere.

An evaluability assessment can help determine whether a An evaluability assessment can help determine whether a proper evaluation can even be conducted.proper evaluation can even be conducted.


Inputs:

The resources, raw materials, clients, and staff that go into a program.

Program Process:

The complete treatment or service delivered by a program.

Outputs:

The services delivered or new products produced by the program process.

Outcomes:

The impact of the program process on the cases processed or the state of the target population or the social conditions that a program is expected to have changed.

Stakeholders are involved in each realm, have diverse interests, and are invested to various degrees.


Inputs:

The resources, raw materials, clients, and staff that go into a program.

Program Process:

The complete treatment or service delivered by a program.

Outputs:

The services delivered or new products produced by the program process.

Outcomes:

The impact of the program process on the cases processed or the state of the target population or the social conditions that a program is expected to have changed.

Stakeholders are involved in each realm, have diverse interests, and are invested to various degrees.

MSW, assistant, HIV education manuals, videos, schools provide kids

Program provides in-school lectures detailing disease process, risky behaviors, and HIV prevention tactics.

Some number of teens attend programs at some number of schools.

Initial: Kids know more about HIV.

Intermediate: Kids influence others to avoid risky behaviors.

Long-term: Disease-free adults and lower HIV infection in community.

For example: An HIV education program:

Evaluation ResearchEvaluation Research Understanding the model of organizational production, one Understanding the model of organizational production, one

can see foci of evaluation studies.can see foci of evaluation studies.

Assessments of: Assessments of: – Needs for the ProgramNeeds for the Program– Program Design and TheoryProgram Design and Theory– Program Process and ImplementationProgram Process and Implementation– Program Outcome/ImpactProgram Outcome/Impact– Program Cost and EfficiencyProgram Cost and Efficiency

At the end of this lecture, you should see that these form a At the end of this lecture, you should see that these form a hierarchy where proper assessment at the next level hierarchy where proper assessment at the next level depends in part on information provided at the previous depends in part on information provided at the previous level.level.

Inputs Program Process Outputs Outcomes


Okay!


Needs Assessment Typical Questions:Needs Assessment Typical Questions:(( ““Typical QuestionsTypical Questions”” are from are from Evaluation, Seventh Edition Evaluation, Seventh Edition by Rossi, Lipsey, and Freeman, pp. 77-78):by Rossi, Lipsey, and Freeman, pp. 77-78):

What are the nature and magnitude of the What are the nature and magnitude of the problem to be addressed?problem to be addressed?

What are the characteristics of the population What are the characteristics of the population in need?in need?

What are the needs of the population?What are the needs of the population? What services are needed?What services are needed? How much service is needed, over what time How much service is needed, over what time

period?period? What service delivery arrangements are What service delivery arrangements are

needed to provide those services to the needed to provide those services to the population?population?

Evaluation ResearchEvaluation Research Approaches to Needs Assessment:Approaches to Needs Assessment:

– Determine what Determine what ““needneed”” is is– Allow stakeholders or others to define the problemAllow stakeholders or others to define the problem– Search for evidence of the problem or lack thereofSearch for evidence of the problem or lack thereof– Commonly demanded of social service agencies, but Commonly demanded of social service agencies, but

also requested by agencies concerned that their clients also requested by agencies concerned that their clients have unmet needshave unmet needs

– Multiple methods can be employedMultiple methods can be employed

E.g., how would one determine whether there is a E.g., how would one determine whether there is a need for crime reduction programs in San Josneed for crime reduction programs in San Joséé??


Assessment of Program Theory Typical Questions:Assessment of Program Theory Typical Questions: What clientele should be served?What clientele should be served? What services should be provided?What services should be provided? What are the best delivery systems for the What are the best delivery systems for the

services?services? How can the program identify, recruit, and How can the program identify, recruit, and

sustain the intended clientele?sustain the intended clientele? How should the program be organized?How should the program be organized? What resources are necessary and appropriate What resources are necessary and appropriate

for the program?for the program?

Evaluation ResearchEvaluation Research Approaches to Program Theory Assessment:Approaches to Program Theory Assessment:

– Programs have:Programs have: Impact Theory (Causal theory such as Media Campaign Impact Theory (Causal theory such as Media Campaign

HIV Knowledge, Awareness HIV Knowledge, Awareness Healthier Habits Healthier Habits– Articulated Program TheoryArticulated Program Theory– Implicit Program TheoryImplicit Program Theory

Service Utilization Plan: Refers to assumptions about how Service Utilization Plan: Refers to assumptions about how and why intended clients will become engaged in the and why intended clients will become engaged in the program and follow through to the end, experiencing program and follow through to the end, experiencing impact as impact theory suggestsimpact as impact theory suggests

Organizational Plan: Refers to coordination of program Organizational Plan: Refers to coordination of program functions and activities and acquisition of necessary functions and activities and acquisition of necessary human, financial and physical resources to implement human, financial and physical resources to implement plans.plans.

– Evaluator must determine whether these are sensible Evaluator must determine whether these are sensible given Needs, program goals, common sense, prevailing given Needs, program goals, common sense, prevailing social science literature, social science literature,


Approaches to Program Theory Assessment:Approaches to Program Theory Assessment:– Evaluator must determine whether these are Evaluator must determine whether these are

sensible given needs, program goals, local sensible given needs, program goals, local context, common sense, prevailing social context, common sense, prevailing social science literature, best practices, staffing, science literature, best practices, staffing, available resources, etc.available resources, etc.

– Without understanding program theory, one Without understanding program theory, one may do black box evaluation but there will be may do black box evaluation but there will be little understanding as to why a program works little understanding as to why a program works or does not work.or does not work.

– Is failure an implementation failure or a theory Is failure an implementation failure or a theory failure?failure?


Are you getting it???Are you getting it???

Evaluation ResearchEvaluation ResearchAssessment of Program Process Typical Questions:Assessment of Program Process Typical Questions: Are administrative and service objectives being Are administrative and service objectives being

met?met? Are the intended services being delivered to the Are the intended services being delivered to the

intended persons?intended persons? Are there needy persons whom the program is not Are there needy persons whom the program is not

reaching?reaching? Once in service, do sufficient numbers of clients Once in service, do sufficient numbers of clients

complete service?complete service? Are the clients satisfied with the services?Are the clients satisfied with the services? Are administrative, organizational, and personnel Are administrative, organizational, and personnel

functions handled well?functions handled well?When program process is evaluated for the purposes of shaping When program process is evaluated for the purposes of shaping

and refining ongoing program operations, the assessment is and refining ongoing program operations, the assessment is called a called a ““Formative Evaluation.Formative Evaluation.””

Evaluation ResearchEvaluation Research Approaches to Process Evaluation:Approaches to Process Evaluation:

– What an organization is supposed to do and what it actually What an organization is supposed to do and what it actually does are two different things. Process evaluation reveals how does are two different things. Process evaluation reveals how a plan is actually implemented.a plan is actually implemented.

– Is delivery of service adequate, uniform?Is delivery of service adequate, uniform?

– Keeping detailed data on the process allows researchers to say Keeping detailed data on the process allows researchers to say for whom the program is effective, for whom it is not, and for whom the program is effective, for whom it is not, and sometimes why.sometimes why.

– Researchers gather and analyze many indicators such as Researchers gather and analyze many indicators such as program records, participant surveys, community surveys, program records, participant surveys, community surveys, participation intensity, and qualitative observations, participation intensity, and qualitative observations, interviews, and focus groups.interviews, and focus groups.

– Analysis is often statistical and qualitative.Analysis is often statistical and qualitative.


Outcomes/Impact Assessment Typical Questions:Outcomes/Impact Assessment Typical Questions:

Are the outcome goals and objectives being Are the outcome goals and objectives being achieved?achieved?

Do the services have beneficial effects on the Do the services have beneficial effects on the recipients?recipients?

Do the services have adverse side effects on the Do the services have adverse side effects on the recipients?recipients?

Are some recipients affected more by the Are some recipients affected more by the services than others?services than others?

Is the problem or situation the services are Is the problem or situation the services are intended to address made better?intended to address made better?

Evaluation ResearchEvaluation ResearchApproaches to Impact Analysis:Approaches to Impact Analysis: The general approach is to The general approach is to

compare what happens after compare what happens after programs are implemented programs are implemented with what would have with what would have happened without the happened without the programs.programs.

An experimental approach is An experimental approach is generally preferred in impact generally preferred in impact analysis because a program is analysis because a program is conceptualized as a conceptualized as a manipulation of an manipulation of an independent variable that is independent variable that is intendedintended to cause change in a to cause change in a dependent variable. dependent variable. E.g., Program Theory: E.g., Program Theory: Education Education Knowledge Knowledge Less Risky Behavior Less Risky Behavior

(from Evaluation, Seventh Edition by Rossi, Lipsey, and Freeman, p207):


Efficiency Assessment Typical Efficiency Assessment Typical Questions:Questions:

Are resources used efficiently?Are resources used efficiently? Is the cost reasonable in relation to Is the cost reasonable in relation to

the magnitude of the benefits?the magnitude of the benefits? Would alternative approaches yield Would alternative approaches yield

equivalent benefits at less cost?equivalent benefits at less cost?

Evaluation ResearchEvaluation Research Approaches to Efficiency Analysis:Approaches to Efficiency Analysis:

– Cost-benefit analysisCost-benefit analysis: Compares program costs to the economic : Compares program costs to the economic value of program benefits (focus is on ratio of dollar cost to dollar value of program benefits (focus is on ratio of dollar cost to dollar benefit)benefit)

– Cost-effectiveness analysisCost-effectiveness analysis: Compares program costs to actual : Compares program costs to actual program outcomes (focus is on what you got for your money)program outcomes (focus is on what you got for your money)

– What are actual costs (easier to measure) and what is the value in What are actual costs (easier to measure) and what is the value in dollars of the tangible and intangible outcomes and impact?dollars of the tangible and intangible outcomes and impact?

– Many assumptions are made, so they must be specified. In addition, Many assumptions are made, so they must be specified. In addition, good reports alter assumptions to see how sensitive determinations good reports alter assumptions to see how sensitive determinations are to alterations.are to alterations.

– Given the vagaries of social outcomes (e.g., it is hard to put a dollar Given the vagaries of social outcomes (e.g., it is hard to put a dollar value on keeping a mentally handicapped child happy), social value on keeping a mentally handicapped child happy), social service evaluations often concentrate on cost-effectiveness analysesservice evaluations often concentrate on cost-effectiveness analyses

Evaluation ResearchEvaluation Research So do you want to be one of these folks????So do you want to be one of these folks???? Institute of Organizational and Program Evaluation Institute of Organizational and Program Evaluation

ResearchResearch

The Institute of Organizational and Program Evaluation Research The Institute of Organizational and Program Evaluation Research (IOPER) is an organized research unit of the School of Behavioral (IOPER) is an organized research unit of the School of Behavioral and Organizational Sciences (SBOS) at Claremont Graduate and Organizational Sciences (SBOS) at Claremont Graduate University.University.

The mission of IOPER is to provide services and to conduct The mission of IOPER is to provide services and to conduct research to improve the effectiveness of a wide range of research to improve the effectiveness of a wide range of programs and organizations. Using programs and organizations. Using ““state of the artstate of the art”” scientific scientific knowledge and methodologies, IOPER has provided applied knowledge and methodologies, IOPER has provided applied research, evaluation, and organizational consulting services to research, evaluation, and organizational consulting services to more than 100 different organizations in the past decade.more than 100 different organizations in the past decade.

Documents

Evaluation Research Applying Research Skills for Everyday Practical Purposes