Evaluation Research

Chapter 11

Evaluation researchSometimes called program evaluation, refers to a research purpose rather than a specific method.

Purpose is to evaluate the impact of social interventions such as new teaching methods, innovations in parole, and a host of others.

Evaluation research is a form of applied research• Intended to have real-world effect

Many methods, like surveys and experiments can be used in evaluation research.

In recent years, the field of evaluation research has become an increasingly popular and active research specialty

Evaluation research reflects social scientists’ increasing desire to make a difference

At the same time, there is the influence of an increase in federal requirements that program evaluations must accompany the implementation of new programs, and

Availability of research funds to fulfill those requirements

Topics Appropriate to Evaluation Research Evaluation research is appropriate whenever some social intervention occurs or is planned.

Social intervention is an action taken within a social context for the purpose of producing some intended result.

Evaluation research is the process of determining whether a social intervention has produced the intended result.

The topics appropriate for evaluation research are limitless.

Practical significance: • Jobs• Programs,• Investments • Values and beliefs

Formulating the Problem: Issues of Measurement Problem: What is the purpose of the intervention to be evaluated?

This question often produces vague results. • Revise General Education Program at WU

To conduct evaluation research, must be able to:• Operationalize• Observe• Measure

What is the outcome or the response variable?

If a social program is intended to accomplish something, we must be able to measure that something.

Essential to achieve agreement on definitions in advance.

In some cases you may find that the definitions of a problem and a sufficient solution are defined by law or by agency regulations;• Must be aware of specifications and accommodate them

MeasurementWhatever the agreed-upon definitions, you must also achieve agreement on how the measurements will be made.

There may be several outcome measures, for instance:Surveys of attitudes and behaviorsExisting statisticsUse of other resources

Measuring Experimental Contexts Measuring the dependent variable directly involved in the experimental program is only a beginning.

Often important to measure aspects of context of an experiment researchers think might affect the experiment.

For example, what is happening in society beyond the experimental group, which may affect the experimental group.

Evaluation Research

“Applied sociology” Ideas that should come to mind:• Like inspection of structure• Field testing a product• This is a “high demand” area

Evaluation ResearchHelps us scientifically analyze whether organizations are doing what we intend for them to do.

Motive:

• If you were considering donating to the Red Cross for Hurricane relief, what questions would you have for the Red Cross?

• If you were a CEO, what questions would you have of your employees regarding your company?

• If you were a judge attempting to make your probation department help prevent juvenile delinquency, what questions would you have of probation?

Evaluation ResearchEvaluation research begins with an understanding of Organizations.

There are stakeholders who are invested in the organization because they want to provide something to the world or because they are getting something out of the organization.• Philanthropists• Board Members• Politicians• Contractors• Managers• Employees

They are typically hierarchically arranged and form the support behind an organization or program.

Evaluation ResearchWho is likely to call for an Evaluation?Why?Who is likely to work with (or against) the researcher doing the evaluation?

Stakeholders who control funding are most likely to request evaluations.

Reasons others wouldn’t?Most ideological of supporters and those getting income and prestige from programs are more likely to interfere.

Evaluation Research

Inputs:

The resources, raw materials, clients, and staff that go into a program.

Program Process:

The complete treatment or service delivered by a program.

Outputs:

The services delivered or new products produced by the program process.

Outcomes:

The impact of the program process on the cases processed or the state of the target population or the social conditions that a program is expected to have changed.

Stakeholders are involved in each realm, have diverse interests, and are invested to various degrees.

Evaluation Research

Inputs:The resources, raw materials, clients, and staff that go into a program.

Program Process:The complete treatment or service delivered by a program.

Outputs:The services delivered or new products produced by the program process.

Outcomes:The impact of the program process on the cases processed or the state of the target population or the social conditions that a program is expected to have changed.

Stakeholders are involved in each realm, have diverse interests, and are invested to various degrees.

MSW, assistant, HIV education manuals, videos, schools provide kids

Program provides in-school lectures detailing disease process, risky behaviors, and HIV prevention tactics.

Some number of teens attend programs at some number of schools.

Initial: Kids know more about HIV.

Intermediate: Kids influence others to avoid risky behaviors.

Long-term: Disease-free adults and lower HIV infection in community.

For example: An HIV education program:

Evaluation ResearchUnderstanding the model of organizational production, one can see foci of evaluation studies.

Assessments of: • Needs for the Program• Program Design and Theory• Program Process and Implementation• Program Outcome/Impact• Program Cost and Efficiency

Inputs Program Process Outputs Outcomes

Evaluation ResearchNeeds Assessment Typical Questions:

(“Typical Questions” are from Evaluation, Seventh Edition by Rossi, Lipsey, and Freeman, pp. 77-78):

What are the nature and magnitude of the problem to be addressed?What are the characteristics of the population in need?What are the needs of the population?What services are needed?How much service is needed, over what time period?What service delivery arrangements are needed to provide those services to the population?

Evaluation ResearchApproaches to Needs Assessment:• Determine what “need” is• Allow stakeholders or others to define the problem• Search for evidence of the problem or lack thereof• Commonly demanded of social service agencies, but

also requested by agencies concerned that their clients have unmet needs

• Multiple methods can be employed

Exercise: How would you determine whether there is a need for crime reduction programs in Rock Hill?

Evaluation ResearchOutcomes/Impact Assessment Typical Questions

• Are outcome goals & objectives being achieved?• Do services have beneficial effects on recipients?• Do services have adverse side effects on

recipients?• Are some recipients affected more by services

than others?• Have the services addressed the problem or

situation they are intended to address?

Operationalizing Success/Failure

One of the most challenging aspects of evaluation research is determining whether the program succeeded or failed.

Definitions of “success” and “failure” can be difficult.

The Social Context

Evaluation research has a special propensity for running into problems.

• Logistical problems

• Ethical problems

Logistical Problems Getting subjects to do what they are supposed to doGetting research instruments distributed and returned

The logistical problems of evaluation research grow out of the fact that it occurs within the context of real life.

Ethical IssuesEthics and evaluation are intertwinedMay involve political, ideological and/or ethical issues

Experimental program may be of great value to participants. •What about the control group not

receiving help? •For example, Tuskegee, Alabama study

Tuskegee StudyIn 1932, the U.S. Public Health Service, working with the Tuskegee Institute

Began a study to record the natural history of syphilis in hopes of justifying treatment programs for blacks.

It was called the "Tuskegee Study of Untreated Syphilis in the Negro Male."

Tuskegee Study

The study initially involved 600 black men399 with syphilis201 who did not The study was conducted without patients' informed consent

Tuskegee StudyResearchers told the men they were being treated for "bad blood" A term used to describe several ailments, including:SyphilisAnemiaFatigueIn truth, they did not receive proper treatment to cure their illness.

Tuskegee StudyIn exchange for taking part in the studyMen received: Free medical examsFree mealsBurial insuranceStudy originally projected to last 6 monthsThe study went on for 40 years In July 1972, an Associated Press story about the Tuskegee Study caused a public outcry

Cost-benefit analysis

How much does the program cost in relation to its benefits?• If benefits outweigh cost, keep the program going. • If reversed, ‘junk it’.

Ultimately, the criteria of success and failure are often a matter of agreement. People responsible for the program may commit in advance to a particular outcome as an indication of success. If so, make certain that research design measures the specified outcome.

Use of Research Results Because the purpose of evaluation research is to determine the success or failure of social interventions, you might think it reasonable that a program would automatically be continued or terminated based on the results of the research. It’s not that simple. Other factors intrude on the assessment of evaluation research results, sometimes blatantly and sometimes subtly.

Making ChangesThree reasons why implications of evaluation research results are not always put into practice.

• Implications may not be presented in a way that non-researchers can understand.

• Evaluation results sometimes contradict deeply held beliefs

• Vested interests in the programs underway

Social Indicators Research Combining evaluation research with the analysis of existing data.

Rapidly growing field in social research involves development and monitoring of social indicators, aggregated statistics that reflect the social condition of a society or social subgroup.

Researchers use indicators to monitor social life.

It’s possible to use social indicator data for comparison across groups either at one time or across some period of time.

Doing both may shed the most light on the subject.

Social Indicators

Social Indicators Research The use of social indicators is proceeding on two fronts:

1. Researchers are developing more-refined indicators; finding which indicators of a general variable are the most useful in monitoring social life

2. Research is being devoted to discovering the relationships among variables within whole societies