Interactive Power PointEvaluating Your Program

Evaluating your ProgramFirst – Review the Steps Step 1

◦ State overall objectives

Step 2◦ State desired outcomes of this program

Long term outcomes Short term outcomes

Step 3◦ Identify outcome variables◦ Operationalize variables by developing indicators

Step 4◦ Draw sample

Step 5◦ Select data collection technique

Step 6◦ Collect and input data

Step 7◦ Analyze data

Step 8◦ Write up the report

Review Steps in More DetailStep 1

◦ State overall objectives Refer to your mission statement

Step 2

◦ State desired outcomes of this program

Long term outcomes◦ The long term outcomes must be selected before you actually

implement your program◦ Look ahead and make certain your stated outcomes are measurable.

Short term outcomes◦ Identify the outputs and short-term outcomes of your program. These

should logically lead to your long-term outcomes.

Step 3

◦Identify concepts that vary which can be used to measure your outcomes

◦Operationalize variables by developing indicators

First define your concepts Second identify indicators that can be used to

“indicate” or measure your concepts.

Step 4

◦ Draw sample

If your program is small, you could select the entire population.

If you want to generalize from your sample to your population, you must

◦ Draw a random sample◦ Draw a sample of sufficient size

The question you want to answer determines the type of sample you will use.

◦ Does this program tend to work? – needs random sample◦ Can this program work? – doesn’t need a random sample.

Step 5

◦Select data collection technique

The type of data collection technique you select depends on

◦What you want to accomplish◦What resources you have available◦The characteristics of your program◦The characteristics of your participants

Step 5

◦Select data collection technique

◦What you want to accomplish

If you simply want to know if this your program “works” (meeting its stated outcomes), you would most likely use the Field Trial/experiment and utilize a survey/questionnaire to measure your outcomes before and after participation. But there are many other types of data collection techniques. To review these, please click on the data collection button on this website.

Step 5

◦Select data collection technique

◦What resources you have available

If you don’t have sufficient time and/or money you should consider using data that has been collected by someone else – IF that data is relevant and available.

Surveys/questionnaire generally take less time than most other types of data collection (e.g., content analysis, participant observation, intensive interviews etc.)

Generally, to use field experiments, you will need a control group as well as an experimental group.

Step 5

◦Select data collection technique

Characteristics of your program

◦ If your program does not allow you access to the participants BEFORE their participation, then you cannot do an experiment.

◦ If your program is a “one shot” training session, you are probably limited to a survey asking participants to evaluate the workshop and then reporting your results using descriptive statistics.

◦ If your program targets a very small group of individuals and/or organizations, you will want to use intensive interviews to complete case studies.

Step 5

◦Select data collection technique

The characteristics of your participants

◦ If your participants belong to a “protected” population, you may not be able to use surveys/questionnaires. These include children, mentally ill etc.

◦ If your participants belong to a counter culture (i.e., gangs, the Klan etc.) they may refuse to fill out surveys or do intensive interviews.

◦Your participants must be literate if you use surveys/questionnaires.

Step 6

◦Collect and Input Data

First make certain you complied with IRB regulations

Create a coding schema or codebook.

Input data yourself or (if using an online survey/questionnaire) you can work with Dennis Hinkamp to distribute and input data.

Step 7

◦Analyze Data

Identify a computer program to use (i.e., SAS, SPSS, etc.).

Create templates of the tables/bar charts etc. that you need. Label the variables used in these tables/bar charts etc.

Integrate data into tables/bar charts etc.

Step 8

◦Write up a Report

When writing up a report complete the following:

◦Meet requirements specified by the granting agency, editors of peer reviewed articles, etc.

◦ Identify your audience and write “to” them.

◦Complete your literature review.

◦Outline your entire paper and identify the tables/graphs etc. you will utilize to present your results. Then start writing!!