1

Types of Evaluation

Decide on the Purpose:

Formative - improve and inform

Summative- identify value/effect

2

Summative Evaluation

“A study conducted at the end of a program (or of a phase of the program) to determine the extent to which anticipated outcomes were produced.”

Weiss, Evaluation, 1998

3

Summative Evaluation Summative Evaluation (judgment)(judgment)

Phases/Stages

Full Implementation (Process measured against criteria for successful implementation)

Outcome ( 0utcomes measured against criteria for success)

Program or product must be in a state of completion or at an interim ending point. If it is not, then offer formative evaluation.

4

Classify evaluation as….Classify evaluation as….

Formative: using evaluation to improve or inform the development of a program

Summative: using evaluation to judge the merit of a program

5

Types of Evaluation

Formative Compares status/performance with developer’s hopes. Purpose is correction/

modification, information for decision making (e.g. finding out unknown or testing assumptions)

Summative Establishes performance record)

6

Formative Evaluation is the systematic gathering and analysis of information used to advise and improve the conceptualization, design, production, and/or implementation of a wide array of products and programs. Formative evaluation takes place before a product or program has been finalized and is premised upon the considerations, patterns of usage, reactions, and comments of intended learners, viewers, listeners, and/or end-users. The purpose of the evaluation is to inform pending decisions specific to the project in order to increase the likelihood that desired outcomes will be reached.

(Holland, 2003, Oryx Dictionary of Education)

7

What is the Purpose of Formative Evaluation?What is the Purpose of Formative Evaluation?(Flagg, 1990) (Flagg, 1990)

The purpose is to delineate, obtain, and provide information for making educational decisions ( p. 4)

The consistent goal is to maximize the program’s potential for success (p. 34)

8

Practical- concrete questions to inform specific decisions.

Checks on Developers’ AssumptionsUncover team’s preconceptions about visitor and their interaction with product

Distinguishing Characteristics of Formative Evaluation (Fisch & Bernstein)

9

Types of Evaluation

Decide on the Purpose:

Formative - improve and inform

Summative- identify value/effect

What Types of Evaluation are available?

10

What Types of Evaluations are Available? • Heuristic Evaluations*

•Expert Evaluations*

•Needs Assessments*

•Formative Evaluation of concepts*/product

•Pilot study

•Implementation/Process Assessments

•Outcome Evaluation

11

Formative Evaluation Formative Evaluation (informs decisions)(informs decisions)

Phases:

Pre-design/Planning/Front-end

Design Phase

Production Phase (product)

Implementation

Program or product is not yet consider to be complete or finished. Focus on interim outcomes.

12

Phases of Formative Evaluation(Flagg, 1990)

Planning Phase (Front-End)Earliest Phase

Working with the Proposal Feasibility of the projectDefines the audienceOutlines the contentStates the goalsIdentifies the mediaWhere and the when it will be used/ experienced

13

Phases of Formative Evaluation(Flagg, 1990)

Planning PhaseEarliest Phase

Front-end analysisNeeds assessmentWho needs what content?What do people already know about the subject?What might people expect to see? learn?

14

Front End Evaluations

(Testing Assumptions by Dierking and Pollock)

•Used to assess plans before any practical design work is carried out

•In museum setting, analysis is visitor based

•In technology setting, analysis is viewer or end-user based

•In educational setting, analysis is learner based.

ASKS: What understandings, feelings, attitudes, expectations and knowledge does the visitor, learner bring to the project?

15

Front End Evaluations(Testing Assumptions by Dierking and Pollock)

• Make assumptions explicit ( DIA)

• Identify the ties to visitors everyday life- “hook”

•Weigh missions and goals vs interest of staff and resources

16

Phases of Formative Evaluation(Flagg, 1990)

Design Phase (no programming or filming has been done yet)

Decisions are made regarding:specific contentobjectives

instructional strategies

17

Phases of Formative Evaluation(Flagg, 1990)

Design Phase (no programming or filming has been done yet)

Working with :Writers’ Notebook (TV)outlines goals, content, audience characteristics and gives examples of treatments

Story Boards& Flowcharts (Computer)script with pictures or descriptions of the screens

18

Phases of Formative Evaluation(Flagg, 1990)

Production Phase ( the camera crew film and computer programmers write codes)

Decisions are made regarding issues

that effect:appealcomprehensibilityuser friendlinesspacingicons

19

Phases of Formative Evaluation(Flagg, 1990)

Production Phase ( the camera crew film and computer

programmers write codes)

Working with scripts

storyboards, flowcharts,

rough cuts,

prototypes

20

Phases of Formative Evaluation(Flagg, 1990)

Implementation Phase (field testing)

Evaluating program in designated context with appropriate target group

Often done to help the development of support materials like print materials or

to inform designers of changes that should be made in future programs.

21

Formative Evaluation Formative Evaluation (informs decisions)(informs decisions)

Phases:

Pre-design/Planning/Front-end

Design Phase

Production Phase (product)

Implementation

Program or product is not yet consider to be complete or finished. Focus on interim outcomes.

22

Needs AssessmentsUsually inform planning

Very often must consider what’s already available to meet the needs. Can these services be utilized?

Need to know something about the extent of the problem.

Need to know something about the nature of intended audience

23

Evaluating Programs- Various Stages Evaluating Programs- Various Stages and Purposes:and Purposes: Planning

•Assessment of the need for the program within a community (extent of the need)

•Assessment of background perceptions, habits, knowledge, skills, expectations of the intended audience ( nature of the need).

Need to know your audience/environment prior to launching something new or extending or changing something that exits.

24

Two Models of Needs Assessments:

Marketing Model

Gather feedback in order to learn about and to adapt to the needs of their client population.

PURPOSE: TO EXPAND AND EXTEND REACH INTO THE COMMUNITY AND TO BETTER ADDRESS THE NEEDS OF THE COMMUNTIY

25

Two Models of Needs Assessments:

Marketing Model

Product portfolio analysis ( what do people like? What brings them in? What are they interested in knowing about? What do we offer them that works? Doesn’t work?

Product/market expansion analysis ( which demographics are underrepresented? , which demographic groups should be targeted?)

GOAL: BE RESPONSIVE TO THE TARGET AUDIENCE

26

Two Models of Needs Assessments:

Decision-Making Model

Evaluating the options available to the developer in terms of the needs of the audience.

Where is the greatest interest? Appeal? Lack of knowledge etc?

Goal: Information will inform a choice.

27

Criteria of success- those measurable indicators that will signify whether the goals or objectives are being achieved.

(Goals and Objectives are the specific desired outcomes.)