Lecture 8A

Designing and Conducting Formative Evaluations

English Study ProgramFKIP _ UNSRI

2014

1

The Dick and Carey Model

Assess Needs to identify Goal(s)

Conduct Instructional

Analysis

Analyze Learners and

Contexts

Write Performance Objectives

Revise Instruction

Develop Assessment Instruments

Develop Instructio

nal Strategy

Design and Conduct

Formative Evaluation

of Instruction

Design and Conduct

Summative Evaluation

Develop and Select

Instructional Materials

2

ObjectivesObjectives

Describe the various stages of formative evaluation .

Describe the instruments used in a formative evaluation.

3

Concepts Concepts

Formative evaluation is:

the process designers use to obtain data that can be used to revise their instruction to make it more efficient and effective.

The emphasis in formative evaluation is:

on the collection and analysis of data and the revision of the instruction.

4

Three basic phases of formative Three basic phases of formative evaluationevaluation

One-to-one, or clinical evaluationA small-group evaluationA field trial

5

Role of subject-matter,Role of subject-matter, learning,learning, and and learner specialists in formative evaluationlearner specialists in formative evaluation

SME: comment in the accuracy and currency of the instruction.

A specialist in the type of learning outcome involved: might be able to critique your instructional strategy related to what is known about enhancing that particular type of learning.

It is also helpful to share the first draft of the instruction with a person who is familiar with the target population.

6

One-to-one evaluation with learners

The purpose of the first stage of formative evaluation,the one-to-one stage,is to identify and remove the most obvious errors in the instruction,and to obtain initial performance indications and reactions to the content by learners.

Criteria : clarity , impact , feasibility.

7

One-to-one evaluation with learners

Selecting learners

During this stage, the designer

works individually with three or

more learners who are

representative of the target

population. Data collection

8

One-to-one evaluation with learners

Procedures The typical procedure in a one-to-one evaluation is

to explain to the learner that a new set of instructional materials has been designed and that you would like his or her reaction to them.

The first critical hallmark of the one-to-one formative evaluation is that it is almost totally dependent on the ability of the designer to establish rapport with the learner and then to interact effectively.

The second critical hallmark of the one-to-one approach is that it is an interactive process.

9

One-to-one evaluation with learners

Assessments and questionnairesLearning timeData interpretationOutcomes

10

Small-group evaluation

Purposes for the small-group evaluation: The first is to determine the effectiveness

of changes made following the one-to-one evaluation and to identify any remaining learning problems that learners may have.

The second purpose is to determine whether learners can use the instruction without interacting with the instructor.

11

Small-group evaluation

Criteria and data

Typical measures used to evaluate instructional effectiveness include learner performance scores on pretests and posttests.

12

Small-group evaluation Selecting learners For the small-group evaluation, you should select

a group of approximately eight to twenty learners. Procedures The evaluator begins by explaining that the

materials are in the formative stage of development and it is necessary to obtain feedback on how they may be improved. having said this, the instructor then administers the materials in the manner in which they are intended to be used when they are in final form. If a pretest is to be used, then it should be given first.

13

Small-group evaluation

Assessment and questionnaires

Data summary and analysisOutcomes

14

Field trialPurposes: One purpose is to determine

whether the changes in the instruction made after the small-group stage were effective.

Another purpose is to see whether the instruction can be used in the context for which it was intended .

15

Field trialLocation of evaluation In picking the site for a field

evaluation, you are likely to encounter one of two situations.

Criteria and dataSelecting learners You should identify a group of about

thirty individuals to participate in your field trial.

16

Field trial Procedure for conducting field

trial. It is similar to that for the small group, with only a few exceptions.

The primary change is in the role of the designer.

The only other change might be a reduction in testing.

17

Field trialData summary and

interpretation

Data summary and analysis procedures are the same for the small group and field trials.

Outcomes18

Formative evaluation in the performance context

The process we will describe next for doing formative evaluation in the performance context could be used after any of the three phases of learning-context formative evaluation.

The purpose of the in-context formative evaluation is to determine fundamentally three things.

19

Formative evaluation in the performance context

Selecting respondents Procedure At the completion of the formative evaluation,

the learners should be told that they will be contacted sometime in the future to discuss the instruction they have just completed and its usefulness. Then, when sufficient time has passed to permit the skills to be used-the learners should be contacted.

Outcomes

20

Collecting data on reactions to instruction

The purpose for the formative evaluation is to pinpoint specific errors in the materials in order to correct them, the evaluation design needs to yield information about the location of and the reasons for any problems.

Five questions would be appropriate for all materials.(1) Are the materials appropriate for the type of learning

outcome?(2) Do the materials include adequate instruction on the

subordinate skills,and these skills sequenced and clustered logically?

21

Collecting data on reactions to instruction

(3) Are the materials clear and readily understood by representative members of the target group?

(4) What is the motivational value of the materials?

(5) Can the materials be managed efficiently in the manner they are mediated?

22

The types of data you will probably want to collect

Test data collected on entry behaviors tests, pretests, posttests, and performance context.

Comments or notations made by learners to you or marked on the instructional materials about difficulties encountered at particular points in the materials.

Data collected on attitude questionnaires and /or debriefing comments in which learners reveal their overall reactions to the instruction and their perceptions of where difficulties lie with the materials and the instructional procedures in general.

23

The types of datayou will probably want to collect

The time required for learners to complete various components of the instruction.

Reactions of the subject-matter specialist.

Reactions of a manager or supervisor who has observed the learner using the skills in the performance context.

24

Formative evaluation of selected materials

Preparation for the field trial of existing materials should be made as they could be for a field trial of original materials .

An analysis should be made of existing documentation on the development of the materials, the effectiveness of the materials with defined group, and particularly any description of procedures used during field evaluations.

25

Formative evaluation of selected materials

Descriptions of how materials are to be used should be studied , any test instruments that accompany the materials should be examined for their relationship to the performance objectives, and the need for any additional evaluations or attitude questionnaires should be determined.

26

Formative evaluation of instructor-led instruction The purposes of formative evaluations are much

the same as they are for the formative evaluation of independent instructional materials: to determine whether the instruction is effective and decide how to improve it.

Once again, the formative evaluation of an instructional plan most nearly approximates that of the field trial phase for instructional materials.

Very often , the field testing of selected materials and the field testing of instructor-led instruction are interwoven.

27

Concerns influencingformative evaluation

Context concerns:1. To ensure that any technical equipment is

operating effectively.2. It is also important in the early stages of

formative evaluation, especially in the one-to-one trials, that you work with learners in a quiet setting-one in which you can command their full attention.

28

Concerns influencingformative evaluation

Concerns about learners:

1. In the selection of learners for participation in any phase of formative evaluation, avoid depending entirely on the instructor to assess entry knowledge of the learners.

2. Learners who do not have the entry behaviors should also be included in a formative evaluation.

29

Concerns influencingformative evaluation

Concerns about formative evaluation outcomes: Concerns with implementing formative evaluation:1. The first consideration should be to determine

whether any kind of formative evaluation can be conducted before the formal usage of the instruction.

2. If instruction is being used with the target population without the benefit of any formative evaluation, then it is still possible to use that opportunity to gather information that can be used to do revisions of the instruction.

30

Any Questions?

If not, End of Lecture

31