Formative evaluation: using diverse tools in order to refineand improve instructional procedures in distance training

ABSTRACTABSTRACT

THE THE

COURSECOURSE

THE EVALUATION THE EVALUATION

PROCESSPROCESS

The present study emphasized how useful the formative evaluation can be, once it makes it

possible to correct global errors verify if the instructional strategies are really leading to the

main goal of the training. Besides that, this procedure allows that the high investment in

distance learning lead up to the expected results by the organization.

Another aspect to consider is the correspondence among the technical results and the pilot-

group opinions, suggesting that these data collection techniques are valid and capable of

measuring ergonomic, psychological and pedagogical factors involved in the training.

Future experience on training evaluation should take into account the process described

here, making sure to improve and adapt the open-question-like questionnaire and the Analysis

of the Didactic Material Checklist. Such materials should constantly be updated and revalidated.

DISCUSSION AND RECOMMENDATIONSDISCUSSION AND RECOMMENDATIONS

Abbad, G. & Borges-Andrade, J.E. (2004). Aprendizagem humana em organizações de trabalho.

In: Zanelli, J.C., Borges-Andrade, J.E. e Bastos, A.V. Psicologia, organizações e trabalho no

Brasil. Porto Alegre. ARTMED.

Bloom B. S. et al. (1956) Taxonomy of Educational Objectives: Cognitive Domain. New York. David

McKay

Gagné, R. M., Briggs, L. J. (1974) Principles of Instrucional Design. New York. Holt, Rinehart and

Winston.

This study summarizes the methodology adopted for the

formative evaluation of a distance training program for a

company’s pilot-group of consultants. The instructional materials

available to learners were technically examined by the

researchers. An open-question-like questionnaire was applied to

the participating consultants at a interview after the second

module of the course. A focal group also took place at such

meeting. The learners’ opinions about the training matched the

technical evaluation, corroborating the necessity of changing

several aspects of the course. The use of qualitative techniques in

data gathering caused a valid and precise diagnosis of the

problems in the instructional design of the course offered to the

pilot-group, leading the organization to important decisions about

the training program. Because of the formative evaluation,

technical solutions were possible before offering the course to all

consultants in the company, preventing it from wasting more time

and money.

A brief description:A brief description: Named “Qualification Course for Internal Consultants”, it was a

hybrid training - presential and at distance - offered by the

corporative university of a big finance organization to a pilot-group of

internal consultants. The main goal: elaborating an internal consultant project,

applied to the consultants work reality, with a strategic focus in

technical, human and relational dimensions of consultation. Divided in five modules: the first four were dedicated to relevant

points about organizational environment; the last one focused in

practicing all the knowledge acquired during the course through the

elaboration of a final project. The learning material was available in digital and printed

versions.

REFERENCESREFERENCES

University of Brasília - Brazil

Institute of Psychology

Impacto: Research on Training and Organizations of Work

Authors:

André Wogel Talita Custódio Karen da

Matta

Lidia Parachin Maria Emília Araújo Gardênia

Abbad

INTRODUCTIONINTRODUCTIONSuccessful organizations are those that provide it’s members

with constant acquisition and application of the necessary

knowledge, abilities and attitudes to work (Abbad & Borges-

Andrade, 2004). Distance Training, Development and Education

(TD&E) arise as an interesting alternative for big companies to

supply such educational actions to a large number of employees

scattered throughout the country and beyond.Considering the strategic value of the

continuous learning and of the high

investments afforded by the organization in

TD&E, it has become essential to evaluate

learning systems systematically. One of the

ways to do so consists on gathering data

continuously during the learning system

development process in order to obtain

information that validates the system and/or

indicates improvement needs.The objective of the present work was to use several

measurement and analysis tools on a distance course offered by an

important Brazilian finance institution, comparing the technical judge

analysis view with the results of a data gathering qualitative strategy

focused on the course target-public.

Technical Evaluation: Based on The Analysis of the Didactic

Material Checklist; 38 items about adequacy of instructional

objectives, instructional strategies, evaluation

strategies, planning, sequence of content, and

information sources (34 items);

Based on Gagné’s Instructional Theory

(Gagné, 1974) and Bloom’s Taxonomy

(Bloom, 1956); Going through all the modules and exercises; Taking notes on what should be improved

(storyboard).

The formative evaluation allowed the company to improve several aspects

from the course, based on its main instructional objectives, avoiding waste

of time and money .

PRONEXPRONEX Fubra

Open-question-like Questionnaire: Five questions about:

application of what participants learned; what aspects of the participant’s work

improved because of the training program; participants’ opinion about the

exercises;

Focal Group:

Together with the

application of the

questionnaire;Comparison of the

results from the

three first steps:

The results from the

three steps showed

great correspondence;

Data was matched

and organized.

Reporting results to the Organization of

Work:

Exchange of information about the whole

instructional design process.

Decisions based on the formative

evaluation:

The organization of work decided to

change many aspects of the course based on

the evaluation results.

For further information about the

analyses results, see the panel

Evaluating Reaction to distance

training programs: combining

quantitative and qualitative

methods. what aspects should

be improved in the

training program;

what participants

would like to learn in

the course and

haven’t learnt Content Analysis of

the answers