Term Evaluation and Assessment” - EJTN › Documents › Model_Kirkpatrick.pdfTerm Evaluation and Assessment” ... JTM 1st 2016 Seminar C on t e n t s 1. Donald Kirkpatrick 2. Basics

JTM 1st 2016 Seminar

“Measuring Learning Results and Training Effects – The Particular Challenge of Long-

Term Evaluation and Assessment”

March 3th – 4th, 2016Academy of European Law (ERA), TRIER

Judge Dr. Stefan TratzDirector German Judicial Academy


C on t e n t s

1. Donald Kirkpatrick

2. Basics of his model

3. The four levels

1) Level 1 – Reaction

2) Level 2 – Learning

3) Level 3 – Behaviour

4) Level 4 – Results

4. Developed model tools of Level 1


Donald L. Kirkpatrick

Born 15.03.1924; dead 09.05.2014

Economist and professor at theUniversity of Wisconsin

Development of the model in 1959

Publication of his modelin the Journal of American Society of Training Directors

Further development in 1994


C on t e n t s



3. The four levels







B a s i c s :

• Progress: model more than (only) assessment of a simple training

• Flexibility: useful for each training andeach training institution

• Helpful to measure the success of a training(Level 1 – 3) and the results (Level 4)

• Useful to design and improve each training

• Also useful to evaluate the trainers


B a s i c s :

• Model based on four levels:

1. Reaction

2. Learning

3. Behaviour

4. Results

• Each level based on the previous level


B a s i c s :

• Opinion of critics:

model largely atheoretical and descriptively(Kraiger 2003)

no proof of a relationship between thelevels (Delahaye 2000)

model no aid for a decision-making(Jung 1997)


C on t e n t s



3. The four levels







L e v e l 1 :

• Level of the „satisfaction“ of the participants

• Assessment of the awareness about thetraining

• Necessary is the reaction ofthe participants …

to plan and improve a (new) training

to get a feedback of the topics ofthe training


L e v e l 1 :

• Necessary is the reaction ofthe participants …

to have an opinion of the participantsabout the materials, the activities andthe trainers

to get (perhaps) a reaction about theconference site and the accomodation


L e v e l 1 :

• Instruments of evaluation on Level 1:

(1) Surveys with participants

(2) Questionnaires for the participants

(3) Observation of participants during the training, i. e. body language

(4) Verbal conversation with the participants

• Time: directly at the end of the training (in Germany: also during the training)


C on t e n t s



3. The four levels







L e v e l 2 :

• Find out …

if and how the participants are learning

if the training methods can reach theintended message

if the training is adapted to the needs ofthe judges and prosecutors

• Very important for initial training programmes

• Also useful for the further training


L e v e l 2 :

• Instruments of evaluation on Level 2:

Interviews

Tests before and after the training

Observation sheets

• Time: at the end or promtly after the training


C on t e n t s



3. The four levels







L e v e l 3 :

• Assessment of the behaviour of theparticipants at workplace

• Useful for the training institution to find out whether the training works

• Also useful to improve and to adjustthe training programme

• Key-question: Have the training changed the behaviour ofthe participants ?


L e v e l 3 :

• Problem: A new behaviour can be also a result of otherexperiences not only of the training

• Useful tools:

Observation of the former participants

Interviews with the participants

• Time: 3 or 6 months after the training


C on t e n t s



3. The four levels







L e v e l 4 :

• Measuring of the final results of the training

• Outcomes of the training good for the courtsand the prosecutor services

• i. e.:

better behaviour in contact with the parties

better behaviour in contact with thecollegues


L e v e l 4 :

• i. e.:

better communication within the courts andthe prosecutor services

higher quality of the work

quicker judgemets / verdicts

lower costs


L e v e l 4 :

• Tools:

Data analysis

Managment reports

• Time: 6 to 12 months after the training


C on t e n t s



3. The four levels







Developed tools :

• Questionnaires to assess …

the training needs

the training process

the trainers

the organisation

the materials


Developed tools :

• Different questionnaires for trainers, the course directors and the lecturers

• Design of questionnaires:

Mixture of open and closed questions

Open questions (more information): for trainers, speakers and to value theimproving of the knowledge of theparticipants


Developed tools :

• Design of questionnaires:

Anonymous questionaire

Confidentiality

No pressure to fill in

Space for proposals for the future


Developed tools :

• Recommendation:

Use questionnaires with a adequate lenght

Simple and clean language

Avoid multiple questions

Avoid double negatives and hypotheticalquestions

Questions should always be ableto stand alone


Developed tools :

• Recommendation:

No mixture of contentual andorganizational questions

Evaluation and feedback at an appropriate level / higher level

(in Germany: by the director)


Developed tools :

Developed questionaire for Model 1:

EJTN Fragebogen Model Tool 1.doc

EJTN Fragebogen Model Tool 1.doc


Thank you for your kind attention !!