The CAP Self-evaluation Survey

Goal: to improve the assessment (the CAP survey is designed for a formative purpose)

Self-evaluation: procedure• Step 1- organize an introduction session: to explain

the criteria and concept of CAP and specify what kind of evidence is needed in the survey.

• Step 2-fill out the online questionnaire: fill out the CAP online self-evaluation questionnaire individually.

• Step 3- organize group interview(s): depending on your situation, you can organize a group interview (selected students and educators interviewed together) or organize two group interviews in which selected students and educators are interviewed separately.

By organizing an introduction session, you, as CAP organizer, want to ensure that survey participants :

understand the concept of the CAP and clarify the purposes and procedure of the CAP self-evaluation

understand the quality criteria and indicators stated in the survey

list all the assessment methods they will evaluate understand what type of evidence is needed have a clear idea on how to fill out the survey

Why do I need to organize the introduction session?

Who should I invite? Who should I invite? Invite all stakeholders of the to-be-evaluated

programme Participants’ profiles:• Have various positions (students, teachers,

coordinators, external evaluators, etc.)Or• A group of people from the teaching team who

share different visions of the to-be-evaluated programme

Number of people:• 8 - 12 (can be more) per programme

Understanding CAP

• You can use the ppt entitled ‘supportive information for the introduction session’ together with the Word document entitled ‘a guide for conducting self-evaluation survey’ to introduce the concept of CAP and explain the 12 quality criteria.

Make an assessment list (CAP)

The online questionnaire will ask each participant to make a list of assessment methods on which they will self-evaluate. Therefore, in the introduction session, please go through the table (see the table on the next page) to ensure that everyone knows what needs to be filled out in the table.

A description of your CAP

Assessment method What is assessed? Who assesses? When/ how often is an assessment carried

out?

For summative or/and formative purpose (s)?

E.g. knowledge test (MCQ, T/F, etc.), project, observation during

internship,…

What competencies / core task / workflows

E.g. teacher / experts in the field / fellow

students

E.g. at the end of acourse / few times per

course

☐ only scores ☐ give individual/group feedback ☐ neither feedback nor score

☐ only scores ☐ give individual/group feedback ☐ neither feedback nor score

☐ only scores ☐ give individual/group feedback ☐ neither feedback nor score

☐ only scores ☐ give individual/group feedback ☐ neither feedback nor score

☐ only scores ☐ give individual/group feedback ☐ neither feedback nor score

What kind of evidence is needed?

A collection of evidence & examples:

• Participants can refer to the following resources to indicate the quality of the assessment:- Student / course evaluation (written & oral; per thematic block and overall evaluation)- Report of grades / results of external examiners- Internship evaluation- Field experts feedback

• Give a few examples to indicate what is expected and what is not:– Invalid evidence is, for instance, “that is obvious”– Solid evidence: such as “at the beginning of the

course a document was provided illustrating the sections (specify how students will be evaluated against what criteria)”

(see more examples given in another ppt)

• their opinion/ experience can also be indicated as evidence : do colleagues or others share the same opinion or experience?

What kind of evidence is needed?

Step 2: filling out the online questionnaire

Be aware that filling out the online questionnaire takes time: about 1 hour!

You need to give at least 1 week to participants to fill out the online survey.

Ask participants to create login names and passwords.

Send reminder.

Step 2: filling out the online survey

Ask all participants to: Go to:

http://fs10.formsite.com/formulierenITG/form296/index.html

Log in with username and password.Try to respond to all questions and provide

evidence!Check ‘Don’t known' if you really do not know

the answer.Fill out the questionnaire before the given

deadline!

Step 3: Group interview

The goals of the group interview are :

to understand different opinions to add evidence to correct the "errors" for spontaneous improvementThe general goal: consensus on the quality -“what

are the strengths and weaknesses of our assessment”

Group interview preparation

A print out of the results for all participants (only if not too many)

Otherwise just give a short PowerPoint presentation•For each criterion show:

- Average scores of the indicators.- Highest and lowest score / standard deviation.- Summary of evidence.

•Note: can transfer the Likert scale into 1-6 point score!

The following slide is an example that shows what kind of issues the online survey may detect.

TRANSPARENCYassessment criteria and standards are communicated

and understandable

Problems:

• When the assessment, for instance, is report writing, the

assessment criteria used to score students’ submitted report are

only in the mind of the teacher and not explicitly communicated

between teacher and students.

• The key criteria are written down. But how they are used to give

scores are vague and not easy to interpret.

The following slide is an example that shows how you can demonstrate the issues the online survey may detect

in the group interview.

• Competency requirements are well described on paper, but are not reflected in the class. No clear link between competencies and assessment.

• Students do not understand what is expected of them in assignments.• Criteria for presentations unclear.• External assessors do not always fully understand the assessment context.

Transparancy

What to discuss?On the basis of the printout of the survey results, the following two situations may

occur :

Situation 1- There is consensus:

High average score + no low scores (variations)(Almost) everyone finds sufficientORLow average score + no high scores (variations)(Almost) everyone is insufficient

Suggested discussion questions:1. Is it true that this is the strength/ weakness of the CAP ?2. Is all the evidence valid? Are the indicators that have a high score substantiated by

strong evidence?3. When there is little evidence: is there any evidence to substantiate the score?4. For the weak points: what are the opportunities for improvement?

What to discuss?Situation 2- There is no consensus:

• Very different scores• Big difference between highest and lowest scores• Some abnormal scores

Suggested discussion questions:• Questioning people results in different scores. Ask them to provide explanations.

Based on what do you make your judgment? For which part of the CAP?• Are his/her scores supported by valid evidence or is it more “a gut feeling"?• Look for evidence : is the provided evidence different for the high scores and

lower scores ? Are they different because of different perspectives?• Confront each other's opinion: are these new insights?• Summarize the evidence: what are the positive and negative points?• Seek consensus.

Tips for the group interview(s) Make it clear that different opinions are welcome.

Interviewees are welcome to correct/restate their statements of particular indicators.

Explicitly question the people who express their dissent in a contrary opinion.

Avoid going astray (stick to the key topics).

The quality criteria are interrelated. Hence, sometimes certain topics come back again in the discussion. Try to make sure that you are the one who leads the conversation to go through the criteria one by one. Please avoid that the discussions jump back and forth between the criteria.

Discussion often arise naturally, because people disagree.

Some practical tips for data collection

• Determine in advance the time to discuss each quality criterion.

For research purpose:• check the recording equipment• ask permission before recording the group

discussion• send the preparation materials (e.g., the survey

results and discussion points /PowerPoint) to Lai• send the interview report / record to Lai