DIALANG Plus

INTUIT Closing conference

30 January 2008

Kevin HainesHans VeenkampLanguage Centre, University of Groningen

Putting the Plus into

The beginning...

- DIALANG is the first major language assessment system to be based on the Council of Europe's "Common European Framework of reference", which has become established throughout Europe as the most widely recognized frame of reference in the field of language learning.

www.dialang.org

- Curriculum: “no immediate connection of piloting with local pedagogy/syllabus” (Alderson 2005: 60).

- Beta version: “accept its limitations in order to appreciate its beauty…?” (KH, Utrecht, January 2006)

Putting the Plus into

Initial evaluation: recommendations

Pre-test training or acclimatization is neededFollow-up is needed to make advice “real” and relevant to contextPractical exercises are needed to deal with the deficiencies defined in the testThe test should take place within the context of a course rather than preceding it – a systematic lead from DIALANG into the tasks on the courseThe link made to the Common European Framework should be consolidated

Putting the Plus into

The first pilot: June 2006

– Group: 18 x 2nd Year Law students

– Purpose: screening for 3rd Year

– Materials: DIALANG + ‘Finland’* materials: scales and self-assessment material

– Exercise: DIALANG Reading and Writing and follow-up group discussion

*Thanks to Ari Huhta of Jyvaskyla University, Finland

Putting the Plus into

The first pilot: Conclusions:

DIALANG technically sound and user-friendly; occasional in-built errors are a minor irritation rather than a major problem

Feedback generally good; students liked reviewing questions; some students would like more detailed explanation of mistakes

Students liked hints and tips for moving to the next level of CEFR

Some students: awareness: confirms information already known

Some students: encouraging advice: practice more, practice often

Follow-up as catalyst for wider discussion of learning and evaluative discussion of course content

Putting the Plus into

Embedding DIALANG in Blackboard

The University of Groningen Nestor matrix

– Make effective link between the interfaces through test instructions in Blackboard

– Create preparation and follow-up procedures

– Create instructions for students AND teachers

– Make link to the test through the Course Organizer matrix

Putting the Plus into

The second pilot: The Set-up

350 students

14 teachers

Non-teaching weeks

Supervision

Putting the Plus into

The second pilot: Preparation

Students

Teachers

Evaluation

Putting the Plus into

The second pilot: The Test

Students

Teachers

Evaluation

Putting the Plus into

Having now supervised my second DIALANG Plus test

group, I must say I'm rather pleased with the way

things are going. It's also clear, however, that, human

nature being what it is, students tend to rely on the

supervisor telling them what to do rather than just

following the written instructions laid down in

DIALANG Plus. Still, as a step-by-step guide to working

with DIALANG, it's very useful.

Weblog Hans Veenkamp, 5th October 2007

Putting the Plus into

We discover that students find it quite complicated to

follow the instructions until they get the idea that

they need to use the Blackboard screen as a backdrop

behind the DIALANG screen. Partly the problems lie in

the connections between our information and partly in

the human tendency not to read instructions.”

Weblog Kevin Haines, 2nd October 2007

Putting the Plus into

It was easy to refer back to the DIALANG Plus pages during the DIALANG test.

Results question 8

0

20

40

60

80

100

120

140

Frequency

strongly disagree

slightly disagree

slightly agree

strongly agree

Putting the Plus into

The second pilot: Follow-up

Students

Teachers

Evaluation

Putting the Plus into

Personally, I think the DIALANG level indication may very well

be right, but despite this, there are some parts on which I

have to disagree with the DIALANG score sheet. One of these

things is that if I were to believe DIALANG, my punctuation

would be consistent. Unfortunately, my punctuation is not

flawless, as I see it. Another point which differs from the

DIALANG outcome is the fact that I hardly ever use a

dictionary but instead try to find the meaning of a word by

looking at its context. Nice as a C1 score might be, there is

still room for improvement. To reach the next level I think I

need to expand my vocabulary, work on my punctuation and

improve my pronunciation.

Putting the Plus into

I was able to keep a good record of the feedback given during the DIALANG Test in order to write my reflective report later.

Results question 9

0

20

40

60

80

100

120

Frequency

strongly disagree

slightly disagree

slightly agree

strongly agree

Putting the Plus into

Recommendations

– Structure DIALANG into the wider learning context to encourage meaningful interpretations:

• Preparation » Test » Follow-up

– Make full use of the formative function of DIALANG to turn feedback into learning goals:

• Encourage students to make their own judgments about the meaning of the DIALANG feedback and advice

• Encourage teachers to engage with students to discuss the relevance of DIALANG feedback in context

– Use DIALANG to increase awareness of the CEFR amongst students and teachers, providing a catalyst for the discussion of language performance

Putting the Plus into

The future

– extend use to a variety of settings, such as• other large-scale courses• individual formative assessments• indicative screening procedures

– make use of an institutional version• will increase test security• will improve reporting• will promote autonomous use

– fine-tune through continual evaluation so that DIALANG Plus remains fit for purpose