EARLI 11th Biennial conference 2005 electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project Jean-Luc Gilles, Sylvie-Anne

EARLI 11th Biennial conference 2005

electronic Construction and Quality Control in Standardized Testing:

the e-C&QCST project

Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê

http://www.smart.ulg.ac.be/earli-2005/e-CQCST.ppt

University of Liège - Belgium

EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST projectJean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê 2

I. ContextI. Context

• Many European Countries have to face a growing number of higher education students

• The training professionals and the human resources managers of companies are often constrained to evaluate staff competencies, often acquired outside the schools

• There is a worldwide movement of the introduction of quality management in education and training activities

(Kaufman & al., 1995; Segers & Dochy, 1996; Ramsden, 1991; Nightingale & O’Neil, 1994; Zink & Schmidt, 1995; Castaigne, Gilles & Hansen, 2001; Gilles, 2002; Gilles et Lovinfosse, 2004; Phelps, 2005)


I. ContextI. Context

(Gilles, 2002)

• The quality approach suggested by the SMART team in this context


II. Principals R&D objectives of the e-C&QCST project:II. Principals R&D objectives of the e-C&QCST project:

• To develop a web based platform which propose methods, procedures and tools for the management of standardized tests.

• To set up economic valorizations of the results of our research with supply of services like:– Audit of standardized testing systems

– Training and coaching of the users of the e-C&QCST platform

– Docimological services for the improvement of standardized tests

– Test creation with the platform

• To propose a quality label for standardized testing


III. Method: the CCQCST modelIII. Method: the CCQCST model

1. AnalysisPoints to evaluate (PE), Priorities (P), Categories

of Performances (CP)

Specification table (PE x CP)

First step:

1. ANALYSIS

- list of the Points to Evaluate (PE)

- their Priorities (P)

- and the Categories of Performances (CP) of the learners or trainees

End product :

a test specification table where Points to Evaluate (PE) are crossed with Categories of Performances (CP) binomials (PE x CP).

First step:

1. ANALYSIS

- list of the Points to Evaluate (PE)

- their Priorities (P)

- and the Categories of Performances (CP) of the learners or trainees

End product :

a test specification table where Points to Evaluate (PE) are crossed with Categories of Performances (CP) binomials (PE x CP).



2. DesignModalities of the Questions (MQ),

importance of each MQ

(CP x MQ) (PE x CP x MQ)

selection

Second step: 2. DESIGN

The test developer must design the Modalities of the Questions (MQ) according to the Categories of Performances (CP) previously defined at step 1.

End products :

- Binomials (CP x MQ) deriving from the Categories of Performances (CP) associated with the Modalities of the Questions (MQ)

- Trinomials (PE x CP x MQ) deriving from the binomials (PE x CP), previously defined at step 1, assembled with the binomials (CP x MQ) of this step 2

- Selection of the trinomials which should be linked to the questions of the test

Second step: 2. DESIGN

The test developer must design the Modalities of the Questions (MQ) according to the Categories of Performances (CP) previously defined at step 1.

End products :

- Binomials (CP x MQ) deriving from the Categories of Performances (CP) associated with the Modalities of the Questions (MQ)

- Trinomials (PE x CP x MQ) deriving from the binomials (PE x CP), previously defined at step 1, assembled with the binomials (CP x MQ) of this step 2

- Selection of the trinomials which should be linked to the questions of the test









selection




3. QuestionsSelection or writing of the questions which

are linked to the selected trinomials

(PE x CP x MQ)Item bank

Third step:

3. QUESTIONS

The test developer selects or writes the questions. These questions are linked to the trinomials objects selected for the test at the former step.

End product :

A series of questions related to the content and the performances to be test. These questions are created or selected in a secured item bank database.

Third step:

3. QUESTIONS

The test developer selects or writes the questions. These questions are linked to the trinomials objects selected for the test at the former step.

End product :

A series of questions related to the content and the performances to be test. These questions are created or selected in a secured item bank database.






selection





are linked to the trinomials selected

Item bank

Fourth step:

4. TRAINING

The person in charge of the test can choose to propose trainings on the modalities of the questions previously defined.

End product :

“blank tests” (paper-pencil or web based)

Fourth step:

4. TRAINING

The person in charge of the test can choose to propose trainings on the modalities of the questions previously defined.

End product :

“blank tests” (paper-pencil or web based)

4. Training…on the modalities of the

questions defined

Blank tests (paper or web based)






selection






Item bank

Fifth step:

5. TESTING

The questions created or selected at the third step are put in the standardized test which is proposed to the people who perform the test.

End products :

- online exams (in a secured environment)- questionnaires linked with OMR

Fifth step:

5. TESTING

The questions created or selected at the third step are put in the standardized test which is proposed to the people who perform the test.

End products :

- online exams (in a secured environment)- questionnaires linked with OMR


questions defined


5. TestingThe standardized test is proposed

Online exams, paper tests linked

with OMR






selection






Item bank

Sixth step:

6. CORRECTION

The standardized test developer carries out the correction of the tests.

End products :

Treatments are designed to check the quality of the questions before the final version of the results

Sixth step:

6. CORRECTION

The standardized test developer carries out the correction of the tests.

End products :

Treatments are designed to check the quality of the questions before the final version of the results


questions defined




with OMR

6. CorrectionSet up of the

correction of the test

Quality check and discussion of the results before the final version






selection






Item bank

Seventh step:

7. FEEDBACK

Personalized and tailored feedbacks are proposed on the basis of the final version of the results.

End products :

Personalized and tailored feedbacks with online access.

Seventh step:

7. FEEDBACK

Personalized and tailored feedbacks are proposed on the basis of the final version of the results.

End products :

Personalized and tailored feedbacks with online access.


questions defined




with OMR

7. FeedbackPersonalized and tailored feedbacks

are proposed on the basis of final version

of the results

online access FB









selection






Item bank


questions defined




with OMR



of the results

online access FB




Eight step:

8. REGULATION

the test developer collects and analyses the opinions of the all the people who intervened in the test construction process.

End products :

Advices in order to improve the next standardized test that will be created.

Eight step:

8. REGULATION

the test developer collects and analyses the opinions of the all the people who intervened in the test construction process.

End products :

Advices in order to improve the next standardized test that will be created.

8. Regulationanalysis of the opi-

nions of the people who intervened in the test

Advices for the next test






selection






Item bank


questions defined




with OMR



of the results

online access FB




8. Regulationanalysis of the opi-

nions of the people who intervened in the test

Advices for the next test




V. Companies & organizations collaborating in this projectV. Companies & organizations collaborating in this project

M. Bernard Van ParysM. Hubert de Broqueville

M. Dominique PierretM. Jean-Marie Pairoux

Mme Sandra SchillemansMme Tania DelhaiseM. Pierre Ramelot

M. Yvan LagageMme Maryse RiffontM. Peter Vanderheyden

M. Francis LajoieM. Jean-Pierre Renault

Mme Régine GendarmeMme Catherine DeruisseauMme Rameault

Mme Catherine WarnantM. Jean-Philippe Léonard

M. Alain Kock


VI. First results: VI. First results: Relevance of the CCQCST modelRelevance of the CCQCST model

e-C&QCSTResearch group

Presentation of the 8 steps of the CCQCST model

Requests for immediate docimological

subcontracting Services

(AXA, FOREM, FORTIS, IFA)


• Interviews of tests developers of the ten companies and organizations:

– Description of the actions;

– Collaborators concerned;

– Temporal dimension of the tests construction process;

– Other processes in which tests are inserted;

– Importance of the quality of the standardized tests.

– …

• First results of the interviews and the immediate subcontracting

services:

– The CCQCST model and its procedures are considered as relevant and well

accepted;

– The principal readjustment we have to plan consists of an integration of different

roles of tests developers working in teams.

VI. First results: VI. First results: Relevance of the CCQCST modelRelevance of the CCQCST model


• Two difficulties for the users’ roles definition:– They must be used in very different companies’ or non academic organizations’

contexts;

– They should be assigned by one person in charge of the test (the head of the project) to a series of collaborators or subcontractors.

• We propose seven roles within the e-C&QCST platform:– Role 1: “INSTALLER” installs the software on the server, … ;

– Role 2: “ADMINISTRATOR” defines the different tests supervisors, … ;

– Role 3: “TEST SUPERVISOR” defines and supervises his collaborators & subcontractors for the tasks related to the test construction, … ;

– Role 4: “TEST DEVELOPER” realizes one or more tasks of one or more steps of the CCQCST model, …;

– Role 5: “QUALITY SUPERVISOR” defines and supervises his collaborators or subcontractors in charge of the quality controls, …;

– Role 6: “QUALITY CONTROLLER” realizes one or more tasks related to the quality controls of the productions of the tests developers, …;

– Role 7: “ASSESSEE” performs the standardized test.

VI. First results: VI. First results: Users’ roles within e-C&QCST platformUsers’ roles within e-C&QCST platform


• Being given the success of the subcontracting services requested by four companies, we filed in a FIRST SPIN-OFF project of the DGTRE.

• Our project entitled “Smart Advices For Evaluation Systems (SAFES)” was well ranked by an independent scientific committee so that we are currently, starting a two years feasibility study that will allows us :

– To create a method of audit of standardized testing systems;

– To improve the quality control procedures to be implemented;

– To define a typology of subcontracted services;

– To realize a market study.

• The creation of the spin-off company should coincide with the finalisation of the e‑C&QCST platform.

VI. First results: VI. First results: The SAFES projectThe SAFES project


• General ergonomics:VI. First results: VI. First results: First interfaces of the e-C&QCST platformFirst interfaces of the e-C&QCST platform

Multi lingual menu

Main menu with the 8 steps of the CCQCST model

Information about the assessment which is selected(an assess-ment inclu-des one or more stan-dardised tests)

Assessment selector

Information about the user who is

connected


• Management of the e-C&QCST users by the test supervisor:VI. First results: VI. First results: First interfaces of the e-C&QCST platformFirst interfaces of the e-C&QCST platform


• 1st module “ANALYSIS” - specification table: PE are crossed with CP

VI. First results: VI. First results: First interfaces of the e-C&QCST platformFirst interfaces of the e-C&QCST platform


VII. ConclusionsVII. Conclusions• Docimological framework:

The CCQCST model is relevant and well appreciated in the context of the ten companies and organisations which are our partners in this project

• Ergonomics aspects:The first results show that the ergonomics of the first interfaces facilitate the production of specification tables related to the first step of the standardized tests construction

• Economical valorization framework:The surprising immediate docimological subcontracting services requested by four of the ten companies (also the acceptation of the SAFES project) shows that there are possibilities for economical valorisations of the results of this project.

• Users framework:The seven categories of users’ roles we defined were validated by our partners. This clarification should contribute to highlight the complementarities and the responsibilities of the persons who are involved in standardized test creation.


Thank you for your attention !

For more information concerning the electronic Construction and Quality Control

in Standardized Tests project(e-C&QCST)

do not hesitate to contact me:

[email protected]

http://www.smart.ulg.ac.be/earli-2005/e-CQCST.ppt

Documents

EARLI 11th Biennial conference 2005 electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project Jean-Luc Gilles, Sylvie-Anne