diogene: A Methodology for the Certificationof Education Systems

GIANSALVATORE MECCA, GIUSEPPE PENTASUGLIA, IRINA COVIELLO, ROSSANA PACIELLODipartimento di Matematica e Informatica

Università della BasilicataPotenza – Italy

http://www.db.unibas.it/users/mecca/diogenehttp://www.db.unibas.it/users/mecca/diogene

Diogene Working Report n. 01-2006Revision #75

ABSTRACTThis paper introduces the diogene approach to the certification of a learning system's

products. The methodology developed in the paper allows an education institution to

produce a detailed certification for each student. The certification describes the subjects

and the depth of knowledge that a student has shown to possess during a learning cycle.

The proposed approach has been experimented for several years in the Computer

Science degree at University of Basilicata. These experiences show that it represents a

very promising step towards the certification of quality in learning processes, both of

the traditional and of the e-learning kind.

KEYWORDS: quality assurance; evaluation methodologies; teaching/learning strategies; post-secondary education

1 Introduction and MotivationsThere is a growing interest towards the certification of quality in learning systems. This phenomenon is related to several factors.

On one side, European higher education institutions have undergone a process of transformation, inspired by the attempt to build a “European Higher Education Area” [4]. This process is characterized by the attempt to increase and simplify inter-institution communication and exchange, also involving high schools, vocational training institutes, and enterprises, in order to produce a real mobility of students and graduated people. It can be seen that a key requirement, in this respect, is the possibility to communicate, in a clear and possibly standardized way what has been taught and what has been learned.

On the other side, there is the e-learning wave. To oversimplify, we may say that e-learning is an opportunity, but also a risk, and therefore quality assurance in e-learning processes is very important [14].

In light of these considerations, in recent years European academic institutions have started a number of initiatives, whose goal was that of gaining experience on the possibility of defining methods and guidelines for the introduction of quality assurance practices in University degrees. We may roughly summarize these initiatives as an attempt at importing

1

quality assurance practices defined and widely adopted in the production of goods and services – essentially the family of ISO-9000 norms – to University courses. However, this attempt has not been completely successful, due to some form of “impedance mismatch” between the needs of a learning system and the notion of quality in a productive environment.

This paper introduces a methodology for the certification of quality in a learning system which represents an innovative approach to the problem. The methodology has been developed in the framework of a research project, called diogene, and has been the subject of a three-year experiment in our computer science degree. One of the most relevant features of diogene is that, besides a model to design a course and its contents, and a sequence of steps to follow to produce the certifications, it also provides an executable tool, called diogene Instructional Designer (diogeneID), a screen shot of which is shown in Figure 1; the tool supports all phases of the methodology, from design to certification.

The paper is organized as follows. Section 2 provides an overview of our approach, along with a comparison to other approaches to the certification of education systems. The methodology is then presented in detail in Sections 3 and 5. Section 6 discusses our experiences with the methodology. Based on these experiences, some evaluations are given in Section 6.2.

2

Figure 1: diogene Instructional Designer

2 OverviewThe main goal of diogene is to produce accurate certifications of the skills that students have acquired at the end of a learning cycle. To make this sentence more concrete, in Figure 2 we have reported an example of certification produced by diogeneID. The certification refers to a fictional student that we assume has attended a fictional e-learning course for “Web Developer”, which includes courses on databases, web site development and others.

It is possible to see that the document has three main sections. The first one contains data about the student, the learning institution and the learning pathway. The second section provides a summary of the courses and of the respective grades obtained by the student. The core of the certification, however, is the last section, called “Detailed Contents and Knowledge Levels”. This part has two main features: (a) it is a rather detailed description of the subjects that have been taught in each course and for which the student has shown to possess some knowledge; (b) for each subject, there is a clear annotation of the depth of knowledge achieved by the student, using terms like “superficial knowledge”, “knowledge”, “application”. These are taken from a specific taxonomy that will be introduced in the following sections.

The most interesting feature of such a certification, however, is that it is specifically tailored for each student. In other terms, by looking at the certification document of another hypothetical student from the same course, even though we may assume that both students have been given the same lessons and the same materials, we would most probably find that the second student's certification differs from the one in Figure 2, essentially for two reasons: (a) there might be subjects in one document that do not appear in the other; (b) subjects that appear in both documents might be annotated by different levels of knowledge; these differences would reflect the different performance levels reached by the two students in examinations.

2.1 Principles of diogene

We are now able to introduce the vision that has inspired the overall methodology. Our vision of the certification process is based on three main principles.

• diogene assumes that students enter the learning system with different backgrounds, different skills, different knowledge; in essence, they start from different starting points;

• moreover, diogene postulates that – even though instructors will employ all possible means to reduce such initial differences – students will anyhow step through the learning pathway in different ways, for many reasons, including degree of commitment, study ability, and psychological factors, among which motivation plays a central role;

• based on this, diogene assumes that it is unavoidable that students exit from the learning pathway with different skills and knowledge. diogene formalizes these diversities, and encourages the definition of different target levels (or exit levels), provided that they are all subjected to control and therefore can be properly certified.

The formalization of the existence of different exit levels – which is largely confirmed de facto by any teaching experience – represents one of the main novelties of diogene. To

3

confirm this, we want now to discuss the main differences among this approach and other approaches that have been recently proposed in the literature towards the goal of quality assurance in learning systems.

As a first example, let us consider works on curriculum construction and personalization in e-learning environments (see, for example, [12] [3]). To summarize, handling

4

Figure 2: A Certification Example

University of Bla BlaCertification Report

Data about the Student

First Name: John Last Name: Doe Student ID: 29410Birthdate: 13-07-1971 Place of Birth: New YorkLearning Cycle: Bachelor Degree in Computer Science

Course SummaryCourse: Databases – Grade: 70% Target Level: The student has shown to possess knowledge of the following subjects and to be able to apply them to the solution of standard problems: (.. omissis ..)

Course: Web Development – Grade: 100% Target Level: The student has shown to possess advanced knowledge of the following subjects and to be able to properly apply them to the solution of complex problems: (.. omissis ..)

(.. omissis ..)

Detailed Contents and Knowledge LevelsFollowing is a detailed report of the knowledge and abilities that the student has shown to posses during the learning pathway. Each subject is annotated by a level that indicates the depth of knowledge achieved by the student during the final examinations. (.. omissis ..)

Course: Databases – Grade: 70%

Contents and Depth of Knowledge

Module 1. The Relational ModelUnit 1.1: Tables

• Syntax (Application)• Tuples (Application) • Null Values (Application)• Integrity Constraints (Application)• Trigger (Superficial Knowledge)

Module 2: Physical ModelUnit 2.1: Introduction

• Hierarchies of Memories (Literacy) • Disk Technology (Literacy)

Unit 2.2: Access Structures• Paging Strategy (Literacy) • Storage Strategy (Literacy) • Access Strategy (Literacy)• Indices (Application)• . . .

(.. omissis ..)

heterogeneous levels and/or failures of students is the primary goal of these proposals; to solve Uthis problem, the learning information system employs forms of automatic or semi-automatic reasoning in order to customize the pathway that each student should follow inside the system – in terms both of learning objects and test procedures. However, the underlying assumption is that there is only one exit level – even if with different grades – and the attempt is exactly that of bringing all students to reach such fixed level.

Similarly, it should be apparent how our approach differs from quality assurance approaches inspired by ISO-9000 norms. Roughly speaking, we may say that a quality assurance system based on ISO-9000 [15] [9] assumes a “linear production system”, in which input materials enter with certifiably levels of quality, undergo a number of transformations, and then exit as finished goods. Given this assumption, it should be apparent that it is sufficient to certify all transformation processes to guarantee the quality of finished goods. We may, in fact, call this approach to quality assurance a “process-oriented” one. On the contrary, diogene lays its foundations on the observation that it is not possible to assume such a linear process – input material (student) > transformation (teaching/learning) > finished good (graduated) – in learning environments, and as a consequence it adopts what we might call a “product-oriented” approach to the certification of quality. We want however to stress that these two perspectives are not in contrast with each other, and that diogene can be profitably used in a learning system that adopts ISO-9000-like norms for the certification of its processes.

2.2 Main Features of diogene

We are now able to summarize some of the main features of the diogene methodology. Let us first discuss what diogene is not, in order to prevent possible misunderstandings.

diogene is essentially a methodology – i.e., a set of steps and guidelines – and has a supporting tool to assist users during these steps. The tool, however, is not a learning management system, since it was not conceived to support the teaching activity, neither a tool to assist instructors in developing their teaching materials, like, for example [16] or, among commercial tools, Macromedia Authorware. On the contrary, it is a tool to support the design and production of certifications.

It is also important to emphasize that the proposed methodology is fully independent of the way of teaching: diogene does not impose a particular way of teaching, nor the adoption of specific pedagogical patterns [1] [7] [13]. In this respect, it is significantly different from tools and theories about instructional design [10], whose main goal is to guide instructors in the process of designing their teaching, by selecting contents, teaching sequences, and evaluation procedures, in order to achieve a higher level of learning effectiveness. diogene is not concerned with these aspects: its primary focus is that of making more explicit the outcomes of the learning process, independently of the way in which they are reached. As a consequence, diogene has very high applicability: it can be adopted in academic institutions, vocational training, and is applicable both to traditional teaching and e-learning.

As a final remark, it is important to note that our proposal has a strongly pragmatic and disciplinary nature: its main goal is to suggest a reference framework that can be practically used for the certification of the products of education systems. It does not pretend to formalize any pedagogical or psycho-cognitive theory. It has a disciplinary nature since it

5

stems from experiences related to computer science education. We believe that it could be extended to different disciplines, but this belief must be supported by concrete experiments.

This said, we may now introduce in greater detail the main elements of the methodology.

3 The ModelA learning pathway is described in the methodology in terms of its component courses. A course is modeled as a structure, with target levels.

A structure describes the organization of contents in a course. As it is rather standard nowadays, the overall course is composed of one or more modules, each containing one or more units, each unit one or more subjects. To oversimplify, we may say that, roughly speaking, a unit contains as much content as it may be taught in a single lesson; a module corresponds to a group of lessons on a subject, for which a credit (ECTS) measure it is usually given. Alternatively, we may consider each unit as a learning object, and each module as a group of related learning objects, or as a learning object of larger size itself. Figure 1 shows the organization of a database course in terms of its modules, units and subjects.

A more interesting part of a structure is represented by the target levels for that course. In order to be able to introduce a definition of a target level, let us first introduce a preliminary notion, that of depth of knowledge.

3.1 Depth of Knowledge in diogene

diogene adopts a taxonomy of knowledge levels to measure the depth of knowledge shown by a student about one subject. Such taxonomy has been influenced by works on instructional design [11] [6] on one side, and by other well known taxonomies, primarily Bloom's taxonomy of educational objectives [2], and the ACM/IEEE Computing Curricula depth of knowledge metrics [8].

diogene adapts these taxonomies to make them better suitable in a certification scenario. As a result, we have six different levels, as follows:

• Level 1: Superficial Knowledge: the student knows that the subject exists – possibly for having repeatedly heard about it during the lessons – but it does not know its conceptual articulations, neither it is capable of correctly answering questions about it

• Level 2: Superficial Knowledge with Practical Examples: beside superficial knowledge, the student has also seen a number of applications of the subject to practical cases, so that s/he has a feeling of how the subject may be applied

• Level 3: Literacy: the student knows the subject; s/he knows the constituent concepts and their relationships, and s/he is able to correctly refer them; in other terms, s/he knows the “theory”

• Level 4: Application: besides knowledge, the student is also able to apply the knowledge s/he has about concepts and methods of the subject to the solution of standard problems

• Level 5: Detailed Understanding and Proper Application: the student knows the subject in its depth, and s/he is able to apply her/his knowledge to the solution of medium-complexity problems, with a correct methodological approach

6

• Level 6: Advanced Knowledge: the student masters the subject; s/he is able to apply it to the solution of complex problems, also making choices based on proper criteria.

It is worth noting that the six levels are strictly ordered: each level includes all the previous ones. To simplify their usage, we have assigned a different color to each level, ranging from gray (level 1) to blue (level 6).

We also want to emphasize that – although these levels have proved quite effective in our experiences related to computer science courses – they represent only one example of a taxonomy. The overall methodology is independent of the actual definition of the taxonomy, and therefore allows to change the definition of the levels, for example to make them more suitable to different disciplinary contexts.

3.2 Target Levels

We are now able to introduce the notion of a target level. In brief, a target level corresponds to the description of certifiable performance level for the associated course. More specifically, it contains a subset of subjects taken from the course structure, each of which has an associated depth of knowledge.

One example of a target level is shown in Figure 3. As it is possible to see, such a “base level” contains several modules, units and subjects; however, differently from what happens when working in the structure perspective, in the target level subjects are colored on the basis of the assigned depth of knowledge. A structure can contain one or more target level (usually more than one). The instructor will typically organize target levels in a hierarchy. Higher levels may contain more subjects than lower ones, and may associate higher depth of knowledge with those subjects that appear in both. It is important to note that the total order on knowledge levels induces a partial order on target levels; more specifically, a target level A is said to be included in a target level B if B contains all subjects contained in A with equal or higher depth of knowledge.

From the instructor's viewpoint, to build target levels for a course means to define one or more certifiable profiles that s/he expects students to have acquired after taking the course and passing the final examinations. In this respect, we may say that these represent the different way of “exiting” the course.

On the other side, from the student viewpoint, each level represents a possible target. We assume that the target levels are disclosed to students as early as possible; students will usually pick one of the levels and then aim in her/his study to learn the subjects included in the level with the required depth of knowledge. In this respect, target levels become part of a “learning agreement” among instructor and students.

We assume also that the course instructor conceives examination procedures that are suitable to verify the achievement of the different levels by the candidates; this is clearly a critical point of the overall methodology. We shall come back to this in the following. For now let us emphasize again that diogene does not prescribes how tests should be performed.

7

4 System ArchitecturediogeneID is a desktop application developed in Java. It is based on the Eclipse rich-client platform1, which provides a sophisticated platform that helped to shape the graphical user-interface, and to improve the overall usability.

It is based on a client-server architecture inspired by two main goals. The first one was to decouple data about the certification process from transactional data generated during the learning process. In essence, we wanted the system to be independent of any other learning information system used to collect data about students and course – typically a learning management system – and at the same time to be easily integrated in any existing environment.

The second design goal was to be able to support the production of certificates in different scenarios. In the simplest one, a single instructor adopts the methodology and the tool to certify the outcomes of courses s/he teaches. In a wider scenario, the learning institution defines a certification process for a whole learning pathway – for example a computer science laurea. In this case, it is conceivable that a certifier role is introduced, as a responsible of the production of certificates that include results of different courses held by different instructors. Instructors must feed their structures into a shared repository. Then, the certifier will access the repository and produce the certifications.

1 http://www.eclipse.org

8

Figure 3: A Target Level

The system architecture was designed in order to achieve these goals. A critical design choice was related to selecting a suitable repository to store course structures and certifications. Even though it would have been possible, in principle, do develop a relational database, we saw that both structures and certifications are more appropriately stored as XML documents. As a consequence, we adopted a client-server architecture with an open-source XML repository2 in order to store structures and certifications. The desktop application assumes the availability of the XML repository and uses it to fetch/store course structures and certifications.

It is worth noting that the XML repository does not store transactional data about the learning process. We assume that all data about the teaching activities related to the course are properly recorded in some external learning information system, namely: students

2 We selected Exists – http://www.exists.org

9

Figure 4: An Example of a Student Card

John Doe 21355 06-09-1984 New York Computer Science Laurea University of Bla Bla 23

30

ProceduralProgramming03-04.xml

12-06-2004 B

21

30

Object-OrientedProgramming03-04.xml

10-04-2005 C

. . .

enrolled in the course, dates of tests and examinations, grades and target levels obtained by each student. Whenever it is necessary to produce a certification, data about the student are extracted from the learning information system and are fed to diogeneID under the form of a student card. A student cards is an XML summary written according to a given DTD of student data and of all courses that the student has passed. For each course, it should contain several pieces of information: name of the course, final grade, reference to the course structure (i.e., name of the corresponding document in the XML repository), list of examinations taken by the student, and for each examination an indication of the target level assigned by the course instructor.

One example of student card is shown in Figure 4. We assume that these cards are either produced by some client-application by accessing data in the learning information system back-end database or are edited by hand, with the clear advantage of effectively decoupling the system from the teaching process, and thus reducing its impact on the system organization.

5 Methodological StepsWe are now ready to sketch the main steps of the methodology.

Step 1: Definition of the Course Contents

To start, the course instructor – or the course designer – should organize the course contents in terms of modules, units, and subjects. This step is usually rather straightforward, and can be practically performed with the help of the graphical user interface of the diogeneID tool (as shown in Figure 1). During this step it is not necessary to reason about target levels.

Step 2: Definition of the Target Levels

This is a critical step of the overall process. During this step the instructor/designer defines the target levels, by selecting for each level the included subject, and then by annotating each subject with a proper depth of knowledge. The overall process is greatly simplified by the diogeneID user interface, which allows to drag&drop subjects from the structure into the target levels. Also, the tool helps to identify when a target level is included into another one, in order to guarantee a proper progressiveness.

Before actually defining the levels, an instructor should design them; designing the target levels imposes to look at the course from a completely different perspective, which, in our experience, usually suggests a number of useful reflexions on the contents of the course itself and of the overall learning system. In essence, in this phase the instructor/designer is required to answer the following questions: “what are the acceptable profiles for students after this course?”, “what are the prerequisites for each profile? Are these prerequisites adequately satisfied by target levels of previous courses?” “how does each profile impact on subsequent courses?”.

Once target levels have been developed, the instructor has also gained a deeper understanding about how to organize the lessons. In fact, s/he has a clear indication about the level of depth that should be used when teaching a giving subject: intuitively, such a level should be such as to allow students to reach the highest depth of knowledge required for that subject in the target levels.

10

Step 3: Collecting Process Data

Once the design phase has ended, the course may start. As discussed above, we assume that all data about the teaching activities related to the course are properly recorded in some learning information system. An important requirement is that of annotating into the system all test results with target levels. In essence, we assume that the course instructor, when publishing grades after the final examination, also assigns to each student one of the available target levels. With reference to our example, the database course instructor, based on the various elements of evaluation that s/he possesses, might decide to assign to student John Smith level “A”, and to student John Doe level “B”.

If, how it is typical, a learning management system is used to collect these data, then suitable metadata should be employed for recording what target level has been assigned to a student that has passed the final examination.

Step 4: Production of the Certification

In order to produce certifications, the certifier will access the learning information system to generate student cards; s/he will then load student cards into diogeneID and generate the certifications. Certification can then be stored into the system, or saved as printable documents in order to distribute them.

6 A Concrete ExperienceThe goal of this section is to discuss a number of experiences with the system that have been conducted in the last few years in our computer science degree. As a consequence, while the previous sections aim at introducing diogene as a general certification framework, in this section we will concentrate on a specific instance of the methodology, and will discuss several choices that we have made along the process. Although these choices do not represent general rules enforced by the methodology, we believe that they may help to clarify how the overall can be adopted in a real context.

Starting from year 2003-2004, we have selected a number of courses from our computer science curriculum. The selected courses represent a large subset of what we call the “core” of the curriculum, and are summarized in Table 1.

Course Name ECTS Units Short DescriptionProcedural Programming 9 Foundations of programming and programming language. Variables,

types, instructions, control structures, functions and parameters

Databases 6 The relational model. Relational database systems. Query languages. Database design

Object Oriented Programming I 9 Foundations of object-oriented programming. Class. Object. Method. Object Reference. Information hiding. Exceptions. Principles of object-oriented design

Object Oriented Programming II 9 Advanced object-oriented programming. Hierarchies. Polymorphism. Event-oriented programming. Graphical toolkits. The build process and tools

Web Development 12 Client-side web technologies. Client-server programming. Access to database systems. Transactions. Persistence frameworks. Server-side technologies. Containers. Server pages. Frameworks for web development

Table 1.Courses Involved in the Experiment

11

After a three-year cycle, in 2005-2006 we have been able to produce certifications for a first group of students. In the following sections we summarize some lessons we have learned from this experience.

6.1 Examination Procedures

As we have said, the methodology does not impose a specific way of conducting examinations. Thus, many systems might be conceived. However, we believe that our experience might provide some hints. To simplify, we have experienced two main examination procedures.

The first one – which we might call a posteriori level selection – is very close to the “traditional” way of grading students. In essence, students take one test – or any other form of examination – which covers all subjects that have been taught in the course. Then, for each student the course instructor examines the answers that have been given and tries to assign a target level to the student, possibly adding a new level if the existing ones do not fit the knowledge profile shown by the student in the test. The main advantage of such an approach is that it only requires an evolution of procedures that are rather common; thus, it is less invasive and has a smaller impact on the course organization. However, we have seen that, in practice, this has two serious shortcomings: first, it is often very difficult to assign a proper target level to a student based on her/his answers to a general test; second, as it is well known, students often confuse the goal of learning with that of passing the exam; this process does not help to clarify this difference, since the target level and the exam appear as largely unrelated.

One alternative is represented by what we might call an a priori level selection. In this model, the course instructor designs a number of target levels for the course. Then, he develops several different forms of final examination. One typical choice would be to have three different examinations: “base”, “intermediate”, “advanced”. Each examination would typically allow to achieve one or two target levels. In our implementation, we have typically used seven levels, shown in the following table:

Level Short Description Evaluation CriterionA+ advanced knowledge on the contents of the advanced examination the student has correctly solved all problems

included in the advanced examination

A detailed understanding and proper application on the contents of the advanced examination

the student has solved some of the problems included in the advanced examination or s/he has given partial solutions

B detailed understanding and proper application on the contents of the intermediate examination

the student has correctly solved all problems included in the intermediate examination

C application on the contents of the intermediate examination the student has solved some of the problems included in the intermediate examination or s/he has given partial solutions

D application on the contents of the base examination the student has solved all problems included in the base examination

E knowledge on the contents of the base exam the student has solved some of the problems included in the base examination or s/he has given partial solutions

F not certifiable the solutions given by the student do not allow to properly certify her/his knowledge levels

Table 2: Target Levels Used in our Experience

12

The process will typically proceed as follows:

• the course instructor will make the target levels available to the students• s/he will also present the different exams, and their connection with target levels• students will select one target level, and therefore will concentrate on passing the

examination that gives access to that level• the course instructor will evaluate the examinations and assign the target levels based

on them.

It can be seen that this second system does not have the disadvantages of the previous one. First, based on the rules it is rather straightforward, usually, to assign target levels to students. Second, it strongly improves the communication between student and instructor: students have a clear goal, and may better finalize their study. At the same time, the overall evaluation process is simplified and less subject to controversies.

6.2 Evaluation

As a general impression, we believe that student tended to appreciate the fact that learning objectives were clearly communicated, and that they could focus on a clear target. To make this impression more precise, after producing the first certifications, we gave each student the respective certificate and asked to answer several questions. Answers are summarized in Table 3.

Question Possible Answers Answer DistributionHow do you judge the idea of adopting a certification system ?

very good – good – poor – very poor 63,6% 36.4% 0% 0%

How do you judge the possibility of having a certificate ?

an opportunity – a risk 100% 0% - -

How do you judge the diogene certification system ? very good – good – poor – very poor 18,2% 72,7% 9,1% 0%

Would you use your certification as a presentation in a job interview ?

certainly – maybe yes – maybe not – certainly not

36,4% 54,5% 9,1% 0%

Do you think the certificate should be produced: for all students – on a voluntary basis – for nobody

18,2% 81,8% 0% -

How much do you think your certificate corresponds to what you have learned ?

completely – partially – not much – not at all

45,5% 45,5% 9,1% 0%

How much do you think the subjects listed in your certificate correspond to those you have learned ?

completely – only subjects – only knowledge levels – not at all

45,5% 45,5% 0% 9,1%

How do you grade the diogene certification system ? a mark from 0 to 30 average 27,0/30

Table 3.Evaluation Results

As a first remark, let us note that students reported a good overall impression of the certification system.

More specifically, they considered the idea of adopting a certification system to be excellent; with respect to the adoption of the certification system, students said they would consider the idea of using their certificate in a job interview, even though most of them think that the production of certificates should not be mandatory for all students, but rather limited to voluntary students.

When asked to specifically evaluate the diogene certification system, the answer was quite good. With respect to this, we had the impression that students tended to judge the

13

system as the system has judged them; in fact, generally speaking, students that had obtained very good certifications gave higher grades with respect to those that had reached lower depth of knowledge in their certificates. In a way, these latter students felt that the certificate had underestimated the depth of their knowledge. It is however important to note that basically all students agreed that the subjects listed in the certificate closely reflected those that they had studied and perceived to have learned. This is, in our opinion, quite important, since it confirms that the methodology actually allows to detail the outcome of a learning cycle.

7 Conclusions and Open ProblemsWe believe diogene represents a novel approach to the certification of education systems. We also believe that it nicely fits into the common framework for the communication and exchange among education institutions that European governments envisioned in the so-called Bologna declaration [4]. The main goal of the “Bologna process” is the standardization of higher-education learning cycles, in order to facilitate exchange of persons and mobility. It can be seen that a critical requirement, in this respect, is the adoption of tools “to co-operate in quality assurance; to design scenarios for mutual acceptance of evaluation and accreditation/certification mechanisms; to collaborate in establishing a common framework of reference.” (Prague Communication, 2001).

Based on these ideas, several practical tools have been introduced. Among these, the standardization of learning cycles, the adoption of the ECTS credit system, and, even more important, the adoption of a diploma supplement, a detailed document that describes the learning goals of a learning cycle. In order to fill out such document, the Joint Quality Initiative introduced a set of shared descriptors for qualifications awarded to students that signify completion of a higher education cycle (first cycle, master or PhD), the so called Dublin descriptors [5]. These descriptors can be considered as a taxonomy of education goals, in which the depth of knowledge is described using terms like “knowledge and understanding”, “application of knowledge and understanding”, “ability to formulate responses to to abstract problems”.

The introduction of a diploma supplement, and the standardization of descriptors clearly represents a step forward towards the adoption of quality assurance systems. However, these tools need some form of methodology – i.e., steps and guidelines – that guide a certificator in the process of producing such documents, and to avoid that they become a bureaucratic exercise. We believe that diogene may represent a contribution towards the definition of a supporting methodology in this context. Note, in fact, that diogene depth of knowledge levels nicely maps to Dublin descriptors.

While these observations do encourage the adoption of the methodology, it is important to note that, according to our experience, this is not a straightforward process. We have realized that, before structuring a whole certification process that spans different courses held by different instructors, it is necessary to “rethink” the courses according to guidelines of the methodology. This process may by long, since it requires a progressive familiarization of both instructors and students with the certification task. As a consequence, it is not reasonable to think that such a system can be “superimposed” by authority to any learning process. Yet, our experiences – especially those conducted in conjunction with our

14

colleagues that teach math courses – suggest that there may be significant benefits in terms of design and teaching quality by the adoption of the methodology even in those cases in which at the end it is not possible to actually produce certifications. The main reason of this is that, by forcing instructors to reason about their courses “from the outside”, i.e., from the perspective of the target levels, the methodology often improves the understanding that the instructor has of the overall goals of her/his course.

To conclude this work, we want to mention what might represent a further research direction for this project. As it can be seen, certifications produced by the system are essentially static, i.e., they represent a “snapshot” of the knowledge and skills that a student has shown to possess at a given time. While this is inherent to all certification systems, still we know that knowledge evolves with time. A stimulating research problem would be to enhance the system in such a way to capture how knowledge evolves – for example, how knowledge of math grows after a physics course – possibly by introducing a notion of “dependency” among target levels. This will be the subject of our future work.

Bibliography[1] J. Bergin. Fourteen Pedagogical Patterns for Teaching Computer Science. In Proceedings of the 5th

European Conference on Pattern Languages of Programs (EuroPLoP), 2000.

[2] B. S. Bloom. Taxonomy of Educational Objectives, Handbook I: The Cognitive Domain. New York: David McKay Co Inc., 1956.

[3] G. Costagliola, F. Ferrucci, G. Polese, G. Scanniello. A Visual Language-Based System for Designing and Presenting e-learning Courses. International Journal of Distance Education Technologies (2005) 3:1-19.

[4] Declaration of Bologna - European Space of Higher Education. http://www.bologna-berlin2003.de/pdf/bologna_declaration.pdf.

[5] Dublin Descriptors. http://www.jointquality.org.

[6] R. M. Gagné, R. Briggs, W. Wager. Principles of Instructional Design. Orlando, FL: Harcourt Brace Jovanovich, 1992.

[7] D. Lynne Goldblatt Anthony. Patterns for Classroom Education. In Proceedings of the 2nd Annual Conference on Pattern Languages of Programs (PLoP), 1995.

[8] J. T. Gorgone, G. B. Davis, J. S. Valacich, H. Topi, D. L. Feinstein, H. E. Longenecker. Information Systems 2002 - Model Curriculum and Guidelines for Undergraduate Degree Programs in Information Systems. (2002) :.

[9] T. Facklam. Certification of Persons - ISO/IEC DIS 17024, General requirements for bodies operating certification of persons. ISO Bullettin (2002) October:31-34.

[10] M. D. Merrill, Z. Li, M. K. Jones. Instructional Transaction Theory: An Introduction. Educational Technology (1991) 31:7-12.

[11] M. D. Merril. Component Display Theory. In Instructional-Design Theories and Models: An Overview of Their Current Status. C.M. Reigeluth (Ed.). 1983.

15

[12] D. J. Muraida, J. M. Spector. Automatic Design Instruction. In Instructional Design: International Perspectives. S. Dijkstra,N. Seel,F. Schott &D. Tennyson (Eds.). 1997.

[13] The Pedagogical Patterns Project. http://www.pedagogicalpatterns.org.

[14] M. Pietronilla Penna, V. Stara. E-learning's Failure: Why Should We Use a Learner-Centred Design. Journal of e-Learning and Knowledge Society (2005) 3:-.

[15] W. Van den Berghe. Application of ISO 9000 Standards to Education and Training. Vocational Training European Journal (1998) 15:20-28.

[16] C. Vrasidas. A Systematic Approach for Designing Hypermedia Environments for Teaching and Learning. International Journal of Instructional Media (2002) 29:13-25.

16

1Introduction and Motivations2Overview2.1Principles of diogene2.2Main Features of diogene

3The Model3.1Depth of Knowledge in diogene3.2Target Levels

4System Architecture5Methodological StepsStep 1: Definition of the Course ContentsStep 2: Definition of the Target LevelsStep 3: Collecting Process DataStep 4: Production of the Certification

6 A Concrete Experience6.1Examination Procedures6.2Evaluation

7Conclusions and Open Problems