CONALI ontologyA framework for design and evaluation of constructively aligned courses in higher education: putting in focus the

Educational Goal VerbsAntonio Maffei, Lorenzo Daghini , Andreas Archenti , Niels Lohse

KTH Royal Institute of Technology. Dept. of production Engineering

Loughborough University, Wolfson School of Mechanical and Manufacturing Engineering,

Outline

• Constructive alignment: international perspective and KTH reception

• Application and Measurment of constructive alignment

Constructive alignment in literature

0

10

20

30

40

50

60

19

95

19

98

20

01

20

04

20

07

20

10

20

13

Scopus

Web of Science

0100200300400500600700800900

19

95

19

98

20

01

20

04

20

07

20

10

20

13

Google Scholar

0

10

20

30

40

50

6019

95

19

98

20

01

20

04

20

07

20

10

20

13

Scopus

Web of Science

0100200300400500600700800900

19

95

19

98

20

01

20

04

20

07

20

10

20

13

Google Scholar

(a) (b)

(a) Annual number of paper with CA as main topic in Scopus and Web of Science. (b) Annual number of documents featuring CA in Google Scholar

Constructive alignment at KTH

• LH201V Learning and Teaching. Available: http://www.kth.se/student/kurser/kurs/LH201V?l=en

• LH202V Knowledge Building in Teachers Practice. Available: http://www.kth.se/student/kurser/kurs/LH203V?l=en

• LH216V Develop the Learning by Using Grading Criteria. Available: http://www.kth.se/student/kurser/kurs/LH216V?l=en

• …

Problem with Constructive alignment...

Problematic areas connected with CA

1. Scarcity of Resource (Biggs)

2. Resistance to Change (Biggs)

3. High interest and consequent risk for uncontrolled and diverging growth (Authors)

4. Assessing CA (Authors plus literature)

...and solution offered by ontological approach

Problematic areas connected with CA Benefit of producing an ontology

1. Scarcity of Resource (Biggs) - To enable reuse of domain knowledge- To analyze domain knowledge

2. Resistance to Change (Biggs) - To make domain assumptions explicit- To separate domain knowledge from

the operational knowledge

3. High interest and consequent risk for uncontrolled and diverging growth (Authors)

- To share common understanding of the structure of information among people or software agents

4. Assessing CA (Authors plus literature)

- Through analysis of the body of knowledge produced and definition of specific KPI

Contribution of this work

Produce the first iteration of an ontological framework model for design and evaluation of constructively aligned courses (Hence CONALI ontology).

Instantiate the defined CONALI ontology on a selected set of courses given at KTH IIP

Define and analyze some KPI connected with the alignment of educational units and interrogate the knowledge base on the performance attained by the represented courses and the areas where improvement is necessary.

Main assumption behind CA

Intended Learning Outcomes

Teaching and Learning Activities

Assessment Task

Educational Goal Verbs

isTestedBy

testsILO

hasEGV isEGVOf

isBasedOn

isAddressedByTLA

addressILO

enactsEGV

isBaseOf

enactedBy

Instantiation of the ontology: EGV

Remembering Verbs Understanding Verbs

....

Creating Verbs

Instantiation on a courseILO EDV TLA AT KoKnow

01 Describe Lecture Multiple Choice Short Answer

Declarative (D)

02 Choose Lecture Multiple Choice D

03 Name Lecture Multiple Choice D

04 Plan Learning Cell

Group Project

Functional (F)

Peer Teaching Student Presentation

05 Use TutorialCase Based Learning

Group Assessment F

06 Produce Interactive Work in Class

Short Practical Exercise

F

07 Describe Lecture Multiple Choice D

08 Understand Learning Cell

Group Project

F

Peer Teaching Student Presentation

KPI#1 %AEdU: Percentage of alignment

Scenario ScoreEGV does not indicate a learner action 0EGV indicates a learner action Partial alignment EGVILO = EGVTLA and EGVILO = EGVAT

EGVILO = EGVAT and EGVILO = EGVTLA3

Perfect Alignment EGVILO = EGVTLA = EGVAT

9

% 𝐴𝐸𝑑𝑈=∑

0

𝑉

𝐸𝐺𝑉 𝑖𝑠𝑐𝑜𝑟𝑒

9∗𝑉

Aggregate KPI

Single EGV KPI

Single ILO KPI

% 𝐴𝐸𝐺𝑉=𝐸𝐷𝐺𝑠𝑐𝑜𝑟𝑒

9

% 𝐴𝐼𝐿𝑂=∑

0

𝐼

𝐸𝐺𝑉 𝑗 𝑠𝑐𝑜𝑟𝑒

9∗ 𝐼

Kind of Knowledge

Intended Learning Outcomes

Functioning Knowledge

Declarative Knowledge

Kind of Knowledge

hasKindOf Knowledge isKindOf KnowledgeOf

hasSubclass hasSubclass

Kind of Knowledge alignment

AT for Functioning Knowledge

AT for Declarative Knowledge

Assessment TaskhasSubclass hasSubclass

Declarative Knowledge

TLA for Declarative Know.

learnedThroughTLAForDK

enablesLearningOfDK

Functioning Knowledge

TLA for Functioning Know.

learnedThroughTLAForFK

enablesLearningOfFK

isATForDK

assessedThroughATForDK

isATForFK

assessedThroughATForFK

Teaching and Learning Activities

hasSubclass hasSubclass

KPI#2 %KAEdU: Kind of Knowledge alignment

ScenarioScore

K EGV /ILO K TLA K AT

D D D 1

D D F 0

D F D 0

D F F 0

F D D 0

F D F 0

F F D 0

F F F 1

Aggregate KPI

Single ILO KPI

% 𝐾𝐴𝐸𝑑𝑈=∑

0

𝑉

𝐾𝐴−𝐸𝐺𝑉 𝑖𝑠𝑐𝑜𝑟𝑒

𝑉

% 𝐾𝐴𝐼𝐿𝑂=∑

0

𝐼

𝐾𝐴−𝐸𝐺𝑉 𝑗 𝑠𝑐𝑜𝑟𝑒

9∗𝐼

Calculation of the Score

ILO EDV TLA AT KoKnow Score%A D/F K

01 Describe Lecture Multiple Choice Short Answer

Declarative (D)

9 3

02 Choose Lecture Multiple Choice D 9 3

03 Name Lecture Multiple Choice D 9 3

04 Plan Learning Cell

Group Project

Functioning (F)

9 3

Peer Teaching Student Presentation

05 Use TutorialCase Based Learning

Group Assessment F 9 3

06 Produce Interactive Work in Class Short Practical Exercise

F 9 3

07 Describe Lecture Multiple Choice D 9 3

08 Understand Learning Cell

Group Project

F 0 0

Peer Teaching Student Presentation

Score

% 𝐾𝐴𝑀𝐺1026=∑ 𝐾𝐴𝑠𝑐𝑜𝑟𝑒

3∗ ¿𝐼𝐿𝑂¿=21

24≃0,875

% 𝐴𝑀𝐺1026=∑ 𝐼𝐿𝑂𝑠𝑐𝑜𝑟𝑒

9∗¿𝐼𝐿𝑂¿=63

72≃ 0,875

Summary of the CONALI ontology class instantiation and KPI scores

Courses Individuals MG1026 MG2100 MG2033 MG2040 MG2028 Total

Clas

ses

ILOs 8 6 21 10 8 53 EGVs 8 14 36 17 11 86 TLAs 9 18 52 16 11 106 ATs 6 6 25 3 12 52

Total 31 44 134 46 42 297

Course %A %KAMG1026 87.5% 87.5%MG2100 81% 86%MG2033 70% 69%MG2040 80% 88%MG2028 91% 100%

Conclusion

The case study tested the main intended functions of the ontology:• Problematic area 3. The framework model was expressive

enough to capture the information required to address the metrics identified as basic requirement for the evaluation of CA. Result: encouraging! The ontology is more expressive than intended

• Problematic area 1 and 2. The ease of use and understanding of the model. Results: good! It took between 2-4 hours to instantiate a course;

Conclusion...some correlation

Course %A %KA YearsTeachersCAEduc/

Teacherstotal

MG1026 87.5% 87.5% 2 1/1

MG2100 81% 86% 1 1/1

MG2033 70% 69% 1 1/5

MG2040 80% 88% 0 1/2

MG2028 91% 100% 8 2/2

Problematic area 4. The following correlation between the results of the analysis and two parameters possibly related with CA quality suggest that the chosen KPI and related method are sound

Future works

Expand the domain of investigation from single EdU to entire programs. This includes also connection between different EdUs.

Include school in different countries and different discipline to validate the requirements identified in this pilot study and introduce new ones.

Introduce the connection between assessment task and grading scale. As Biggs point out, having a clear alignment among these two entities is fundamental for the alignment of the course. In view of this different classes must be created for formative and summative assessment in future releases.

Introduce capability to track EdU across different years with specific metrics to identify and address the “child diseases”.

Problematic area 4. Integrate the CONALI ontology with valid existing contributions in the domain of CA measurement.

Required tools

Ad-hoc system to interrogate the knowledge base Expert system that can suggest the best way of

teaching content based on best practice in the available knowledge base.

KTH ROYAL INSTITUTEOF TECHNOLOGY

Thank You!