New Teaching conceptual modelling: how to guide the conceptual …merode.econ.kuleuven.be/lectures/NEMO_Tutorial_slides... · 2018. 6. 29. · Teaching conceptual modelling: how to

Teaching conceptual modelling: how to guide the conceptual

modelling process of students?

Monique Snoeck LIRIS, FEB@KULeuven

[email protected]

1

4

1.Conceptual

Modelling (CM)is difficult... to learn... to teach

Scaffolding(instructional

design)

Feedback(automated)

Simulation(model execution)

Learning Analytics(does it help?)

5

1.Conceptual

Modelling (CM)is difficult... to learn... to teach Why?

Case Study: Conceptual Modelling

• Conceptual modelling is a complex learning task • requires rigorous analytical skills• many solutions to a problem, many routes to finding a good solution

• Current standard tooling is not adequate• Many tools do not provide any form of feedback• Some do focus on model validation as a form of feedback• Simulation tools focus on code (skeleton) generation• Lack of logging functionalities

• Research• Specific tool for educational purposes including logging, predictive analytics

and providing personalized feedback in real-time

6

Conceptual modelling…is a complex learning task analyzing business requirements constructing a semantically correct conceptual model

that reflects the structural and dynamic views of a given domain description

17:40

7

problem : skill transferability

• Teaching “experience” (domain specific knowledge) is difficult

• Some aspects cannot be gained with reading and lecturing alone, e.g. dynamic representation of a system-to-be

• Lack of technical insights

16:36

8

novice expert

Contributing factorsCM requires combining competences from different areas

Each contributing factor introduces own challenges

17:42

9

Domain knowledge

Modeling knowledge

Language knowledge

Skill

Teaching complex knowledge

• Compartmentalization• Atomistic domains of learning

• Fragmentation• Sequential teaching

• Transfer Paradox• Missing generalization skill to integrate

knowledge into unfamiliar aspects of the tasks

16:36 14

16

1.Conceptual



design)

Learning outcomes in CM Each learning outcome can be categorized by the following parameters:

• Knowledge level• Cognitive level• Content area (which can be defined by means of related keywords)

Example:

LO1: The student should be able to interpret a given UML diagram (translate from a model to a text)

Knowledge level: Conceptual Cognitive process: UnderstandingContent area: Models

17

Content areas tree

Bogdanova & Snoeck, 2018

18

Content areas and scaffolding levels

Adapted from Bogdanova & Snoeck, 2017

19

ClassObject Attribute

Class level

Inheritance

Relationships level: generalization

Binary Association

Recursive Association

Aggregation

N-aryAssociation

Association Class

Relationships level: association

Simple Model

Model level: simple

Complex Model

Model level: complex

Pattern

GENERAL

KNOWLEDGE

Revised Bloom’s Taxonomy

CreateEvaluateAnalyzeApply

Understand

Remember Define, Recall, Identify

Discuss, Explain, Match

Use, Practice, Execute

Examine, Analyze, Compare

Check, Verify, Critique

Design, Build, Improve

Cognitive dimension

Adapted from Krathwohl, 2002

20


Create

EvaluateAnalyzeApply

Understand

Remember

Cognitive dimension


21


CreateEvaluateAnalyzeApply

Understand

Knowledge dimension

Remember


22

Knowledge levels

Factual Terminology, notation and basic elements


Sources of information on modelling (textbooks, standards and thematic websites)

Modelling notation(s)

General modelling conventions – how to name classes, attributes, associations, etc.

Terms and their definitions according to the literature

23

Knowledge levels

Terminology, notation and basic elements

Conceptual Interrelationships between elements, principles and theories


Conceptual differences between various terms

Classification of information

Fundamental principles of conceptual modelling

Commonly used modelling patterns

24

Knowledge levels


Interrelationships between elements, principles and theories

Procedural How to do it: Subject-specific techniques and skills

25Revised Bloom’s Taxonomy

Step-by-step guidelines or algorithms for performing a modelling task at any stage

Criteria for use of a method or technique when solving a particular kind of modelling task

25

Knowledge levels


Interrelationships between elements, principles and theories

How to do it: Subject-specific techniques and skills

Metacognitive Awareness about your own knowledge and learning/thinking strategies


Knowledge of cognition as such

Strategic knowledge for learning the subject

Self-knowledge

26

Revised Bloom’s TaxonomyCognitive levels

Define, Recall, IdentifyRemember

Identify a false statement in the list of term definitions. (Remember – Factual – General)

Define properties characterizing inheritance (overlapping/disjoint, etc.)(Remember – Factual/Conceptual – Generalization)

Examples from educational resources:

27


Discuss, Explain, MatchUnderstand


Give an example of an instance belonging to more than one class(Understand – Conceptual – Class, Generalization)

Interpret cardinalities of the given simple class diagram(Understand – Factual/Conceptual – Model (Simple))

28


Use, Practice, ExecuteApply

Examples from educational resources :

Use a given textual analysis technique to identify all the relevant classes in the text(Apply – Procedural – Class)

Modify the attributes for a new kind of subclass(Apply – Conceptual – Generalization)

29


Examine, Analyse, CompareAnalyse


Compare main approaches to conceptual model design(Analyse – Conceptual/Procedural – General)

Identify the similarities between the given class diagrams(Analyse – Conceptual – Model (Simple))

30


31

Check, Verify, CritiqueEvaluate


Find the mistakes in given class diagrams and justify the answer(Evaluate – Conceptual – Model (Complex))

Find which of the given object diagrams is valid/invalid, according to the given model(Evaluate – Conceptual – Model (Simple)

Current state: what do we teach?


“Remember” and “Evaluate” are heavily underrepresentedMetacognitive knowledge level is not represented at all, factual – almost not represented

36



The majority of tasks address model and relationships levels; MOOCs “like” simple models, while books and exams “like” complex ones

37


UnderstandCreate

Conceptual

RememberEvaluateAnalyze

FactualMetacognitive

Apply

Procedural

Result: uneven scaffolding; lack of constructive approach

VS

38

39

1.Conceptual



design)


Modelling & Simulation• Understanding a model (e.g. for validating it against requirements)

requires “picturing” the resulting artefact

16:36 40

????

Specific Problem of CM• Complexity associated with ANALYZING & EVALUATING a CM

• Experienced modellers • Can mentally picture and simulate a model• They can analyse & evaluate its impact

• Junior modellers• lack experience to achieve these skills

• Common solutions• Explanations & Demo’s• Case Studies• Lot’s of exercises with individual feedback

41

Create

EvaluateAnalyzeApply

Understand

Remember

Simulation of requirements?• is considered as an effective instrument

allowing to achieve complex learning goals by

• Visualizing design choices into concrete forms• Learning by experiencing (knowledge that emerges from own practice)• Successful transfer of the skills to real-world environments

16:36

42

CM Simulation environment

• Adapted to learning context makes it suitable for research goals

• the use of a restricted part of UML as proposed by the MERODE methodology (Snoeck et al., 1998): creating executable PIMs (EPIM)

• Structural view: UML class diagram• Behavioural view: multiple interacting statecharts• Interaction model : CRUD + collaboration rules

• starting from a high-level PIM allows removing or hiding details irrelevant for a conceptual modelling view.

16:36

43

Requires three clicks1 2 3

JMermaid Simulation

16:36

44

Current State

• Simulation by means of 3 clicks

2

“Order” is not required to create an instance of an invoice

3

1

45

Built in data storage with query builder

16:36

46

CM Simulation environment

• Fully functional (single click) prototype• Teaching with JMermaid

• single conceptual model generate simulate• test and early defect detection revisit/refine model

18:20 47

Test -> early detection of

defects

Revisit / refine

For more details, see EMMSAD 2012, EMMSAD 2013, and ModelsWard 2013, papers by Sedrakyan, Snoeck

Reflect on

48

1.Conceptual



design)

Feedback(automated)


Feedback in CM

• Personal feedback • + immediate = most demanded form of feedback by students• is the most “expensive” in terms of teacher time• automated “just in time” or “anytime” feedback

• Group feedback• for home works• for group assignment

• peer feedback

49

Feedback in JMermaid

• Observed problem: students cannot interpret the behaviour of the prototype

• Solution: Augment generated prototype with a feedback feature that links results of a model test to its causes in the corresponding part of the model [EMMSAD 2012]

18:01 50

Feedback - Current State• Feedback in the modelling

environment:• Model to text feature• Model validation checks

51

Feedback - Current State• “To buy wholesale products customers need to place an order.

However ordering is not required for buying a retail product.”

18:21

52

2“Order” is required to create an instance of an invoice

3

1

Feedback - Current state

18:23

53

Examples of other Artefacts for Feedback• Online tests (Automated !)

• small exercises per topic• individual feedback

• Homework/Assignments• small cases per topic• individual feedback + group feedback

• Self-study exercises / lab exercises• exercises archived from previous home works + lab exercises• model solutions + commented student solutions• individual feedback + group feedback

• Group work• larger, integrated case• Peer, group/individual feedback

• Automated feedback in simulation environment• Individual immediate/on-demand

54

Online Test Feedback Example 55

56

1.Conceptual



design)

Feedback(automated)



Research question 1

• Does a feedback-enabled prototyping (simulation) improve modeling knowledge of a novice modeler in terms of his/her capability of assessing a model’s semantic quality ?

• Method empirical evaluation• Assessing the effects on learning outcomes by comparing the test scores with

and without a use of proposed simulation technique

18:01

57

Experimental study

• 5 studies : with participation of 169 final year master-level students from Leuven and Brussels campuses, spanning 3 academic years (2012-2013-2014-2015)

• Dependent variable = model quality

• Experimental design : pre/post test control group design• 4-group factoral with cycle rotation• Extraneous variables: personal characteristics such as gender, previous knowledge and skills [Venkatesh, 2003],

computer self-efficacy• Intended utility: perceived utility and ease of use [Technology Acceptance Model, Davis 89].

• More in [Sedrakyan, G., Snoeck, M., Poelmans, S. (2014)]

18:30

58

ConclusionsH1: Feedback-enabled simulation significantly improves model validation capabilities of a novice business analyst.

confirmed : magnitude of the effect = 2.33 – 4 ( out of 8 )

H2: The use of the prototype has a persisting learning effect on student’s test scores when is no longer used.

confirmed : no significant diff. withoutPT cycle following withPT cycle

H3 : The test scores are not influenced by any particular personal characteristics of users (previous knowledge and gender).

confirmed : no impact of variables on the relative advantage (with PT – without PT) was identified

H4 : The proposed simulation method is suitable for novice business analysts (user acceptance).

confirmed : no impact of acceptance variables on the relative advantage (with PT – without PT) was identified

User preferences: 4.5 – 5.58 (6 - point Likert scale)18:32

59

Research question 2

• Can modeling and validation process affect its outcomes?• How we can adapt teaching guidance to achieve process-oriented guidance (how to

do it right ?) based on learning process observations vs. outcome feedback (is the solution correct? Why(not)?)

16:36

60

Process Mining - Current State• Process Mining of the modelling process• Log of user activities in JMermaid for the group task during a whole

semester’s duration.TIMESTAMP

GROUP ID

SESSION ID

SESSION TYPE

SCORE

ORIGINAL ACTIVITY

ABSTRACTED ACTIVITY

MODELING VIEW

DIAGRAMMING TYPE

19/11/2013 1:54:00 1 Session1 EARLY 6 CREATE OBJECT CREATE S EDG 19/11/2013 1:54:16 1 Session1 EARLY 6 CREATE OBJECT CREATE S EDG 19/11/2013 1:55:55 1 Session1 EARLY 6 CREATE DEPENDENCY CREATE S EDG 19/11/2013 2:08:03 1 Session1 EARLY 6 CREATE ATTRIBUTE CREATE S EDG 19/11/2013 2:08:36 1 Session1 EARLY 6 CREATE EVENT CREATE B OET 19/11/2013 4:37:28 1 Session1 EARLY 6 CREATE EVENT CREATE B OET 19/11/2013 4:40:05 1 Session1 EARLY 6 DELETE EVENT EDIT B OET 19/11/2013 4:40:18 1 Session1 EARLY 6 UNDO DELETE EVENT EDIT B OET 19/11/2013 5:09:53 1 Session1 EARLY 6 REDO DELETE EVENT EDIT B OET 19/11/2013 5:10:58 1 Session1 EARLY 6 EDIT ATTRIBUTE EDIT S EDG 10/12/2013 11:04:18 1 Session2 LATE 6 CREATE METHOD CREATE B OET 10/12/2013 11:04:23 1 Session2 LATE 6 CREATE STATE CREATE B FSM 10/12/2013 11:05:05 1 Session2 LATE 6 CREATE STATE CREATE B FSM 10/12/2013 11:05:23 1 Session2 LATE 6 CREATE TRANSITION CREATE B FSM 12/12/2013 11:02:34 1 Session3 LATE 6 DELETE DEPENDENCY EDIT S EDG 12/12/2013 11:02:41 1 Session3 LATE 6 CREATE DEPENDENCY CREATE S EDG 12/12/2013 11:02:44 1 Session3 LATE 6 CREATE ATTRIBUTE CREATE S EDG 12/12/2013 11:02:47 1 Session3 LATE 6 DELETE STATE EDIT B FSM 12/12/2013 11:03:00 1 Session3 LATE 6 EDIT TRANSITION EDIT B FSM

61

Analysis of logs: example of findings

• Process Mining:• More iteration in the better performing groups (R),

whereas worst performing groups (L) show a more linear pattern

• Better students simulate more• Better students also verify behavioural part of the model through simulation

62 62

Worse Better

Worse

Better

Process Mining of the modelling process63

TIMESTAMP

GROUP ID

SESSION ID

SESSION TYPE

SCORE

ORIGINAL ACTIVITY

ABSTRACTED ACTIVITY

MODELING VIEW

DIAGRAMMING TYPE

19/11/2013 1:54:00 1 Session1 EARLY 6 CREATE OBJECT CREATE S EDG 19/11/2013 1:54:16 1 Session1 EARLY 6 CREATE OBJECT CREATE S EDG 19/11/2013 1:55:55 1 Session1 EARLY 6 CREATE DEPENDENCY CREATE S EDG 19/11/2013 2:08:03 1 Session1 EARLY 6 CREATE ATTRIBUTE CREATE S EDG 19/11/2013 2:08:36 1 Session1 EARLY 6 CREATE EVENT CREATE B OET 19/11/2013 4:37:28 1 Session1 EARLY 6 CREATE EVENT CREATE B OET 19/11/2013 4:40:05 1 Session1 EARLY 6 DELETE EVENT EDIT B OET 19/11/2013 4:40:18 1 Session1 EARLY 6 UNDO DELETE EVENT EDIT B OET 19/11/2013 5:09:53 1 Session1 EARLY 6 REDO DELETE EVENT EDIT B OET 19/11/2013 5:10:58 1 Session1 EARLY 6 EDIT ATTRIBUTE EDIT S EDG 10/12/2013 11:04:18 1 Session2 LATE 6 CREATE METHOD CREATE B OET 10/12/2013 11:04:23 1 Session2 LATE 6 CREATE STATE CREATE B FSM 10/12/2013 11:05:05 1 Session2 LATE 6 CREATE STATE CREATE B FSM 10/12/2013 11:05:23 1 Session2 LATE 6 CREATE TRANSITION CREATE B FSM 12/12/2013 11:02:34 1 Session3 LATE 6 DELETE DEPENDENCY EDIT S EDG 12/12/2013 11:02:41 1 Session3 LATE 6 CREATE DEPENDENCY CREATE S EDG 12/12/2013 11:02:44 1 Session3 LATE 6 CREATE ATTRIBUTE CREATE S EDG 12/12/2013 11:02:47 1 Session3 LATE 6 DELETE STATE EDIT B FSM 12/12/2013 11:03:00 1 Session3 LATE 6 EDIT TRANSITION EDIT B FSM

63

Log of user activities in JMermaid for the group task.


• Analysis of types of activities in early versus late modelling sessions

64

37%43%

8%12%

0%10%20%30%40%50%60%

PHASE1 PHASE2

create edit

31% 33%

13%23%

0%10%20%30%40%50%60%

PHASE1 PHASE2

create edit

35%22%

18%25%

0%10%20%30%40%50%60%

PHASE1 PHASE2

create edit

Worse Satisfactory Best


• Dotted charts: analysis of distribution of modelling effort over time:

• Students are deadline driven ...• Better students start earlier• Better students simulate more often

65

BB

B

B

B

W

S

S

W

W

W

WW

AA D1 PD& T LC D2

S

SS

66

1.Conceptual



design)

Feedback(automated)



1.Conceptual

Modelling (CM)can be made

easier... to learn

By Scaffolding anddeveloping the analysis

& evaluation skills

Feedback is required to let students understand

simulation results(automated)

Simulation is anexcellent way of developing these

skills

Learning Analytics helpsunderstanding & proved the

effect of interventions

Publications• See http://feb.kuleuven.be/monique.snoeck

• Selected Articles, Presentations & Conferences

• Daria Bogdanova, Monique Snoeck. Domain Modelling in Bloom: Deciphering How We Teach It. The Practice of Enterptise Modelling (2017), pp.3-17.

• Estefanía Serral, Monique Snoeck. Conceptual Framework for Feedback Automation in SLEs. 20th International Conference on Knowledge-Based and Intelligent Information & Engineering Systems (KES-SEEL2016). Tenerife, Spain. 2016.

• Sedrakyan G, De Weerdt J, Snoeck M, Process-mining enabled feedback: “tell me what I did wrong” vs. “tell me how to do it right”, 2016, Computers in Human Behavior, vol. 57, pp. 352 - 376.

• Sedrakyan G, Snoeck M, Poelmans S, Assessing the effectiveness of feedback enabled simulation in teaching conceptual modeling, 2014, Computers and Education, vol. 78, pp. 367 - 382.

• ....

Software• merode.econ.kuleuven.be

Contact: [email protected]

67

http://feb.kuleuven.be/monique.snoeck

mailto:[email protected]

Documents

New Teaching conceptual modelling: how to guide the conceptual …merode.econ.kuleuven.be/lectures/NEMO_Tutorial_slides... · 2018. 6. 29. · Teaching conceptual modelling: how to