Upload
jan-claes
View
321
Download
3
Embed Size (px)
Citation preview
1/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Jan ClaesSupervisors UGent : Geert Poels & Frederik Gailly
Supervisors TU/e : Paul Grefen & Irene Vanderfeesten
Investigating the process of process modeling and its relation to modeling quality
The Role of Structure Serialization
2/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Structure of the PhD
CHAPTER 5CONCLUSION
CHAPTER 4THEORIZATION
CHAPTER 3EXPLORATION
CHAPTER 2VISUALIZATION
CHAPTER 1INTRODUCTION
3/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
CHAPTER 1 – INTRODUCTIONResearch objectives
4/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Context
Business Process Management
Conceptual ModelingPhD
5/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Context
Article availableOrder received
Check availability
No
Yes
late deliveryundeliverable
Payment received
Procurement
Remove articleFrom catalogue
Inform customer
Financial settlement
Ship article
Customer informed
Article removed
Inform customer
+
+
(From BPMN Quick Guide, OMG, 2015)
6/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Definitions Definition 1: Business process“A business process consists of a set of activities that are performed in coordination in an organizational and technical environment. These activities jointly realize a business goal.” (Weske, 2007, p. 5)
Definition 2: Business process model“A business process model is a mostly graphical representation that documents the different steps that are or that have to be performed in the execution of a particular business process under study, together with their execution constraints such as the allowed sequence or the potential responsible actors for these steps.”
Definition 3: Process of process modeling“the sequence of steps a modeler performs in order to translate his mental image of the process into a formal, explicit and mostly graphical process specification: the process model.”
7/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 1: visualization
Research Objective 1Build knowledge about
how people create models
Overall objectiveCuriosity-driven
Build knowledge about PPM
Research Objective 2Build knowledge about
relation with quality
Research Objective 3Build knowledge about
structured modeling
Study 3: theorizationStudy 2: exploration
Research Objectives
8/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
CHAPTER 2 – VISUALIZATIONPPMChart
9/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Data collection
Pnina SofferMatthias Weidlich
Barbara WeberJakob Pinggera
Stefan ZugalJan Mendling
Hajo ReijersIrene Vanderfeesten
Dirk Fahland
Observational data
Cheetah Experimental
Platform
‘Experiment’ design
10/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
CREATE_ACTIVITYCREATE_START_EVENTCREATE_END_EVENTCREATE_ANDCREATE_XORCREATE_EDGE
Data collection
MOVE_ACTIVITYMOVE_START_EVENTMOVE_END_EVENTMOVE_ANDMOVE_XOR
DELETE_ACTIVITYDELETE_START_EVENTDELETE-END_EVENTDELETE_ANDDELETE_XORDELETE_EDGE
NAME_ACTIVITYRENAME_ACTIVITYNAME_EDGERENAME_EDGE
11/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 1 – Visualization
(From Pinggera et al., 2014)
Not enough detail
12/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 1 – VisualizationPoor visual expressivenessPoor perceptual discriminabilityPoor graphic economyPoor semiotic clarityPoor semantic transparencyPoor complexity managementPoor cognitive integration
13/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 1 – VisualizationPPMChart
CREATE_ACTIVITY CREATE_START_EVENT CREATE_END_EVENT CREATE_AND CREATE_XOR CREATE_EDGE MOVE_ACTIVITY MOVE_START_EVENT MOVE_END_EVENT MOVE_AND MOVE_XOR DELETE_ACTIVITYDELETE_START_EVENT DELETE-END_EVENT DELETE_AND DELETE_XOR DELETE_EDGE NAME_ACTIVITY RENAME_ACTIVITY NAME_EDGE RENAME_EDGE
Start event Edge Activity
Gateway
Edge
Activity
Edge
Edge
Activity
Edge Gateway
Edge
7298
9
32
14
30
31
10
3356
34
time
mod
el e
lem
ents
14/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
FastmodelingSlow
modelingInitialdelayMany
pauzesFewelementsMany
elements No(separate)lay-outing
Quicklay-outingDedicated
lay-outingphase
Continuouslay-outingUnpaired
eventcreation
Pairedevent
creationNo pauzes
SerializationPaired
gatewaycreation
Delayededge
creationChunked
modeling
Study 1 – Visualization
Based on dataset of 357 unique modeling executions
15/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 1 – VisualizationEvaluation
Sample of intended users (6 academic researchers) Five extreme examples in PPMChart or Dotted Chart Observe and measure amount, quality, and timing of
insights gained through the visualization Observe and ask about perceived usefulness
Results Perceived as useful More cognitive effective than Dotted Chart
16/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
CHAPTER 3 – EXPLORATIONRelation with quality
17/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 2 – Exploration
Based on dataset of 40 unique modeling executions
Fastmodeling
Slowmodeling
Initialdelay
Manypauzes
Fewelements
Manyelements
No(separate)lay-outing
Quicklay-outing
Dedicatedlay-outing
phase
Continuouslay-outing
Unpairedevent
creation
Pairedevent
creation
No pauzesSerializationPairedgatewaycreation
Delayededge
creation
Chunkedmodeling
18/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 2 – ExplorationFast
modelingSlow
modelingInitialdelay
Manypauzes
Fewelements
Manyelements
No(separate)lay-outing
Quicklay-outing
Dedicatedlay-outing
phase
Continuouslay-outing
Unpairedevent
creation
Pairedevent
creation
No pauzes Pairedgatewaycreation
Delayededge
creation
Chunkedmodeling
Based on dataset of 40 unique modeling executions
Serialization
19/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 2 – ExplorationFast
modelingSlow
modelingInitialdelay
Manypauzes
Fewelements
Manyelements
No(separate)lay-outing
Quicklay-outingContinuous
lay-outingUnpaired
eventcreation
Pairedevent
creation
No pauzesSerializationPairedgatewaycreation
Delayededge
creation
Chunkedmodeling
Based on dataset of 40 unique modeling executions
Dedicatedlay-outing
phase
20/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 2 – ExplorationFast
modelingSlow
modelingInitialdelay
Manypauzes
Fewelements
Manyelements
No(separate)lay-outing
Quicklay-outing
Dedicatedlay-outing
phase
Unpairedevent
creation
Pairedevent
creation
No pauzesSerializationPairedgatewaycreation
Delayededge
creation
Chunkedmodeling
Based on dataset of 40 unique modeling executions
Continuouslay-outing
21/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 2 – ExplorationSlow
modelingInitialdelay
Manypauzes
Fewelements
Manyelements
No(separate)lay-outing
Quicklay-outing
Dedicatedlay-outing
phase
Continuouslay-outing
Unpairedevent
creation
Pairedevent
creation
No pauzesSerializationPairedgatewaycreation
Delayededge
creation
Chunkedmodeling
Based on dataset of 40 unique modeling executions
Fastmodeling
22/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 2 – ExplorationFast
modelingInitialdelay
Manypauzes
Fewelements
Manyelements
No(separate)lay-outing
Quicklay-outing
Dedicatedlay-outing
phase
Continuouslay-outing
Unpairedevent
creation
Pairedevent
creation
No pauzesSerializationPairedgatewaycreation
Delayededge
creation
Chunkedmodeling
Slowmodeling
Based on dataset of 40 unique modeling executions
23/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 2 – ExplorationStructuredness Movement Speed
Based on dataset of 40 unique modeling executions
Fastmodeling
Slowmodeling
Quicklay-outing
Dedicatedlay-outing
phase
Continuouslay-outing
Serialization
Chunkedmodeling
CONJECTURE 1
CONJECTURE 2
CONJECTURE 3
24/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 2 – ExplorationConjecture 1: Structured modeling
results in understandable models
Conjecture 2: A high number of move operations results in less understandable models
Conjecture 3: Slow modeling results in less understandable models
25/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Structuredness• MaxSimulBlock• PercNumBlockAsAWhole
Speed• TotTime• TotCreateTime
Movement• AvgMoveOnMovedElements• PercNumElementsWithMoves
Study 2 – Exploration
Model quality• Perspicuity
a model that is unambiguously interpretable and can be made sound with only small adaptations based on minimal assumptions on the modeler’s intentions with the model
CONJ
ECTU
RE 1
CONJECTURE 2
CONJECTURE 3
26/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 2 – Exploration
T-testt=-2,231 (p=0,028)
T-testt=2,199 (p=0,030)
Based on dataset of 103 unique modeling executions
CONJ
ECTU
RE 1
27/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 2 – Exploration
T-testt=-1,984 (p=0,049)
T-testt=0,457 (p=0,648)
Based on dataset of 103 unique modeling executions
CONJ
ECTU
RE 2
28/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 2 – Exploration
T-testt=-2,183 (p=0,031)
T-testt=2,505 (p=0,014)
Based on dataset of 103 unique modeling executions
CONJ
ECTU
RE 3
29/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
CHAPTER 4 – THEORISATIONStructured Process Modeling Theory
(SPMT)
30/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 3 – Theorization
Based on dataset of 103 unique modeling executions
T-testt=-2,231
(p=0,028)
T-testt=2,199
(p=0,030)
CONJECTURE 1structuredness
T-testt=-1,984
(p=0,049)
T-testt=0,457
(p=0,648)
T-testt=-2,183
(p=0,031)
T-testt=2,505
(p=0,014)
CONJECTURE 2movement
CONJECTURE 3speed
WHY?
31/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 3 – Theorization
Combined
Flow-oriented Aspect-oriented
Undirected
Based on dataset of 118 unique modeling executions
32/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 3 – Theorization Observation 1. Almost all modelers paused frequently
during the modeling process Observation 2. A large group can be categorized as
“flow-oriented process modeling” Observation 3. A smaller group can be categorized as
“aspect-oriented process modeling” Observation 4. Another large group used a combination
of both former styles Observation 5. Another small group can be categorized
as “undirected process modeling” Observation 6. The “undirected” sessions lasted longer
than the other approaches
Based on dataset of 118 unique modeling executions
33/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 3 – Theorization Impression 1. Modelers need serialization of the modeling
process to deal with its complexity Impression 2. Structured serializing of the modeling
process helps avoiding ‘mistakes’
Impression 3. Structured serializing does not support every
modeler to avoid ‘mistakes’ to the same extent
Based on dataset of 118 unique modeling executions
34/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 3 – TheorizationCognitive Load Theory
Working memory capacity is limited Working memory overload causes decrease in
• Effectiveness (i.e., more mistakes)• Efficiency (i.e., more time and effort)• Learning
Cognitive Fit Theory Load is lower when there is a fit
• Between representation, tool or strategy on the one hand• And task or modeler on the other hand
35/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 3 – Theorization
A B A determines BA B The more A, the more B+ A B The more A, the less B– A B A translates into B
learning style
degree of serialization
adopted serialization style
field-dependency need for structure
– +
course of intrinsic cognitive load for process modeling phases
course of intrinsic cognitive load for aggregation phases
course of cognitive overload
course of intrinsic cognitive load for strategy building phases
+ + +
serialization style fitstructuredness of serialization– – – –
1 2 3
36/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Study 3 – Theorization
Evaluation of utilityNovelty
(uses existing theories in fundamental new way)Parsimony
(11 constructs, 15 associations)Consistency
(can explain additional observations)Plausibility
(accurate and profound explanation)Credibility
(building blocks are established theories)Transferability
(problem solving in general)
Consistency based on dataset of 143 unique modeling executions
37/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
CHAPTER 5 – CONCLUSIONSummary & Future work
38/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
FUTURE WORK
Knowledge gaps
Knowledge contribution Research instrument
Knowledge gap 4How to change one’s modeling strategy?
Knowledge gap 3How should one model
in a specific context?
Study 1: visualizationContributions
A – PPMChartB – 22 patternsC – 13 observations
Study 2: explorationContributionsD – 8 patternsE – 3 conjecturesF – 1 metric
Knowledge gap 1How do people
currently model?
Knowledge gap 2Which strategy is
intrinsically better?
A BC
D E F
H HG I
Study 3: theorizationContributions
G – 5 stylesH1 – 6 observationsH2 – 3 impressionsI – SPMT
I
39/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Future Work
Use developed knowledge toDevelop prescriptive theory
Cognitive profile best modeling approachDevelop method (SPMM)
(1) Determine (2) learn (3) apply best approachDevelop tool support
Cognitive tests Interactive digital approach tutorial
40/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Reflection – Deduction, induction, abduction
Rule Case
Result
Deduction
Reveals effectResults in certainty
InductionCase Result
Rule
Reveals mechanismResults in probability
AbductionResult Case
Rule
Reveals causeResults in possibility
Process modeling as application domain> ordering of data facilitates pattern recognition <
41/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Reflection – Student observationAssumption: overload has same causes
for students and practitionersAssumption: overload has same consequences
for students and practitioners
STUDENTS…… are representative participants
… don’t suffer from Expert-Reversal Effect… form a homogeneous group
… provide heterogeneous set of observations… reach point of overload faster than practitioners
42/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Reflection – Empirical behavioral researchProcess modeling = complex and dynamic taskIdentify/measure/control confounding variables Ignore/assume constant/assume minimal effect
Include multiple variables in modelMix techniques (de/in/abduction, quanti/qualitative)
Open-world assumption> conclusions are incomplete unless proven otherwise <
> no conclusions from insignificant results <> more accurate, but slower progress <
43/43www.janclaes.info
INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION
Thanks for your attention! Do you have any questions?
http://www.janclaes.infoTwitter: @janclaesbelgium