Mohammad Ali Jabbari Sabegh - QUT · Chapter 4: Conceptual Modelling Research in Information Systems: ... AIS Association for Information Systems BWW Bunge, Wand, and Weber BPMN Business

A STUDY OF THE COMBINED USE OF

CONCEPTUAL MODELS

Mohammad Ali Jabbari Sabegh

MBA, BSc

Submitted in fulfilment of the requirements for the degree of

Doctor of Philosophy

School of Management

QUT Business School

Queensland University of Technology

2017

A Study of The Combined Use of Conceptual Models i

Supervisory Panel

Principal Supervisor

Professor Jan Recker, PhD

School of Management

QUT Business School


Associate Supervisor

Professor Peter Green, PhD

School of Accountancy

QUT Business School


ii

Keywords

Conceptual Models, Combined Use, Model Combinations, Ontology, Completeness, Overlap,

Ontological Analysis, Context, Faithful Use

A Study of The Combined Use of Conceptual Models iii

Abstract

Conceptual models are graphical representations of the relevant features of a system

domain. They play a significant role in the early detection and correction of information

systems development errors and help analysts, developers, and end users to better understand

the domain. However, despite conceptual modelling being an active research area in

information systems, much understanding of conceptual modelling practice remains

outstanding. This thesis addresses such aspect: while practitioners typically use multiple, and

different, types of models in system analysis and design tasks, no intense scrutiny of how and

why they use model combinations has been undertaken.

The objective of this research is to provide an in-depth investigation into the use of

model combinations during information systems analysis and design. A multi-method

research design was used to achieve this objective. The contribution of this thesis is built on

three streams of studies: a systematic literature review, exploratory interviews, and a lab

experiment for theory testing.

First, a systematic literature review was conducted to analyse the extent to which

research over the past two decades has identified research opportunities and provided answers

to Wand and Weber’s (2002) question regarding “How can we model the world to better

facilitate our developing, implementing, using, and maintaining more valuable information

systems?”.

The next two studies focussed on the theory of faithful use of conceptual model

combinations. This theory was developed to predict and explain the use of multiple models in

combination; however, this theory has not yet been tested. In the second study, semi-

structured interviews were conducted to explore and determine the circumstances that lead

information system (IS) professionals to use multiple models.

In the third study, lab experiments were designed and executed to test the hypotheses

derived from the propositions of the theory of faithful use of conceptual model combinations.

The results led to a refined and improved theory that provides explanation and in-depth

understanding of how and why practitioners use model combinations.

iv

Table of Contents

Supervisory Panel ................................................................................................................. i

Keywords ............................................................................................................................ ii

Abstract .............................................................................................................................. iii

Table of Contents ................................................................................................................ iv

List of Figures ................................................................................................................... vii

List of Tables .................................................................................................................... viii

List of Abbreviations ........................................................................................................... x

Statement of Original Authorship ........................................................................................ xi

Acknowledgements ............................................................................................................ xii

Part A: Preface ..................................................................................................................... 1

Chapter 1: Introduction ................................................................................... 1

1.1 Introduction ............................................................................................................... 1

1.2 The Problem Statement .............................................................................................. 3

1.3 Research Questions .................................................................................................... 5

1.4 Scope of Study ........................................................................................................... 7

1.5 Thesis Structure ......................................................................................................... 8 1.5.1 Part A .............................................................................................................. 8 1.5.2 Part B ............................................................................................................... 8 1.5.3 Part C ............................................................................................................. 10

Chapter 2: Research Background .................................................................. 11

2.1 Conceptual Modelling in Information Systems ......................................................... 11

2.2 Research on Single Grammars or Models ................................................................. 13 2.2.1 Conceptual Model Design .............................................................................. 13 2.2.2 Conceptual Model Interpretation .................................................................... 15

2.3 Research on Multiple Grammars or Models .............................................................. 16

2.4 The Theory of Faithful Use of Conceptual Model Combinations .............................. 17

2.5 Conclusion ............................................................................................................... 20

Chapter 3: Research Methods ........................................................................ 22

3.1 Research Method Choices ........................................................................................ 22 3.1.1 Systematic Literature Review ......................................................................... 22 3.1.2 Semi-Structured Interviews ............................................................................ 23 3.1.3 Experiment .................................................................................................... 24

3.2 Reliability and Validity ............................................................................................ 24 3.2.1 Reliability and validity of systematic literature review .................................... 24 3.2.2 Reliability and Validity of Semi-Structured Interviews and the Experiment .... 25

A Study of The Combined Use of Conceptual Models v

Part B: Studies ................................................................................................................... 29

Chapter 4: Conceptual Modelling Research in Information Systems: What We

Know Now and What We Still Do Not Know ...................................................... 29

Abstract ............................................................................................................................. 29

4.1 Introduction ............................................................................................................. 30

4.2 Background ............................................................................................................. 31

4.3 Literature Review Procedures................................................................................... 35

4.4 Analysis of the Literature ......................................................................................... 36 4.4.1 Conceptual Modelling Grammars ................................................................... 38 4.4.2 Conceptual Modelling Methods ...................................................................... 42 4.4.3 Conceptual Modelling Scripts ........................................................................ 45 4.4.4 Conceptual Modelling Context ....................................................................... 48 4.4.5 Identified articles that did not match the research framework by Wand and Weber

(2002) altogether ............................................................................................ 53

4.5 Guiding the next wave of conceptual modelling research: A new framework ............ 54

4.6 Conclusion ............................................................................................................... 59

Chapter 5: Combined Use of Conceptual Models in Practice: An Exploratory Study

…………………………………………………………………………………….61

Abstract ............................................................................................................................. 61

5.1 Introduction ............................................................................................................. 62

5.2 Background ............................................................................................................. 63 5.2.1 Ontological Analysis as a Theoretical Foundation of Conceptual Modelling ... 64 5.2.2 Empirical Studies of Conceptual Modelling Practice ...................................... 66

5.3 Research Method ..................................................................................................... 67 5.3.1 Design ........................................................................................................... 67 5.3.2 Participants and Procedures ............................................................................ 69 5.3.3 Data Analysis ................................................................................................. 70

5.4 Findings ................................................................................................................... 72 5.4.1 Why practitioners use multiple types of models .............................................. 73 5.4.2 How practitioners decide on which models to use ........................................... 78

5.5 Discussion ............................................................................................................... 84 5.5.1 Implications for ontological research on conceptual modelling ....................... 88 5.5.2 Implications for empirical research on conceptual modelling .......................... 90 5.5.3 Implications for the practice of conceptual modelling ..................................... 90 5.5.4 Limitations ..................................................................................................... 91

5.6 Conclusion ............................................................................................................... 92

Chapter 6: An Experimental Study to Test and Extend the Theory of Faithful Use of

Conceptual Model Combinations......................................................................... 94

6.1 Introduction ............................................................................................................. 95

6.2 Use of Multiple conceptual models .......................................................................... 96

6.3 The Theory of Faithful Use of Conceptual Model Combinations .............................. 97

vi

6.4 Research Method ..................................................................................................... 99 6.4.1 Preliminaries .................................................................................................. 99 6.4.2 Experimental Design ...................................................................................... 99 6.4.3 Materials ...................................................................................................... 101 6.4.4 Establishing Levels of Combined Ontological Completeness and Ontological

Overlap ........................................................................................................ 103 6.4.5 Measures ...................................................................................................... 106 6.4.6 Covariates .................................................................................................... 108 6.4.7 Participants .................................................................................................. 109 6.4.8 Procedures ................................................................................................... 109

6.5 Hypothesis Development ....................................................................................... 112 6.5.1 Selection of Model Combination .................................................................. 112 6.5.2 Development of Domain Understanding ....................................................... 114 6.5.3 Perceived usefulness of model combinations ................................................ 116

6.6 Data Analysis ......................................................................................................... 117 6.6.1 Preliminary Tests ......................................................................................... 118 6.6.2 Hypothesis Testing: Model Selection ............................................................. 118 6.6.2 Hypothesis Testing: Domain Understanding ................................................. 124

6.7 Discussion ............................................................................................................. 134 6.7.1 Summary of Findings ................................................................................... 134 6.7.2 Theoretical Implications ............................................................................... 138 6.7.3 Practical Implications ................................................................................... 139 6.7.4 Limitations ................................................................................................... 140

6.8 Conclusion ............................................................................................................. 141

Part C: Retrospective ....................................................................................................... 142

Chapter 7: Discussion ................................................................................... 142

7.1 Synthesis of Findings from Studies ........................................................................ 142

7.2 Implications ........................................................................................................... 147 7.2.1 Implications for Research ............................................................................. 147 7.2.2 Implications for Practice............................................................................... 149

Chapter 8: Conclusions ................................................................................ 151

8.1 Contributions ......................................................................................................... 151 8.1.1 For research ................................................................................................. 151 8.1.2 For practice .................................................................................................. 152

8.2 Limitations............................................................................................................. 153

8.3 Future Research ..................................................................................................... 155

8.4 Conclusion ............................................................................................................. 156

Bibliography ....................................................................................................... 157

Appendices .......................................................................................................... 176

A Study of The Combined Use of Conceptual Models vii

List of Figures

Figure 1.1 Examples of different models developed using various conceptual modelling

grammars .................................................................................................... 2

Figure 1.2 Outline of thesis structure and research objectives ................................... 9

Figure 2.1 Premises of the Theory of Faithful Use of Conceptual Model Combinations

(Recker, 2014) .......................................................................................... 18

Figure 4.1 Examples of different scripts developed using four conceptual modelling

grammars .................................................................................................. 32

Figure 4. 2 Framework for Research on Conceptual Modelling (Wand & Weber, 2002,

p. 364) ...................................................................................................... 33

Figure 4.3 A New Research Framework to support future conceptual modelling research

................................................................................................................. 57 Figure 5.1 Interview sampling and theoretical saturation ........................................ 71

Figure 5.2 Illustration of coding procedure ............................................................. 72

Figure 5.3 Example of diagram use to represent different aspects of a system ......... 76

Figure 5.4 The role of different factors that determine the use of multiple models... 86

Figure 5.5 The role of different factors that determine the selection of different types of

models ...................................................................................................... 87

Figure 6.1 Illustration of Ontological Completeness and Overlap of Two Conceptual

Models (Recker, 2014) .............................................................................. 98

Figure 6.2 Four Conceptual Models for the Library Management System Case

(Whiteley, 2013, pp. 228-263) ................................................................ 102

Figure 6.3 Overview of experimental procedure ................................................... 110

Figure 6.4 Estimated Marginal Means of Performance on Domain Understanding 132

Figure 6.5 Estimated Marginal Means of Perceived Usefulness ............................ 134

viii

List of Tables

Table 1.1 Papers included in this dissertation ............................................................ 5

Table 4. 1 Research Opportunities in Conceptual Modelling (Wand & Weber, 2002, p.

373) .......................................................................................................... 34

Table 4.2 Reviewed papers ..................................................................................... 36

Table 4. 3 Research on conceptual modelling based on focus of the study .............. 37

Table 4. 4 Number of papers per type of evidence .................................................. 37

Table 4. 5 Number of papers per type of model ...................................................... 38

Table 4. 6 Papers on conceptual modelling grammars and type of evidence ............ 39

Table 4. 7 Papers on conceptual modelling method and type of evidence ................ 43

Table 4. 8 Research on conceptual modelling scripts and type of evidence ............. 45

Table 4. 9 Papers in conceptual modelling context and type of evidence ................. 49

Table 5.1 Examples of studies on ontological completeness of modelling grammars66

Table 5.2 Demographic information ....................................................................... 70

Table 5.3 Examples of open codes collected ........................................................... 71

Table 5.4 Most widely used models ........................................................................ 73

Table 6.1 Ontological evaluation of conceptual model constructs - Library Management

System and High Peak Bicycles .............................................................. 105

Table 6.2 Overlap Analysis................................................................................... 105

Table 6.3 MOC and MOO for combinations of three and four models - High-Peak

Bicycles Case.......................................................................................... 106

Table 6.4 Measurement instrument for dependent variables .................................. 107

Table 6.5 Distribution of participants in each group .............................................. 111

Table 6.6 Reliability and validity analysis of scaled measures............................... 118

Table 6.7 Test of homogeneity of variances for dependent measures .................... 118

Table 6.8 Frequency of participants and the number of models in combination ..... 119

Table 6.9 Frequency and descriptive statistics for second model selection ............ 119

Table 6.10 Mean and standard deviation of constructs for model selection............ 119

Table 6.11 Correlation matrix for selection of second model................................. 120

Table 6.12 Hierarchical logistic regression for selection of second model ............. 121

Table 6.13 Correlation matrix for the selection of the third model......................... 121

Table 6.14 Hierarchical logistic regression for the third model selection ............... 122

A Study of The Combined Use of Conceptual Models ix

Table 6.15 Number of models selected and number of models they used in combination

to answer the questions ........................................................................... 123

Table 6.16 Frequency of types of models .............................................................. 124

Table 6.17 ANOVA test to compare familiarity difference between groups .......... 124

Table 6.18 Descriptive statistics............................................................................ 125

Table 6.19 Construct correlations ......................................................................... 126

Table 6.20 Results for model comprehension ........................................................ 127

Table 6.21 Results for problem-solving ................................................................ 128

Table 6.22 Simple effect analysis for problem-solving performance ..................... 129

Table 6.23 Results for Cloze test .......................................................................... 130

Table 6.24 Results for perceived usefulness .......................................................... 133

Table 6.25 Summary of hypothesis testing ............................................................ 134

x

List of Abbreviations

AIS Association for Information Systems

BWW Bunge, Wand, and Weber

BPMN Business Process Modelling Notation

DFD Data Flow Diagram

EPC Event-driven Process Chains

ERD Entity Relationship Diagram

IS Information Systems

MOC Maximal Ontological Completeness

MOO Minimal Ontological Overlap

ORM Object Role Modelling

UML Unified Modelling Language

YAWL Yet Another Workflow Language

A Study of The Combined Use of Conceptual Models xi

Statement of Original Authorship

The work contained in this thesis has not been previously submitted to meet

requirements for an award at this or any other higher education institution. To the best of my

knowledge and belief, the thesis contains no material previously published or written by

another person except where due reference is made.

QUT Verified Signature

xii

Acknowledgements

There are several people that I am delighted to express my immense gratitude and

appreciation to for helping me in many ways throughout my PhD journey.

I am incredibly fortunate and truly proud to have Professor Jan Recker and Professor

Peter Green as my supervisors. I am a great admirer of your immense knowledge and

enthusiasm toward science. It inspires me to continue my career as an academic. I would also

like to thank you for your continued support, thoughtful feedback, criticism, guidance,

motivation, and patience during my research. You have been tremendous mentors to me. Your

advice about my research and my career has been priceless. I could not imagine having better

supervisors and mentors for my PhD study.

I would also like to thank other people who helped me during my research. Thanks to

Professor Michael Rosemann, Dr Ross Brown and Professor Marcello La Rosa for their

helpful comments on my research. I was privileged to have Professor Ron Weber as a panel

member of my PhD final seminar. I appreciate your detailed and constructive feedback. A

debt of gratitude also to Dr Artem Polyvyanyy, Professor Alistair Barros and Tim Dreesen for

their support in data collection. I also received helpful advice from participants at the ICIS

Doctoral Consortium and anonymous editors and reviewers of the studies included in this

thesis – thank you all.

I would also like to thank professional editor, Kylie Morris, who provided copyediting

and proofreading services, according to university-endorsed guidelines and the Australian

Standards for editing research theses.

During these years in Australia, I had the opportunity and pleasure to meet, interact, and

work with colleagues, both in the IS School and the Business School that had a positive

influence on my work and my life. There were also friends who made this journey more

enjoyable. I would like thank you for being in my life and for making my work-life easier and

extraordinary.

A big thank you to my family, who has been with me all along. Thanks to my late father

for everything he taught me. Thanks to my mother, my siblings, and the rest of the clan for

A Study of The Combined Use of Conceptual Models xiii

their unconditional love and their continued support in every possible aspect of my life. Your

encouragement and motivation are what has sustained me thus far.

Last, but not least, I would like to express my appreciation to my beautiful partner,

Farzaneh. Thank you for your unwavering love, admirable support, understanding, patience,

and encouragement.

Chapter 1:Introduction 1

Part A: Preface

Chapter 1: Introduction

1.1 INTRODUCTION

Conceptual models are simplified visual representations of the real world and are used

by humans to make sense of the world. The human mind conceptualises subjects by

constructing concepts and mapping them together as diagrams (Ausubel, 1968; Nesbit &

Adesope, 2006). Although the term “conceptual models” was developed approximately 40

years ago (P. P.-S. Chen, 1976), visual representations and mapping concepts as diagrams

have been used for learning and communication since at least the 13th century (Sowa, 2000).

Information systems analysts and designers often develop conceptual models to

represent the relevant features of a real-world domain (Gemino & Wand, 2004; Siau & Tan,

2005). Information systems are socio-technical systems developed based on an understanding

of a certain domain to support organisational activities in that domain (Bera, Burton-Jones, &

Wand, 2014; Wand & Weber, 1995). Conceptual models play a significant role in analysing

and promoting users’ knowledge of the requirements of the application domain for the

analysis and design of information systems (Gemino & Wand, 2004; Siau & Tan, 2005).

A conceptual model is developed using a modelling grammar1 (Wand & Weber, 2002),

which is a set of constructs and rules that demonstrate how to combine the constructs to

model real-world domains. There are many different types of conceptual modelling

grammars, which can be used for different purposes and organisational objectives (Fettke,

2009). For instance, process models are developed to describe organisational activities, and

business operations (Petri, 1962; Scheer & Nüttgens, 2000), while data models represent data

structures of systems (P. P.-S. Chen, 1976). Figure 1.1 represents only a few types of models

developed using different modelling grammars.

1 In practice, conceptual modelling grammars may also be referred to as modelling languages or techniques.

2

Check the order

Issue bill

Check quality

Ship the product

Not

in g

ood

qual

ity o

r dam

aged

BPMN

Check the order

V

Issue billXOR

Check the product

XOR

V

Start shipping

Not good quality or damaged

Good quality

EPC

ERD

Orders

-customerName

-dateCreated-dateShipped

-orderID

Customer

-address-customerName

User

-password-UserID

Administrator

-adminInfo-adminName

-email-contactdetails-shippinginfo

1..1

0..*

UML

Shop

Process order

Shop

Management information

Shop

Maintain rates

Customer

Discount rates

Product

Customer

Staff

2

1

3

D3

D2

D1

P1

T1

T2

T3

P2

P3

P4

DFD Petri Nets ORM YAWL

Figure 1.1 Examples of different models developed using various conceptual modelling grammars

The decision regarding which models to use requires an upfront decision about which

type of model to develop. It also requires substantial investment to develop those models.

Organisations spend a significant amount of time and money on conceptual modelling, such

as purchasing appropriate tools, licensing, training of employees, and developing

organisational conventions to create and implement the models (Recker, Indulska, Rosemann,

& Green, 2010; Rosemann, 2006). However, practitioners believe that the plethora of

modelling grammars and the availability of different methods to use these grammars are

major barriers to making investment decisions, and therefore increase the risk of investment

in appropriate modelling initiatives (Fettke, 2009). Therefore, knowledge regarding which

types of models to use is instrumental in facilitating well-founded decisions in organisations

and well-informed investments in conceptual modelling initiatives.

This knowledge about which types of models to use is also important given the high

failure rate of information systems projects. More than 60% of IT/IS projects fail to satisfy

their users or fail to meet deadlines (Palvia, Jacks, & Brown, 2015). Researchers recognise

that the cost of fixing errors grows exponentially as a function of elapsed time to discovery

(Shanks, Moody, Nuredini, Tobin, & Weber, 2010). IS failures in Australia include the


Queensland Health payroll debacle, where reports reveal that approximately $210 million and

18 months were needed to fix the system, while the original system development cost was $65

million (Glass, 2013). Conceptual models play a significant role in facilitating early detection

and correction of system development errors, such as design flaws or misunderstanding

between users and developers (Liddle, 2011; Maes & Poels, 2007; Wand & Weber, 2002).

This is particularly true when dealing with large and complex domains, as using appropriate

models improves understanding of the domain and its requirements, and therefore contributes

to a high-quality information system.

1.2 PROBLEM STATEMENT

System analysts and designers normally use more than one conceptual model in most

system analysis and design tasks (Fettke, 2009; Whittle, Hutchinson, & Rouncefield, 2014).

In fact, practitioners use multiple models, often developed using different grammars, to better

represent and understand the system under study. Although using multiple models assists

practitioners to better understand a domain (da Silva, 2015; J. Kim, Hahn, & Hahn, 2000;

Siau & Lee, 2004), it introduces problems, such as challenges in integrating different models,

inconsistency between models, or disproportionate or incomplete representation (e.g., more

details on some parts than others) (Lange, Chaudron, & Muskens, 2006; Mussbacher et al.,

2014; Petre, 2013).

Given the importance of conceptual modelling in system analysis and design (Bera et

al., 2014; Burton-Jones, Wand, & Weber, 2009; Wand & Weber, 2002), it is expected that IS

literature provides practical and theoretical guidance to explain the use of multiple and

different types of models. However, IS literature is inconsistent with practice. While empirical

evidence suggest that practitioners often use multiple models in their analysis and design

tasks (Fettke, 2009; Hutchinson, Whittle, Rouncefield, & Kristoffersen, 2011), the vast

majority of relevant research on conceptual modelling has focused on the use of single

modelling grammar or model. By contrast, research on the use of multiple models is very

sparse. To address this problem, Recker (2014) proposed a theory of faithful use of

4

conceptual model combinations2 to predict and explain the combined use of models.

However, this theory has not yet been tested.

In addition to the lack of research in IS literature to explain how or why practitioners

use multiple models, the ever-changing world also stresses the importance of reviewing and

revising our own research efforts in conceptual modelling (Recker, 2015). Similar to other

fields, conceptual modelling research is rightfully under constant scrutiny in terms of its

validity, applicability, relevance, and utility in the ever-changing world. New requirements

have emerged that go beyond the boundaries of traditional information systems. For example,

open information environments (Parsons & Wand, 2014) extend information systems beyond

organisational boundaries. Open information environments provide opportunities for users to

use and generate new information where organisations may have no control over structure and

content. In these environments, new sources and new uses of data may emerge (Lukyanenko

et al., 2017). In addition, the rapidly changing IS field creates new demands (Storey & Song,

2017) and novel forms of representations that challenge assumptions of traditional conceptual

modelling (Lukyanenko & Parsons, 2013; Recker, 2015).

Therefore, this research is set out in a multi-method research program to make

contributions to the body of knowledge on conceptual modelling in a several ways: first,

through a systematic literature review, this thesis shapes the current understanding of

conceptual modelling and stimulates a forward-looking wave of conceptual modelling

research. Second, through an exploratory study and a theory-testing study, this research

provides empirical insights to explain the use of model combinations and refines and

improves the theory of faithful use of conceptual model combinations. These contributions are

positioned in three individual but related studies (see Table 1.1): one systematic literature

review, with the primary focus on the existing studies on conceptual modelling and proposing

a research framework that proposes opportunities for future studies in the field; one

qualitative study, with the primary focus on exploring the use of multiple conceptual models

in practice; and one laboratory experiment, with the primary focus on testing and extending

the theory of faithful use of conceptual model combinations.

2 See also Section 2.4 for more details on the theory


Table 1.1 Papers included in this dissertation

Relevant

Chapters Study Type

4

Conceptual Modelling Research in Information Systems: What we know now

and what we still do not know. To be submitted to Information Systems

Research

A short version of the paper was published as: Jabbari Sabegh, M. A., Lukyanenko, R., Recker, J. C., Samuel, B., & Castellanos, A. (2017).

Conceptual modeling research in information systems: What we now know and

what we still do not know. Paper presented at the Symposium on Research in

Systems Analysis and Design (AIS SIGSAND 2017).

Systematic

Literature review

5

Combined Use of Conceptual Models in Practice: An Exploratory Study

Published as: Jabbari Sabegh, M. A., & Recker, J. (2017). Combined Use of

Conceptual Models in Practice: An Exploratory Study. Journal of Database

Management, 28(2), 56-88

Qualitative

6

An Experimental Study to Test and Extend the Theory of Faithful Use of

Conceptual Model Combinations. To be submitted to MIS Quarterly

An initial version of the paper was published as: Jabbari Sabegh, M. A., Recker, J., & Green, P. (2016). Designing Experiments to Test the Theory of

Combined Ontological Coverage. Paper presented at the International

Conference on Information Systems, Dublin, Ireland

Quantitative

1.3 RESEARCH QUESTIONS

The three studies in this thesis aim to answer three research questions. The first research

question is a general question regarding the current state of knowledge about conceptual

modelling. Conceptual modelling is regarded as a core activity of information systems

analysis and design. Research in this field is an active research area, with the aim of

improving both the process and the outcome of conceptual modelling. In 2002, Wand and

Weber proposed a seminal research agenda to motivate research addressing the question of

“How can we model the world to better facilitate our developing, implementing, using, and

maintaining more valuable information systems?” (Wand & Weber, 2002, p. 363). After 15

years of raising this question, the first research question in this study is:

Research Question 1: What is the current state of knowledge of conceptual modelling

in IS?

This research question aligns with Study 1 (Chapter 4). To answer this question, an

understanding of the current state of knowledge in conceptual modelling and addressing new

perspectives of conceptual modelling is required. This involves a systematic literature review

of existing studies about conceptual modelling in IS to identify the aspects of conceptual

6

modelling addressed by existing studies, the extent of the contributions of these studies, and

the aspects of conceptual models that have not yet been addressed. In doing so, this thesis

contributes to the body of knowledge in conceptual modelling by synthesising the current

understanding and assumptions about conceptual modelling and providing substantive

directions for future research.

The second and third research questions focus on the combined use of conceptual

models. Empirical evidence suggests that practitioners use model combinations in their

systems analysis and design tasks. While Recker (2014) provided a theoretical explanation for

the combined use of conceptual models, relatively little is currently known about the nature of

the use of multiple models in practice. Therefore, the second research question is:

Research Question 2: Why do systems analysts and designers use multiple conceptual

models?

The second research question aligns with Study 2 (Chapter 5). To answer this question,

an exploratory study was required to understand the current practices in the actual use of

model combination. By answering this question, this thesis contributes to theory in a number

of ways: First, it makes an important contribution to the literature on conceptual modelling.

To the best of my knowledge, this study presents the first empirical investigation into the

actual use of model combinations in practice. Second, the study explores factors that impact

the use of conceptual models and suggests how existing theories might be improved to

provide better explanations about the use of models in practice.

The third research question of this thesis follows the findings from the exploratory

study, with the primary focus on the propositions of the theory of faithful use of conceptual

model combinations. This theory provides three propositions to evaluate the combined use of

conceptual models in terms of user’s selection, understanding, and perceived usefulness of

model combinations. Therefore, the third research question is:

Research Question 3: What characteristics of models affect users’ selection,

understanding, and perceived usefulness of model combinations?

This research question aligns with a theory-testing study (Study 3, Chapter 6). A

controlled investigation was required to investigate the exact impact of proposed factors on

users’ selection of models, generation of domain understanding from multiple models, and


perceived usefulness of model combinations. Therefore, a laboratory experiment was set to

control extraneous variables and manipulate the causes. Study 3 makes a number of

significant contributions. First, to the best of my knowledge, it is the first dedicated empirical

study set to evaluate the propositions of the theory of faithful use of conceptual model

combinations. The results of this study provide empirical evidence regarding the propositions

of the theory and extend the boundaries of the theory. Second, the findings of this study

provide practical guidance for both model users and designers to use and develop appropriate

combinations of conceptual models.

1.4 SCOPE OF STUDY

Research on conceptual modelling appears prolifically in academic journals. To scope

this research, Study 1 focuses on the papers published in the Association for Information

Systems (AIS) basket of eight journals (F. Liu & Myers, 2011), plus the Journal of Database

Management. These journals were selected because the AIS basket of eight journals are

considered by the academic community as appropriately representative for the mainstream

research in the field by the academic community and the Journal of Database Management

has traditionally been one of the leading substantive journals publishing studies about

conceptual modelling. However, the literature review is a continuing process, and the

literature review sections of Studies 2 and 3 are independent and not limited to the papers

reviewed in Study 1.

System analysis and design tasks typically have many different stakeholders, such as

top management, project management, system analysts, system designers, and system end-

users. Conceptual modelling during the systems analysis and design process has two phases:

model creation, which applies to those who develop models; and model interpretation, which

applies to those who read the models to understand the domain represented by models

(Gemino & Wand, 2004). These two phases are not often mutually exclusive. Stakeholders

can be a model user or both a model creator and a user. In either case, stakeholders read the

models and apply them in support of their tasks. A model creator may read models for the

purpose of redesigning and modification. The scope of Studies 2 and 3 was to investigate the

combined use of models at the individual level and from a model user’s perspective, as

opposed to model creators.

8

Information systems analysts and designers may use various types of conceptual

models. At least 40 different modelling grammars (languages) are cited in a current

practitioners’ survey (Whittle et al., 2014). This thesis focusses on Unified Modelling

Language (UML) and users of UML models. Previous studies (Davies, Green, Rosemann,

Indulska, & Gallo, 2006; Fettke, 2009) indicate that UML is among the most common

conceptual modelling techniques used in systems analysis and design. Object-oriented

modelling is often promoted as an ideal option to solving problems in software developments,

and UML has achieved significant popularity as the standard for object-oriented system

development (Batra, 2009; Dobing & Parsons, 2008). In addition, UML offers a variety of

conceptual modelling constructs to support an object-oriented approach to system analysis

and design (Gemino & Parker, 2009). Moreover, a large number of practitioner articles and a

substantial body of research on UML has emerged (Evermann & Wand, 2001; Petre, 2013),

such as evaluation of the effectiveness of various UML models (Siau, 2010), and analysis of

the language’s complexity (Siau, Erickson, & Lee, 2005).

1.5 THESIS STRUCTURE

This thesis is organised into three parts. Part A provides the preface and an overview of

the studies. Part B presents the main body of this thesis, which includes three studies. Part C

provides the summary of the findings from the three studies.

1.5.1 Part A

Part A is structured into three chapters.

- Chapter 1 provides an introduction to this thesis and introduces the motivations

of the study, followed by the research questions and scope of the research.

- Chapter 2 provides the background of the research on conceptual modelling in

information systems and then discusses the theoretical foundation of the thesis.

- Chapter 3 outlines the research methods implemented in the three studies. The

reliability and validity of the selected research methods are then discussed.

1.5.2 Part B

Part B presents the three studies that were conducted to answer the three main research

questions of this thesis. Figure 1.2 illustrates the three studies as the main body of the thesis,


corresponding to the research questions. In Part B of this thesis, the term “we” is used

because the three studies were conducted in collaboration with the co-authors of the studies.

Thesis title:

A Study of the Combined Use of Conceptual Models

What is the current state of knowledge

of conceptual modelling in IS?

Semi-structured interviews

What characteristics of models affect

users’ selection, understanding, and

perceived usefulness of model

combinations?

Study 1:

Why do systems analysts and designers

use multiple conceptual models?

Experiment

Research Question 1 Research Question 2 Research Question 3

Literature review

Study 2: Study 3:

Provided an overview of the

current state of knowledge in

conceptual modelling

Motivated further research on


Explored how practitioners

reportedly use conceptual

models

Extended implications of

ontological theories in


Tested and extended the theory

of faithful use of conceptual

model combinations

Developed an understanding on

the use of model combinations

Provided results from an extensive investigation into the combined use of conceptual models , and it

attempts to validate and extend the theory of faithful use of conceptual model combinations.

Contribution of Study 1: Contribution of Study 2: Contribution of Study 3:

Contribution of the research:

Figure 1.2 Outline of thesis structure and research objectives

- Study 1 (Chapter 4) reviewed existing research on conceptual modelling in IS.

In doing so, the research on conceptual modelling published in high-quality IS

journals was reviewed. The objective of this research was to provide an

understanding of what is already known and what is unknown in conceptual

modelling. This paper highlights the research areas in conceptual modelling that

appear to be neglected and provides a new framework that draws the attention of

conceptual modelling research to future research opportunities. However, the

literature review is a continuing part of the research program in all stages of the

research to express the link of the current research with the previous work and

contributions of current research to the knowledge in the area (Neuman, 2006).

10

- Study 2 (Chapter 5) conducted an exploratory study to understand why

practitioners use different types of conceptual models in combination. From the

findings of Study 1, it became clear that there is a lack of knowledge in the IS

literature to explain the use of multiple models in system analysis and design

tasks. However, in practice, practitioners use more than one model in their

design and analysis tasks (Dobing & Parsons, 2008; Fettke, 2009). On the other

hand, there are arguments that challenge the use of models in requirements

gathering in system analysis and design (B. Ramesh, Cao, & Baskerville, 2010).

The different viewpoints highlight that the use of multiple models is a poorly

understood phenomenon. Therefore, semi-structured interviews were conducted

to explore why IS professionals use conceptual models in practice.

- Study 3 (Chapter 6) was a theory-testing study. The findings of Study 1

highlighted the lack of knowledge in IS literature describing the use of multiple

models. Study 2 indicated that similar to predictions of the theory of faithful use

of conceptual model combinations, ontological factors have potential impacts on

using different types of conceptual models. The theory of faithful use of

conceptual model combinations provides explanations about how ontological

factors may affect model selection, domain understanding, and perceived

usefulness of model combinations. However, there are no empirical insights to

explain the phenomena. Study 3 aimed at investigating the use of multiple

models. An experimental study was conducted to examine the propositions of

the theory of faithful use of conceptual model combinations.

1.5.3 Part C

Part C of this thesis provides a summary of the findings from three studies.

- Chapter 7 synthesises the findings, and discusses the implications that stem

from the combination of the three individual studies.

- Chapter 8 discusses the substantive and theoretical contributions of this

research. The limitations that pertain to this thesis and the outlines of future

research opportunities are then presented.

Chapter 2:Research Background 11

Chapter 2: Research Background

2.1 CONCEPTUAL MODELLING IN INFORMATION SYSTEMS

Research on conceptual modelling has gained widespread attention as the basis for

information systems analysis and design (Moody, 2005). Information systems are developed

to support organisational activities in a certain domain based on an understanding of that

domain (Bera et al., 2014). Such an understanding is supported by developing a conceptual

model independent of the technology used to construct the system (P. P.-S. Chen, 1976;

Denning, 2003).

Conceptual models are often graphical representations of human perceptions of the

semantics of a real world domain. These models are normally developed in such a manner

that they can be represented in an information system (Kent, 1978; Wand & Weber, 2002).

Conceptual modelling involves capturing various aspects of the real world domain to promote

an understanding of the conceptual aspects of a domain to be supported by the information

system prior to designing and programming the system (Bera et al., 2014; Burton-Jones et al.,

2009). However, it also extends to further application areas, such as supporting

communication between users and analysts, or analyst and developers; providing input for the

design process; and documenting the original requirements for future reference (Siau, 2004;

Wand & Weber, 2002).

The field of conceptual modelling was founded approximately 40 years ago (Storey,

Trujillo, & Liddle, 2015). Since then, research on conceptual modelling has actively

investigated conceptual modelling related phenomena and evaluated capabilities of conceptual

models (Burton-Jones et al., 2009; Maes & Poels, 2007). Researchers argue that using high-

quality conceptual models in the early stages of information systems analysis and design

processes can enable early detection and correction of errors (Wand & Weber, 2002). The

relatively high failure rate of information systems highlights the importance of conceptual

modelling in systems analysis and design (Dwivedi et al., 2015).

Conceptual modelling consists of a grammar (i.e., a set of constructs and rules that show

how to combine the constructs to model a real-world domain), a method (i.e., provide

12

procedures that explain how to use a grammar), a script (i.e., the product of the modelling

process), and a context (i.e., the setting in which conceptual modelling occurs, and scripts are

used) (Wand & Weber, 2002). For example, entity-relationship modelling grammar (P. P.-S.

Chen, 1976) provides constructs such as entity, relationship, and attribute. Rules in the entity-

relationship modelling grammar specify that an entity can have an attribute and two entities

can be associated only through a relationship.

Research on conceptual modelling has focused on evaluating capabilities of existing

conceptual modelling grammars and methods and developing new ones. Different types of

conceptual modelling grammars and methods have been developed for multiple reasons. First,

different types of models are used for different purposes. For example, data models represent

the data structures of systems (P. P.-S. Chen, 1976), process models represent the operations

and dynamics of systems (Petri, 1962), and enterprise models represent the structure of an

organisation (Checkland & Scholes, 2000). Second, different models have been developed

that claim to be superior to previous ones. For example, different types of process modelling

grammars, such as Business Process Modelling Notation (BPMN) grammar (OMG, 2011a),

Event-driven Process Chains (EPCs) (Scheer & Nüttgens, 2000), Yet Another Workflow

Language (YAWL) (van der Aalst & ter Hofstede, 2005), or UML-Activity (OMG, 2011b).

Thirdly, different approaches to system analysis and design emerged. For example, object-

oriented modelling to describe a system in the forms of objects (Booch, 1986) or object-

process modelling to describe both the structure and the operations of the system by

representing blocks object and process (Dori & Goodman, 1996), just to name a few. Still, the

ever-changing world requires constant improvement of modelling methods and innovative

approaches as conceptual modelling moves beyond traditional boundaries (Lukyanenko &

Parsons, 2013; Lukyanenko, Parsons, & Samuel, 2015; Recker, 2015), for example, flexible

database technologies (i.e., noSQL databases) (Kaur & Rani, 2013) and instance-based

conceptual modelling (Lukyanenko et al., 2015).

Traditional research on conceptual modelling can be categorised into two streams of

research viz. research on conceptual model design and research on conceptual model

interpretation (Gemino & Wand, 2004). Research on conceptual model design focuses on the

capabilities of modelling grammars, how to use grammars and the context in which

conceptual modelling occurs from a model designer (model creator) perspective. On the other


hand, research on conceptual model interpretation focuses on the capabilities of developed

models and how a model user (model viewer) reads and understands a developed model.

These two types of research about conceptual modelling are not mutually exclusive,

because, first, the capability of a model (script) is constrained by the capabilities of the

modelling grammar used to develop that model. Second, researchers normally evaluate the

capabilities of modelling grammars based on developed scripts (i.e., how understandable is

the model?) (Aguirre-Urreta & Marakas, 2008; Burton-Jones et al., 2009; Gemino & Wand,

2004; Parsons & Cole, 2005; Siau, 2004; Wand & Weber, 2002). The dominant objective of

research on conceptual modelling is to improve the conceptual modelling process and

outcomes (Siau & Rossi, 2011). Improving the quality of conceptual models is also likely to

improve the quality of developed systems (Moody, 2005). The next section briefly reviews

the research on single modelling grammars and models.

2.2 RESEARCH ON SINGLE GRAMMARS OR MODELS

2.2.1 Conceptual Model Design

Research on conceptual model design has widely focused on the evaluation of

capabilities of grammars and methods (e.g., Bera, 2012; Bera, Burton-Jones, & Wand, 2011)

and developing new ones (e.g., Loucopoulos & Kadir, 2008; Lukyanenko et al., 2017).

Researchers proposed several guidelines to evaluate conceptual modelling grammars and

scripts (e.g., Burton-Jones et al., 2009; Gemino & Wand, 2003). For many years, the leading

approach in this area has relied on Bunge, Wand, and Weber (BWW) representation theory

(Wand & Weber, 1990a, 1993, 1995).

The BWW representation theory developed from an adaptation of an ontology proposed

by Bunge (1977). In general, ontology explains the nature of the real-world, the interaction

between what exists in reality, and describes the properties and structures of real-world things

(Bunge, 1977). Since information systems are representations of real-world domains that are

essentially created by humans, the representation theory based on ontology may lead

modellers to conceive of conceptual structures to represent a real-world domain (Wand &

Weber, 1993, 1995).

The BWW representation theory suggests that conceptual modelling grammars must be

able to represent all features of the real-world domain that the information systems users are

14

interested in. Otherwise, the created models based on the grammars would be incomplete, and

the designers/analysts would have to somehow augment the models (Weber, 1997). The

theory proposes four possible ontological deficits: construct deficit, construct overload,

contract redundancy, and construct excess.

Construct deficit: an ontological construct does not map into any grammatical

construct (a 1:0 mapping).

Construct overload: one grammatical construct maps into several ontological

constructs (a m:1 mapping).

Construct redundancy: two or more grammatical constructs map into one

ontological construct (a 1: m mapping).

Construct excess: a grammatical construct does not map into any ontological

construct (a 0:1 mapping).

Research on conceptual modelling based on BWW theory has mostly focused on

ontological completeness (i.e., the existence of construct deficit) and ontological clarity (i.e.,

the existence of construct overload, redundancy, and excess) (Wand & Weber, 1995) of

modelling grammars and scripts.

Three dominant findings from ontological studies are highlighted below:

Any single modelling grammar, and therefore any single model, has some level

of ontological deficit (e.g., Green & Rosemann, 1999; Irwin & Turk, 2005;

Opdahl & Henderson-Sellers, 2002; Recker et al., 2010; Wand & Weber, 1989,

1993; Weber, 1996; Weber & Zhang, 1996).

Following ontological guidelines improves the conceptual modelling process

and outcome (e.g., Bowen, O'Farrell, & Rohde, 2006; Milton, Rajapakse, &

Weber, 2012; Saghafi & Wand, 2014; Shanks, Tansley, & Weber, 2004).

The existence of ontological deficits affects the use of conceptual modelling

grammars and models (e.g., Recker, Rosemann, Green, & Indulska, 2011).

Research into conceptual modelling grammars is not limited to ontological evaluations.

Researchers have gone beyond the ontological capabilities of grammars and proposed other

quality measurements and guidelines to develop high-quality conceptual models. For


example, Clarke, Burton-Jones, and Weber (2016) proposed that ontological evaluation of

modelling grammars should be coupled with logical evaluation. Other examples of quality

measures are syntactic quality (i.e., how well all statements in the model correspond to the

syntax of the grammar), and semantic quality (i.e., how well all statements in the model

correspond to the domain), pragmatic quality (i.e., how well is the correspondence between

the model and the audience’s interpretation of the model defined), and perceived semantic

quality (e.g., Krogstie, 2001; Krogstie, Sindre, & Jørgensen, 2006; Recker, 2007). Similarly,

Mendling, Reijers, and van der Aalst (2010) proposed seven guidelines to develop high-

quality conceptual models, or activity labelling (Mendling, Reijers, & Recker, 2010).

In developing conceptual models, researchers have also investigated the contextual

factors affecting the modelling process. For example, studies uncovered factors such as

organisational conventions (Delfmann, Herwig, Lis, & Stein, 2015; Recker et al., 2010;

Rosemann, 2006; Samuel, Watkins III, Ehle, & Khatri, 2015), and tool supports and tool

functionality (Fettke, 2009; La Rosa et al., 2011; Recker, 2012) as influential factors on

conceptual modelling processes. However, the findings uncovered that the availability of tools

does not always lead to producing high-quality conceptual models (Poppe, Brown, Recker,

Johnson, & Vanderfeesten, 2017; Recker, Mendling, & Hahn, 2013).

2.2.2 Conceptual Model Interpretation

Research on conceptual model interpretation has focused on both model related factors

and contextual factors, such as cognitive capabilities and individual difference factors. Similar

to research on modelling grammars, research on model interpretation has evaluated the impact

of ontological capabilities of models on model understandability. Examples of this research

include the impact of ontological clarity on model understandability (e.g., Allen & March,

2006; Bodart, Patel, Sim, & Weber, 2001; Bowen et al., 2006; Shanks, Tansley, Nuredini,

Tobin, & Weber, 2008). The findings of these studies are equivocal. Some findings have

revealed that ontological clarity has not always led to better understanding or does not have a

clear effect on model user’s performance (e.g., Bowen, O'Farrell, & Rohde, 2009; Fickinger

& Recker, 2013).

Examples of research beyond the ontological capabilities of models are the effects of

perceptual discriminability and semiotic clarity of different process models on model

16

understandability (Figl, Mendling, & Strembeck, 2013), the effect of perceptual

discriminability of gateway constructs on a user’s abilities to interpret process models

(Recker, 2013), the effects of personal, model, and content-related factors (Mendling &

Strembeck, 2008), or the effects of the interactivity of model elements and presence of a

different control flow (Figl & Laue, 2015).

In addition to the capabilities of a model stemming from the capabilities of a grammar,

researchers have identified additional aids, such as visual cues (J. Kim et al., 2000), text

(Gemino & Parker, 2009), customisation (Samuel et al., 2015), colours (Masri, Parker, &

Gemino, 2008; Reijers, Freytag, Mendling, & Eckleder, 2011), and routing symbols (Figl,

Recker, & Mendling, 2013) that can influence model interpretation.

Researchers have also found contextual factors and individual differences to be

influential factors on a user’s ability to understand a model (Figl, 2017). Examples of this

stream of research are the effects of cognitive style and task characteristics, cognitive abilities,

and learning style (Figl & Recker, 2014; Recker, Reijers, & van de Wouw, 2014), and IS and

domain knowledge (Bera et al., 2014; Khatri & Vessey, 2016; Khatri, Vessey, Ramesh, Clay,

& Park, 2006) on an individuals’ ability to interpret a model.

2.3 RESEARCH ON MULTIPLE GRAMMARS OR MODELS

Unlike research on single grammars or models, research on multiple grammars or

models is sparse. Conceptual modelling can take many forms. With the availability of

different modelling grammars and methods, empirical evidence suggests that practitioners

normally use multiples in combination in their systems analysis and design tasks (Davies et

al., 2006; Fettke, 2009; Green, Rosemann, Indulska, & Recker, 2011; Petre, 2013; Recker,

2012; Whittle et al., 2014).

Ontological theories predict how multiple grammars can be used in combination

(Green, 1996; Weber, 1997). Based on an extension of the BWW representation theory, Green

(1996) argued that the combination of multiple modelling grammars should achieve Maximal

Ontological Completeness (MOC) (i.e., a combination of grammars that maximises

grammatical constructs that cover all phenomena to be modelled) and Minimal Ontological

Overlap (MOO) (i.e., a combination of grammars that minimises overlapping grammatical

constructs to represent real-world phenomena). A few studies then investigated which


combinations of grammars model designers might select. For example, Green, Rosemann,

Indulska, and Manning (2007) analysed multiple grammars to define systems interoperability

and examine which combinations of grammars could theoretically achieve the best coverage.

zur Muehlen and Indulska (2010) investigated the appropriate combination of business rules

and business process modelling grammars, and Green et al. (2011) analysed how users

employed multiple modelling grammars in combination to model a system in a CASE tool.

Few studies have investigated the use of multiple models. The existing studies in this

area found that understanding of multiple models is facilitated by incorporating visual cues

and contextual information (J. Kim et al., 2000). Use case and class diagrams have very little

overlap in the information captured, and both are perceived as necessary for requirement

analysis (Siau & Lee, 2004), and using use cases in combination with use case diagrams

results in better performance (Gemino & Parker, 2009). These studies indicate that the

combined use of models improves domain understanding. However, these findings did not

indicate which attributes and factors define the best combination of models. Recker (2014)

proposed a theory of faithful use of conceptual model combinations to explain the factors that

may affect the combined use of conceptual models. Recker’s (2014) theory is explained in

more detail in the following section.

2.4 THE THEORY OF FAITHFUL USE OF CONCEPTUAL MODEL

COMBINATIONS

Practitioners normally use several types of conceptual models or grammars during

information systems analysis and design processes (Dobing & Parsons, 2008; Fettke, 2009;

Recker, 2012). Empirical evidence has suggested that practitioners do not substitute models,

they use combinations of models developed using various grammars (Green et al., 2011;

Petre, 2013; Recker, 2012). However, research on conceptual modelling has focused almost

exclusively on single grammars or models and largely neglected this use of combined models.

Recker (2014) developed the theory of faithful use of conceptual model combinations,

which suggests propositions to explain and predict the use of multiple conceptual models in

combination. Recker (2014) indicated that model use occurs as part of a particular task at

hand. Model users are those individuals who read and apply models in support of the task at

hand (Gemino & Wand, 2004). Model users also refer to model readers or model interpreters

as opposed to those who develop models. The use of models involves reading models to

18

develop an understanding about the presented domain in support of the task at hand.

Therefore, the faithful use of models by an individual in a particular task will be evaluated in

light of the extent to which the individual was able to accomplish the task (Recker, 2014).

Consequently, Recker (2014) indicated that the faithful use of conceptual models can be

evaluated in the three steps of engaging in a particular task:

1) Prior to engaging in a model-based task: can be captured as the perceived

performance model-users expected by selecting a set of conceptual models for a

specific task from different types of models.

2) During the engagement in the model-based task: evaluate the performance based

on the extent of domain understanding learned from using conceptual models.

3) After engaging in the model-based task: evaluate the beliefs about the performance

based on the extent of domain understanding learned from using conceptual models.

The primary conjecture of the theory of faithful use of conceptual model combinations

is that the user’s selection of a set of models, the domain understanding they gain from

multiple models, and the perceived usefulness of multiple models will be dependent on the

models’ combined ontological completeness and ontological overlap. Combined ontological

completeness refers to the level of representational coverage a set of models provides about a

focal real-world phenomenon. Ontological overlap refers to the set of redundant

representation across a set of models representing real-world phenomena (Recker, 2014).

Figure 2.1 illustrates the two premises of the theory of faithful use of conceptual model

combinations.

Figure 2.1 Premises of the Theory of Faithful Use of Conceptual Model Combinations (Recker, 2014)

Completeness of

domain representation

of model A

Completeness of


of model B

Resulting

ontological overlap

between models A

and B

Remaining construct

deficit of combined

representation


The basic rationale behind this conjecture is that, first, users will select multiple models

to maximise their coverage of some relevant real-world phenomena (Green, 1996). This is

because ontological evaluations of modelling grammars show that any single modelling

grammar suffers from construct deficit (Green et al., 2011; Irwin & Turk, 2005; Siau, 2010).

In other words, no one grammar can be used to model all potentially relevant real-world

phenomena. Moreover, often, not all constructs provided via a grammar are used in any given

model (Siau et al., 2005; zur Muehlen & Recker, 2008). Therefore, users will use multiple

models in combination to overcome ontological deficits in a single model and to maximise

combined ontological completeness through multiple models.

Second, users will also seek to minimise ontological overlap in the representation to

keep the coverage as simple and clear as possible (Wand & Weber, 1990a, 1990b, 1993).

According to Green (1996), the primary intention to select multiple grammars is to reach

maximum ontological completeness. However, some studies suggest that a simpler

representation may be more useful than a more complete representation (Siau & Lee, 2004;

zur Muehlen, Recker, & Indulska, 2007).

On the basis of these two main factors, the theory suggests three propositions that detail

the faithful use of multiple models:

Model selection: model users will select a combination of conceptual models to

maximise combined ontological completeness; however, they will select additional

models until the model combination reaches a particular level of overlap that users

perceive to be bearable.

Domain understanding: the level of understanding can be increased by presenting

more elements of a domain that are represented in conceptual models (ontological

completeness); however, this level of understanding will be moderated by the level

of ontological overlap among models.

Perceived usefulness: Perceptions of the usefulness of model combinations will

increase when the ontological completeness of the achieved representation

increases; however, a combination of models with high ontological overlap, that is,

high redundant constructs, will be evaluated as less useful.

20

2.5 CONCLUSION

Three main points are evident from the research on conceptual modelling in information

systems and the theory of faithful use of conceptual model combinations.

First, apart from a few studies (Dobing & Parsons, 2008; Fettke, 2009; Gemino &

Parker, 2009; Green et al., 2007; Green et al., 2011; J. Kim et al., 2000; Siau & Lee, 2004),

most studies on conceptual modelling have focused on using single grammars or models.

Research on conceptual modelling is motivated by evaluating single grammars (Green et al.,

2007; Krogstie, 2001; Maes & Poels, 2007; Opdahl & Henderson-Sellers, 2002; Wand &

Weber, 1989), different conceptual modelling methods (Siau & Rossi, 2011), as well as the

performance of users of a single model (Allen & March, 2006; Bowen et al., 2006; Chan,

Wei, & Siau, 1993). Most of the studies on conceptual modelling have used Wand and

Webers’ (1990a, 1993, 1995) representation theory adapted from Bunge’s (1977) ontology as

their theoretical foundation to evaluate a single model or grammar.

Second, evaluation of conceptual modelling grammars revealed that almost any single

model has some level of construct deficits (Green & Rosemann, 1999; Irwin & Turk, 2005;

Opdahl & Henderson-Sellers, 2002; Recker et al., 2010; Wand & Weber, 1989, 1993; Weber,

1996; Weber & Zhang, 1996). Therefore, no single model would provide a complete

representation of a real-world domain. This is because the representational completeness of

any single model is limited to the extent of the level of construct deficit in any single

grammar. Moreover, the actual level of completeness of any single model is often less than

the level of completeness of the grammar used to develop the model, as most models do not

contain all grammatical constructs - only a small subset (zur Muehlen & Recker, 2008; zur

Muehlen et al., 2007).

Third, Recker’s (2014) theory proposed that the combined use of conceptual models

depends on two ontological factors of combined ontological completeness and ontological

overlap. Based on these factors, the theory then proposes how users may select models, how

domain understanding can be generated from the model combinations, and how the perceived

usefulness of model combinations may vary.

This thesis focusses on the use of multiple models and how conceptual models in

combination are interpreted and understood by users, as opposed to how people create or


develop models (see Gemino & Wand, 2004). Previous studies have highlighted the use of

model combinations (Fettke, 2009). Recker’s (2014) theory of faithful use of conceptual

model combinations provides a foundation and insights to predict and explain the use of

models in combination. The theoretical foundation of the current study is therefore built on

the principles of the theory of faithful use of conceptual model combinations, with the

objectives to test and extend the propositions of the theory.

22

Chapter 3: Research Methods

3.1 RESEARCH METHOD CHOICES

This section explains the research methodologies adopted to collect, analyse, and report

data on the use of conceptual model combinations. This research employed a multi-methods

design. The success of each stage of the research depends on the selection of appropriate

research methods based on the objectives of each stage and the state of knowledge in the area

being studied (Mingers, 2001). This research aims to answer three main questions.

The first research question of this thesis requires understanding of the current stage of

knowledge in the field; thus, a systematic literature review was carried out. An exploratory

study was conducted to answer the second research question. To answer the third research

question, an explanatory, theory-testing, approach to the research was undertaken.

IS researchers have used various types of research methods that can, at one level, be

broadly categorised into qualitative or quantitative methods (Lee & Hubona, 2009;

Venkatesh, Brown, & Bala, 2013). Both qualitative and quantitative methods can possibly be

used to answer similar research questions. However, qualitative methods have typically been

used to investigate little-known phenomena, and explore and develop new theories

(Eisenhardt, 1989; Walsham, 2006). On the other hand, quantitative methods are typically

explanatory studies and have been used to test theories.

3.1.1 Systematic Literature Review

In order to advance the field of conceptual modelling research in IS, it is useful to

explore the history of the field and to determine its current state by identifying, analysing, and

interpreting the relevant research to date that is related to conceptual modelling. To achieve

this objective, a rigorous and systematic approach was undertaken (Bandara, Furtmuller,

Gorbacheva, Miskon, & Beekhuyzen, 2015; Kitchenham et al., 2009; Webster & Watson,

2002) and a descriptive literature review was conducted as described by Paré, Trudel, Jaana,

and Kitsiou (2015) and Rowe (2014).

Chapter 3:Research Methods 23

In contrast to what is occurring in practice, there is a lack of knowledge in IS literature

to explain how and why practitioners use different types of conceptual models. Research on

conceptual modelling continues to be an active research area in information systems research

(Burton-Jones et al., 2009). Many aspects of conceptual modelling have been addressed

through ongoing research studies appearing in top IS journals. However, it appears that little

research has been undertaken into some other aspects of conceptual modelling. A systematic

literature review was therefore conducted to determine the current state of the knowledge in

the field and propose a framework to motivate further research on little-known aspects of

conceptual modelling.

3.1.2 Semi-Structured Interviews

To achieve the second objective of this study, that is, to explore how and why

practitioners use multiple models in combination, an exploratory, qualitative research design

was used. Qualitative research enabled the completion of a study in an area in which few

previous studies exist (Benbasat, Goldstein, & Mead, 1987; Myers & Newman, 2007; Yin,

2009). Specifically, this was achieved through the use of semi-structured interviews.

Interviews are a dominant data collection method in qualitative research techniques

(Yin, 2009), such as case study research or grounded theory studies (Myers & Newman,

2007). Semi-structured interviews can be used with any philosophical perspective, whether

positivist, interpretive, or critical (Myers & Newman, 2007), deductive or inductive (Shanks,

2002), descriptive, explanatory, or exploratory (Dubé & Paré, 2003). To ensure reliability and

validity, the guidelines of leading positivist qualitative researchers were applied (Shanks,

2002; Yin, 2009). An inductive approach was also used to explore theoretical concepts and

patterns from qualitative data (Bhattacherjee, 2012).

The notion that multiple grammars are used in practice has previously been explored,

firstly, by Weber (1997) and Green (1996). However, there is a little knowledge about the use

of conceptual models in combination. Recker (2014) formulated a theory of faithful use of

conceptual model combinations. However, what is notably absent is empirical evidence about

how practitioners work with multiple models. The existing theories predicted the use of

multiple models; however, use of multiple models is a poorly understood phenomenon. For

example, in addition to the characteristics of models, contextual factors and different

24

individuals’ involvement in model-based activities may affect how and why practitioners use

model combinations.

3.1.3 Experiment

To achieve the third objective of this study, that is, to test which characteristics of

models affect users’ selection, understanding, and perceived usefulness of model

combinations, an experimental study was adopted. Given the nature of this objective, the

experimental study was used to obtain preliminary evidence to explain and predict the spirit

of the use of model combinations, rather than only describing the phenomena. Experiments

have commonly been used in information systems research to build knowledge on system

analysis, design, development, and management (Palvia et al., 2004). The third objective of

this study is to test and extend the theory of faithful use of conceptual model combinations.

While this is the first study that aims to test the theory, the primary objective was to ensure

strong internal validity (Calder, Phillips, & Tybout, 1981).

Experimental research is one of the most rigorous of all research designs (Bhattacherjee,

2012). The main strength of experimental research is its internal validity due to its ability to

link cause and effect through treatment manipulation. An experimental design also provides

unique abilities in controlling external factors that might have spurious effects (Shadish,

Cook, & Campbell, 2002).

3.2 RELIABILITY AND VALIDITY

The reliability and validity of the research methods adopted for this thesis are discussed

in two parts. First, the reliability and validity considerations for the systematic literature

review as discussed, this is followed by a discussion of the reliability and validity of the semi-

structured interviews and the experiment.

3.2.1 Reliability and validity of systematic literature review

To ensure the reliability and the reproducibility of the literature review process, the

steps and approach used is clearly documented in the process of the systematic literature

review (Paré et al., 2015). Moreover, to ensure the reliability of the analysis, the multi-coder

approach was used to interpret the existing literature. Regarding the validity of the review

process, top IS journals were the focus, using multiple databases (Webster & Watson, 2002).


3.2.2 Reliability and Validity of Semi-Structured Interviews and the Experiment

Reliability and validity are important cornerstones to evaluate the quality of a research

design. Validation principles have been applied and suggested for both qualitative (Maxwell,

1992; Mertens, 2014) and quantitative research (Straub, Boudreau, & Gefen, 2004).

Researchers have reached general agreement with regards to validation principles and

processes of quantitative studies; however, when it comes to applying validation principles in

qualitative studies, researchers do not have any such agreement (Venkatesh et al., 2013).

However, the validity of both qualitative and quantitative studies are discussed in this

research based on the main validation principles; that is, internal validity, external validity,

construct validity, and reliability (Lee, 1991; Shanks, 2002; Venkatesh, Brown, & Sullivan,

2016; Yin, 2009). In general, reliability is discussed as stability and consistency of the

research data, and validity as the concerns with regards to whether the collected data in

different situations really measures what the researcher sets out to measure and leads to the

same conclusions (Shanks, 2002).

Internal Validity

Internal validity refers to the validity of the research design in quantitative studies,

which can refer to descriptive validity and credibility of the design of qualitative study

(Lincoln, Lynham, & Guba, 2011; Venkatesh et al., 2013). Internal validity concerns the

validity of assumptions about causal relationships between independent and dependent

variables (in a quantitative study) (Straub et al., 2004) and accuracy of reports of events,

objects, behaviours, and credibility of the explanations (in a qualitative study) (Maxwell,

1992).

Given the exploratory nature of the semi-structured interviews in this study, internal

validity was not considered a threat (Yin, 2009). However, to ensure the internal validity of

the semi-structured interviews for explanatory purposes, previous literature was used to

develop clear explanations for the definitions of terms to avoid confusion in understanding the

phenomena (e.g., a precise definition of a conceptual model). In addition, all interviews were

audio recorded and transcribed to be sure that all of the information that participants intended

to deliver in the interviews was collected. Finally, six pilot studies were conducted and

previous literature was used to develop a relevant protocol based on the initial propositions of

26

the research. Some of the guiding questions were adopted from Recker et al. (2010), Green et

al. (2011), Davies et al. (2006), and Fettke (2009).

The experimental research design is the most rigorous design to collect and analyse data

that highly relies on theory to design treatments, specify hypotheses, and analyse data. In

addition, a high level of control ensures that the results of the experiment are attributable to

the defined constructs and not some extraneous variables (Shadish et al., 2002). The

experimental design in the current research mainly followed the previous experimental studies

on conceptual modelling (e.g., Bera et al., 2014; Burton-Jones & Meso, 2008; Figl & Recker,

2014; Gemino & Wand, 2005; Khatri et al., 2006).

External Validity

External validity also refers to the validity of the research design in quantitative studies,

which can refer to transferability of the design of the qualitative study (Lincoln et al., 2011).

External validity concerns the generalisability of the findings over different settings and

contexts, such as variations in persons, measurement variables, and treatment variables

(Venkatesh et al., 2013).

External validity is ensured by including an inductive approach in the design phase of

semi-structured interviews (Shanks, 2002; Yin, 2009). Semi-structured interviews aim to

extend existing theories. They also aim to explore and develop a complementary knowledge

to explain the use of combinations of models in practice. In this thesis, the theory of faithful

use of conceptual model combinations was used as a foundation from which to derive the

interview protocol. More specifically, the results of ontological analysis of modelling

grammars motivated the design of the guiding questions. Moreover, to increase the

generalisability of this research, practitioners were interviewed from different countries,

different organisations, with different experience, and with different roles. The diversity of the

respondents also allowed for replication logic in the data analysis.

Generalisability is a problem in most experimental studies, which raises questions about

their relevance to practice (Moody, 2000). In this research, as with many experimental

studies, undergraduate students were used as participants. In the context of this study, students

were regarded as reasonable proxies of junior working experts. For instance, Arisholm and

Sjøberg (2004) found little difference between students and junior professionals in


understanding UML diagrams and maintaining Java programs. The design of tasks and

models in the experimental study is considered to be very simple compared to those used in

practice. However, the primary strength of experimental research is internal validity, which

ensures the validity of the results of empirical evaluation of theoretical propositions (Calder et

al., 1981). Moreover, very little is actually known about how combinations of conceptual

models are used in practice. Therefore, the primary aim of this thesis was to ensure internal

validity and control potential extraneous factors that may influence the dependent variables

(Parsons & Cole, 2005).

Construct Validity

Construct validity refers to measurement validity in quantitative research and can also

be referred to as analytical validity of qualitative research (Venkatesh et al., 2013). Construct

validity concerns the validity of designed variables and measures to be able to collect relevant

information to study the phenomena of interest and whether the findings of the study fit the

data from which they are derived (Straub et al., 2004).

To ensure construct validity, multiple stakeholders were interviewed from different

backgrounds (Yin, 2009). Moreover, the constructs and factors in the research design were

derived from the literature review based on relevant theories. The exploratory nature of the

semi-structured interviews also contributed to construct validity by providing opportunities

for new or refined constructs to emerge. To ensure the validity of the findings, the established

methodologies of qualitative analysis were followed (Silverman, 2006; Urquhart, 2013).

Moreover, the collected data have been maintained in a database to provide opportunities to

reconstruct the data collection and analysis procedures.

To meet construct validity in the experiment, the previous studies, measurements, and

instruments were used to operationalise the constructs of the current study (Burton-Jones &

Meso, 2008; Figl & Recker, 2014; Gemino & Wand, 2005). To do so, the definitions of the

constructs used in this thesis are close to those of the constructs used by other authors (e.g.,

Bera et al., 2014; Burton-Jones & Meso, 2008; Cierniak, Scheiter, & Gerjets, 2009; Figl &

Recker, 2014; Recker, 2010a; Recker & Rosemann, 2010a, 2010b).

28

Reliability

Reliability refers to the quality of measurement and is a precondition for validity in

quantitative research. In qualitative research, consistency and dependability of data and

analysis are two terms that are conceptually similar to reliability in quantitative

research(Venkatesh et al., 2013). Reliability concerns the repeatability and consistency of

measurement variables (Lincoln et al., 2011; Straub et al., 2004).

To build reliability of collected data in the semi-structured interviews, the interview

protocol was developed to strengthen the reliability of the collected data. The protocol was

consistently used during interviews to collect data. Subsequent and consistent use of the

designed interview protocol for all interviews assisted in providing consistency to the

structure and conduct of data collection. This consistency contributed to the reliability of the

gathered data. Moreover, all interviews were audio-recorded and transcribed by the

researcher. All collected data from the interviews are stored in an interview database.

To test the reliability of measurement variables of the experimental study, the most

widely used statistical measure, “Cronbach’s α”, was used to test for internal consistency

(Straub et al., 2004). Moreover, previously validated measures from the literature were used

to ensure the reliability. Moreover, multi-coder procedures were followed to code the

collected data and to establish the reliability of interpreting mappings and ontological

analysis, similar to procedures described in previous studies (Recker et al., 2010).

Chapter 4:Conceptual Modelling Research in Information Systems: What We Know Now and What We Still Do Not Know 29

Part B: Studies

Chapter 4: 3 Conceptual Modelling Research in

Information Systems: What We

Know Now and What We Still Do

Not Know4

ABSTRACT

Conceptual modelling tasks involve building a representation of a system domain to

better understand the system requirements. For many years, researchers have evaluated

different aspects of conceptual modelling to determine ways to improve information systems

analysis and design. Much of this research was guided by a seminal research agenda

developed by Wand and Weber (2002), which illustrated twenty-two research opportunities

and categorised them into four main areas: conceptual modelling grammars, methods, scripts,

and contexts. We examine the extent to which research over the past two decades has

addressed the identified research opportunities and answered the question of “How can we

model the world to better facilitate our developing, implementing, using, and maintaining

more valuable information systems?”. Our findings from a review of the literature show a

dialectic: several of the opportunities noted in 2002 have been addressed substantially; other

have been entirely neglected. We also found several path-breaking studies that did address

3 This study was conducted in collaboration with co-authors. The conceptualisation of the work and its

realisation was primarily the responsibility of the principal author. The principal author wrote the main parts of

the manuscript, compiled the literature sources, and analysed and interpreted the data. The co-authors assisted

with data interpretation, checked the references, and edited and wrote parts of the manuscript.

4 Conceptual Modelling Research in Information Systems: What we know now and what we still do not know.

To be submitted to Information Systems Research.

A short version of the paper was published as: Jabbari Sabegh, M. A., Lukyanenko, R., Recker, J. C., Samuel,

B., & Castellanos, A. (2017). Conceptual modeling research in information systems: What we now know and

what we still do not know. Paper presented at the Symposium on Research in Systems Analysis and Design (AIS

SIGSAND 2017).

30

problems not noted by the initial framework. To stimulate a new, relevant wave of conceptual

modelling research, we provide a new framework that draws the attention of conceptual

modelling research to the interplay between digital representations and outcomes.

Keywords: Conceptual modelling, research opportunities, literature review, research

agenda

4.1 INTRODUCTION

The importance of conceptual modelling to the analysis and design of information

systems has long been noted (Burton-Jones & Meso, 2006). High-quality conceptual models

play a significant role in the early detection and correction of systems development errors

(Wand & Weber, 2002), which is important given the high failure rates in information

systems projects (Gemino & Wand, 2004).

Conceptual modelling, while a core activity since the early days of the information

systems discipline (Bubenko, 1979; Lyytinen, 1987; Olle, Sol, & Verrijn-Stuart, 1982), has

long been regarded as a niche topic of interest, largely only to the community of scholars

interested in systems analysis and design. Moreover, much of the research was long

characterised as theoretical, of little practical relevance, and with limited empirical support

(Kung & Sølvberg, 1986; Moody, 2005; Siau & Rossi, 2011; Wand & Weber, 1989).

Researchers have suggested research frameworks and research agendas to promote

further research on conceptual modelling. Examples of these works include: ontological

foundations of conceptual modelling (Guizzardi, 2005; Hirschheim, Klein, & Lyytinen, 1994;

Wand & Weber, 1990b), frameworks to evaluate the quality of conceptual models (Lindland,

Sindre, & Solvberg, 1994; Mendling, Reijers, & Cardoso, 2007; Moody, 2005; Recker, 2007),

current issues and challenges in conceptual modelling (Indulska, Recker, Rosemann, &

Green, 2009; Recker, 2015), and the framework that defines the constructs of conceptual

modelling (Wand & Weber, 2002). Among these research frameworks, the current study

focuses on the research framework and agenda proposed by Wand and Weber (2002) to

answer the question regarding how to create high-quality conceptual models to better

facilitate developing, implementing, using, and maintaining more valuable information

systems. Wand and Weber’s (2002) framework provides a more comprehensive research

agenda concerning the foundations, quality, understandability, usability, and context of


conceptual modelling. Since then, a large share of studies on conceptual modelling guided by

this research agenda have been published in the field’s main journals, and conceptual

modelling is now widely regarded as a key topic in information systems research (Allen &

March, 2012; Siau & Rossi, 2011).

Fifteen years after its publication, we believe it is time to ask whether the extant

research addressed the opportunities highlighted by Wand and Weber (2002), and which

aspects of conceptual modelling require further investigation. In this paper, we therefore

pursue a two-fold objective; first, we examine the published research on conceptual modelling

since the publication of the Wand and Weber (2002) paper. Our objective is to synthesise

relevant studies on conceptual modelling that in our view contribute to and shape

understanding of the conceptual modelling discipline, and then identify the remaining gaps in

the field that require further investigation. Second, we also ask whether the framework by

Wand and Weber (2002) remains relevant to this day, or whether a new, innovative agenda

should be set. In addressing both objectives, our paper will provide a complete perspective of

conceptual modelling research, as well as substantive directions for future research.

The rest of the paper proceeds as follows. We begin by explaining conceptual modelling

and the research framework presented by Wand and Weber (2002). The methodology that we

followed to identify relevant studies and the procedure of reviewing them will then be

explained in the research methodology section. In section 4.4, we analyse the literature by

explaining research opportunities for conceptual modelling grammars, methods, scripts and

context and reviewing the studies that have been undertaken about each of them. We then

outline the research areas in which there is a lack of study and propose a revised research

agenda. Finally, we conclude with the findings of this study.

4.2 BACKGROUND

Conceptual models are typically graphical representations of a real-world domain used

to assist analysts to develop an understanding of the system domain; facilitate communication

among different stakeholders to better understand system requirements; document the system

for maintenance, operation, and training purposes; and provide input for the design process

(Fettke, 2009; Wand & Weber, 2002). Conceptual models are often employed to represent

either dynamic aspects of a real-world domain (e.g., processes and events), or static aspects of

32

a domain (e.g., things and properties of things), or both (Burton-Jones & Weber, 2014).

Figure 4.1 provides some examples of different conceptual models developed using, from left

to right, the BPMN grammar, EPC grammar, ERD grammar, and UML grammar.

Check the order

Issue bill

Check quality

Ship the product

No

t i

n g

oo

d q

ua

lity

or d

am

ag

ed

Check the order

V

Issue billXOR

Check the product

XOR

V

Start shipping

Not good quality or damaged

Good quality

Orders

-customerName

-dateCreated-dateShipped

-orderID

Customer

-address-customerName

User

-password-UserID

Administrator

-adminInfo-adminName

-email-contactdetails-shippinginfo

1..1

0..*

Figure 4.1 Examples of different scripts developed using four conceptual modelling grammars

Both the conceptual modelling process and the models as the products of the process of

conceptual modelling are fundamental means in developing and analysing information

systems (Wyssusek, 2006). As a result, research in this area has focused on different aspects

of conceptual modelling to improve the effectiveness and efficiency of the conceptual

modelling process and products generated through it. For example, part of this research

tradition has focused on the development of new grammars and methods (Oei, Van Hemmen,

Falkenberg, & Brinkkemper, 1992; Wand, Woo, & Wand, 2008), while others have studied

individual factors (Recker, 2010a; Topi & Ramesh, 2002) or matters related to cognition and

perception (Evermann, 2005; Veres & Mansson, 2005).

In a seminal paper that structured and guided research on conceptual modelling, Wand

and Weber (2002) delineated four main concepts relevant to conceptual modelling research

(Figure 4.2):


Figure 4. 2 Framework for Research on Conceptual Modelling (Wand & Weber, 2002, p. 364)

Conceptual modelling grammars are sets of constructs and rules that demonstrate

how constructs should be combined together to model a real-world domain. For

instance, the entity-relationship modelling grammar provides constructs such as:

“entity” and “relationship”. The grammar’s rules indicate that two entities can be

associated only via a relationship.

Conceptual modelling methods explain and rationalize how a modelling grammar

can be used. Methods explain, for example, how domain observations can be

mapped to the grammar’s constructs or which instances of phenomena are relevant

to the model (Wand & Weber, 2002, p. 364).

Conceptual modelling scripts are the products of using conceptual modelling

grammars. Conceptual modelling scripts are created by using modelling grammars

based on various conceptual modelling methods. For example, process models are

created with process modelling grammars to graphically represent relevant aspects

of business processes.

Conceptual modelling context refers to the setting in which conceptual models are

created and in which they are employed for specific task goals. Wand and Weber

(2002) proposed three contextual factors: individual difference factors, which

include the matters relating to modellers and model users such as training,

experience, or cognitive ability; task factors, which describe different tasks in

which scripts and grammars are employed; and social agenda factors, which

Conceptual Modelling-Method

Conceptual-

Modelling

Grammar

Conceptual-

Modelling

Scripts

Conceptual-Modelling

Context

Individual Difference Factors

Task Factors

Social Agenda Factors

34

describe the broader setting and organisational change in which grammars are

employed.

Using these four main areas, Wand and Weber (2002) proposed several research

opportunities for each of the four main areas of conceptual modelling research. Table 4.1

briefly summarises these research opportunities.

Table 4. 1 Research Opportunities in Conceptual Modelling (Wand & Weber, 2002, p. 373)

Research

Framework Code Research Opportunities

Conceptual

Modelling

Grammars

G1 Evaluating ontologies based on empirical testing of their predictions.

G2 Evaluating grammars for ontological expressiveness.

G3 Assigning ontological meaning to constructs of design grammars and generating

ontologically motivated modelling rules.

G4 Resolving outstanding ontological problems that impact conceptual modelling (e.g.,

nature of the part-of relationship).

G5 Empirically testing predicted strengths and weaknesses in new and existing grammars

based on their ontological expressiveness.

G6 Determining which combinations of grammars best support users who undertake conceptual modelling work.

G7 Empirically testing the predicted implications of construct deficit and overload in

grammars.

Conceptual

Modelling

Methods

M1 Evaluating how well different methods allow users to elicit and model critical domain

knowledge.

M2 Developing procedures to assist users of a grammar in identifying and classifying

phenomena according to the grammar's constructs.

M3 Determining the beliefs and values that underlie different methods and evaluating the

consequences of these beliefs and values for practice.

Conceptual Modelling

Scripts

S1 Evaluating competing scripts generated via the same grammar to describe some

phenomenon.

S2 Evaluating competing scripts generated via different grammars to describe the same

phenomenon.

S3 Evaluating different combinations of scripts to determine which combination best

supports the task at hand.

S4 Developing theory to predict and understand how humans use scripts to accomplish

various tasks.

Conceptual

Modelling

Context

Individual

differences

Task

Social

agenda

CI1 Development of knowledge-based tools to support conceptual modelling.

CI2 Predicting which cognitive and personality variables bear on a user's ability to undertake

conceptual modelling work.

CI3 Predicting and testing empirically which social skills affect the outcomes of conceptual modelling tasks.

CT Evaluating the strengths and weaknesses of conceptual modelling grammars, methods,

and scripts in the context of different tasks.

CS1 Understanding which values and beliefs underlie conceptual modelling work in practice.

CS2 Determining the costs and benefits of adopting different values and beliefs when

undertaking conceptual modelling work.

CS3 Articulating detailed conceptual modelling procedures that are congruent with different

beliefs and values.

CS4 Understanding how existing conceptual modelling grammars and methods facilitate

conceptual modelling work under different values and beliefs.


Since then, it would appear that their call for research was heeded – as just one

indicator, at the time of writing this manuscript, their research agenda has been cited over 780

times on Google Scholar.

4.3 LITERATURE REVIEW PROCEDURES

We wanted to ascertain whether the past fifteen years of research on conceptual

modelling has provided appropriate, sufficient, and exhaustive answers to the questions Wand

and Weber posed in 2002. To that end, our literature review method drew on several

approaches (Bandara et al., 2015; Boell & Cecez-Kecmanovic, 2015; Paré et al., 2015; Rowe,

2014; Vessey, Ramesh, & Glass, 2002; Webster & Watson, 2002). It involved four steps:

First, we selected journals for the scope of the review. Given the vastness of the

research on system analysis and design and conceptual modelling, we chose to limit our

sample of studies to those published in the AIS basket of eight journals (F. Liu & Myers,

2011) plus the Journal of Database Management. Papers published in the top IS journals

were considered appropriately representative for mainstream high-quality research in the

information systems field. To broaden the validity of our review results and to accommodate

the view of conceptual modelling as a potential niche topic, we also included studies

published in the Journal of Database Management. This journal has traditionally been one of

the leading substantive journals publishing studies on conceptual modelling.

Second, we performed a full-text search in all papers in the selected journals using

keywords such as “conceptual modelling”, “conceptual model*”, “conceptual modelling

grammar”, “ontology” and meaningful variations of these terms. With these search terms, we

retrieved 3546 papers by October 2016 (See Appendix A). We then examined the title and

abstract of each of the papers to exclude all papers not dealing with conceptual modelling

research. Through this process, the total number of papers related to conceptual modelling

research reduced the total to 133 potentially relevant papers. The vast majority of excluded

papers discussed the term “conceptual model” to refer to a theory or research framework

rather than a description of information systems for purposes of analysis and design. Two

authors then reviewed the 133 potentially relevant papers to confirm relevance to our study.

We excluded papers that defined models as computer executable procedures, programs, or

codes (e.g., Krishnan, Mukhopadhyay, & Kriebel, 2004), or used concepts of conceptual

36

modelling to define other aspects of information systems, such as the effective use of

information systems (e.g., Burton-Jones & Grange, 2013), which we also did not regard as

relevant. Through this process, we arrived at a final number of 105 relevant articles. Table 4.2

summarises the distribution of these papers.

Table 4.2 Reviewed papers

Journals Number of relevant papers

MIS Quarterly 6

Information Systems Research 9

Journal of the Association for Information Systems 17

European Journal of Information Systems 15

Journal of Management Information Systems 1

Information Systems Journal 5

Journal of Information Technology 1

Journal of Strategic Information Systems 2

Journal of Database Management 49

Total 105

Third, we developed and applied a coding scheme. This coding scheme (summarised in

Appendix A) allowed us to categorise papers alongside multiple broad dimensions: the focus

and goal of the paper (Vessey et al., 2002), prominent conceptual modelling area addressed

(building on the classifications used in Wand & Weber, 2002), research method used (W.

Chen & Hirschheim, 2004), and evidence (Atkins et al., 2004), if any.

Fourth, to ensure a reliable application of the coding scheme, one author coded all 105

papers, while a second author independently coded a random subset of 30 papers. Their inter-

coder agreement was 62%. The two authors then discussed disagreements, updated coding

criteria and instructions, and then independently revised the coding over two more rounds

until 100% agreement was reached. The first author then revised the coding of the remaining

75 articles.

4.4 ANALYSIS OF THE LITERATURE

In presenting the findings from our analysis, we discuss the identified literature on the

four main conceptual modelling categories (grammars, methods, scripts, and context) in turn.

We start by providing three broad overviews: Table 4.3 summarises the distribution of papers

by their focus on conceptual modelling areas (i.e., grammars, methods, scripts, and context).


Table 4. 3 Research on conceptual modelling based on focus of the study

Focus Basket of 8 JDM All Papers

No. % No. % No. %

1 Grammar 5 8.93 7 18.92 12 11.43

2 Method 7 12.50 10 27.03 17 16.19

3 Script 3 5.36 6 16.22 9 8.57

4 Context 6 10.71 1 2.70 7 6.67

5 Grammar and method 2 3.57 1 2.70 3 2.86

6 Grammar and script 6 10.71 1 2.70 7 6.67

7 Grammar and context 2 3.57 2 5.41 4 3.81

8 Method and script 7 12.50 3 8.11 10 9.52

9 Method and context 4 7.14 3 8.11 7 6.67

10 Script and context 2 3.57 1 2.70 3 2.86

11 Grammar, method, and script 0 0.00 1 2.70 1 0.95

12 Grammar, method, and context 0 0.00 0 0.00 0 0.00

13 Method, script, and context 0 0.00 0 0.00 0 0.00

14 Grammar, script, and context 3 5.36 1 2.70 4 3.81

15 Grammar, method, script, and context 0 0.00 0 0.00 0 0.00

16 Other 9 16.07 12 32.43 21 20.00

Total 56 - 49 - 105 100

Table 4.4 summarises the papers by type of empirical evidence collected, if any. Table

4.5 summarises the most popular conceptual modelling grammars examined in the literature.

Table 4. 4 Number of papers per type of evidence

Grammar Method Scripts Context Other

Research Method # % # % # % # % # % # %

Empirical 47 44.76 18 58.06 23 60.53 27 79.41 17 68.00 3 14.29

Case study 5 4.76 2 6.45 2 5.26 1 2.941 2 8.00 1 4.76

Action research 1 0.95 0 0.00 1 2.63 0 0 0 0.00 1 4.76

Survey 3 2.86 3 9.68 0 0.00 0 0 1 4.00 0 0.00

Experiment 31 29.52 13 41.94 15 39.47 24 70.59 9 36.00 1 4.76

Interview 4 3.81 1 3.23 2 5.26 0 0 4 16.00 0 0.00

Design science 4 3.81 0 0.00 3 7.89 1 2.941 1 4.00 0 0.00

Leveraging

previous data 0 0.00 0 0.00 0 0.00 1 2.941 0 0.00 0 0.00

Prototype 1 0.95 0.00 1 2.63 0 0 1 4.00 0 0.00

None-Empirical 58 55.24 13 41.94 15 39.47 7 20.59 8 32.00 18 85.71

Literature review 5 4.76 0 0.00 0 0.00 2 5.882 2 8.00 1 4.76

Commentary 7 6.67 3 9.68 0 0.00 0 0 2 8.00 1 4.76

Theoretical 46 43.81 10 32.26 15 39.47 5 14.71 4 16.00 16 76.19

38

Table 4. 5 Number of papers per type of model

Grammar # Grammar #

UML 7 ISO TC87 1

UML Class 12 Merise 1

UML Use Case 5 ebXML 1

UML Activity 5 BPML 1

UML State Machine 2 WSCL 1

UML Sequence 2 WS-BPEL 1

UML Collaboration 1 DEMO 1

ER 22 ProH 1

BPMN 7 REA 1

EPC 6 ORM 1

EER 4 IFO 1

Petrinets 4 FDM 1

Workflow 3 SDM 1

ANSI Flowchart 2 NIAM 1

DFD 1 OMT 1

YAWL 1 OML 1

MibML 1 DFRD 1

As presented in these tables, 46 percent of reviewed studies were published in the

Journal of Database Management, followed by the Journal of the Association for Information

Systems with 16% of the reviewed papers. The notable theme of the reviewed articles is that

more than 37% of the research effort concentrated on more than one area of conceptual

modelling (e.g., grammar and script or method and script). UML and Entity-Relationship

(ER) grammars were the most popular grammars investigated in the reviewed papers.

4.4.1 Conceptual Modelling Grammars

In all, 31 studies of the reviewed articles primarily investigated conceptual modelling

grammars. The types of papers included 20 empirical studies, two commentary papers, and

nine theoretical studies. The choice of research methodologies for the empirical studies

included 13 experimental studies, three industrial surveys, two interviews, one case study, and

one case study with experiment.

Identified research that addressed opportunities proposed by Wand and Weber (2002)

Unlike the early research on conceptual modelling grammars (Oei et al., 1992), the

research on conceptual modelling grammars since 2002 apparently shifted its focus from

developing new grammars to evaluating and improving existing grammars. The common


theme emerging from these studies is that studies on conceptual modelling grammars mostly

used Wand and Weber’s (1990a, 1993, 1995) representation theory (Wand & Weber, 1990a,

1993, 1995) to evaluate conceptual modelling grammars and the implications of ontological

deficits of the modelling grammars, with the exception of one study, which used an

alternative ontological benchmark to evaluate modelling grammars (Milton & Kazmierczak,

2004). Table 4.6 provides a summary of these studies based on their main focus on conceptual

modelling grammars.

One of the central findings from this body of research on the evaluation of modelling

grammars is that any one grammar suffers from some level of construct deficit (Irwin & Turk,

2005; Recker et al., 2010; Recker, Rosemann, Indulska, & Green, 2009; Hong Zhang,

Kishore, & Ramesh, 2007). For example, Irwin and Turk (2005) indicated that use case

modelling grammars are ontologically incomplete due to construct deficits in representing the

system structure and decomposition.

Table 4. 6 Papers on conceptual modelling grammars and type of evidence5

Code Focus #

Type of

Evidence #

G1 Evaluating ontologies based on empirical testing of their predictions. 0

G2 Evaluating grammars for ontological expressiveness. 6 Interview 1

Theoretical 5

G3

Assigning ontological meaning to constructs of design grammars and

generating ontologically motivated modelling rules. 8

Case Study 2

Experiment 4

Commentary 1

Theoretical 2

G4

Resolving outstanding ontological problems that impact conceptual

modelling (e.g., nature of the part-of relationship). 2

Experiment 1

Theoretical 1

G5

Empirically testing predicted strengths and weaknesses in new and existing

grammars based on their ontological expressiveness. 0

G6

Determining which combination of grammars best support users who

undertake conceptual modelling work. 0

G7 Empirically testing the predicted implications of construct deficit and

overload in grammars. 10

Survey 1

Experiment 8

Interview 1

G

Other Other 7

Survey 2

Experiment 1

Commentary 1

Theoretical 3

5 Note that one paper contained more than one type of empirical evidence and two papers focussed on more than

one research opportunity. These papers were coded in each element, respectively.

40

Studies on conceptual modelling grammars have also focused on understanding and

improving existing grammars (Allen & March, 2006; Bera et al., 2011; Evermann & Wand,

2005; Laurier & Poels, 2012; Rittgen, 2006; Soffer & Kaner, 2011; Soffer, Kaner, & Wand,

2010). In doing so, researchers aimed to assign real-world semantics or ontological meaning

to the constructs of the grammars to facilitate their use in conceptual modelling. Eight studies

focused on assigning ontological meaning to constructs of design grammars and generating

ontologically motivated modelling rules. The dominant idea of these studies is to develop

rules to guide the modellers in creating ontologically well-formed models. The findings

indicate that diagrams created with the ontologically motivated rules lead to better domain

understanding (Bera, Krasnoperova, & Wand, 2010).

However, there are still some important real-world phenomena that remain problematic

for conceptual modelling, as they are not ontologically well-defined. For example, Shanks et

al. (2008) argued that representing the part-of construct remains a contentious issue. To

overcome this issue, the researchers suggested representing composites as entities (Shanks &

Weber, 2012).

Several studies (10 out of 31), focused on empirically testing the predicted implications

of construct deficit and overload in grammars. The main argument of these studies is that the

existence of construct overload in conceptual modelling grammars decreases the clarity of

conceptual modelling and therefore decreases the understandability and usefulness of the

designed models (Shanks et al., 2010; Shanks et al., 2008).

However, there are also some contrary arguments. Bowen et al. (2009) conducted an

experiment to evaluate the relationship between the level of model expressiveness and

ontological clarity on user performance during query development. The results indicate that

participants using the implementation that exhibited greater expressiveness via greater

ontological clarity made significantly more semantic errors, took significantly more time to

compose their queries, and were significantly less confident in the accuracy of their queries in

contrast to the findings of Bowen et al. (2006), Parsons (2011), and Shanks et al. (2010).

Likewise, Allen and March (2012) also found that construct overload does not have a

significant effect on subjects’ task performance.


Identified opportunities in addition to those proposed by Wand and Weber (2002)

Through our review of studies across conceptual modelling grammars, we found seven

studies that did not fall into any of the grammar-related research opportunities proposed by

Wand and Weber (2002). Three main themes emerged from these studies. First, the

researchers highlighted the importance of factors other than ontological elements in

evaluating conceptual modelling grammars. These included semiotic and perceptual

discriminability of the grammatical constructs (Figl, Mendling, et al., 2013), and logical

qualities of conceptual modelling grammars (Clarke et al., 2016). Second, apart from

evaluating the capabilities of conceptual modelling grammars, researchers investigated what

other factors affect the usage behaviour of conceptual modelling grammars. The results from

surveying practitioners showed that the decision about which type of modelling grammar to

use depends on the purpose of the task in hand (Dobing & Parsons, 2008), and the willingness

to continue using the selected grammars depends on various factors, including the modeller’s

experience in modelling, background, and familiarity with the modelling grammar (Recker,

2010a). Third, researchers highlighted the complexity of modelling grammars and difficulties

in learning how to use the grammars. For example, VanderMeer and Dutta (2009) focused on

the unified modelling language (UML) and more specifically on UML sequence diagram and

proposed guidelines based on cognitive theories and the theory of learner-centred design to

decrease the complexity of learning how to build UML sequence diagrams.

Opportunities proposed by Wand and Weber (2002) that have not been addressed

Some of the research opportunities related to conceptual modelling grammars have not

yet been answered, for example, evaluating different ontologies based on empirical evidence.

Weber (2003) argued that some other alternative ontological theories can be used to improve

the understandability of modelling techniques or to predict different sorts of conceptual

modelling phenomena. One of the arguments about ontologies is that the selected ontology

should be understandable and applicable for selected modelling techniques, the purpose of

modelling and the types of users (Green & Rosemann, 2004).

As noted above, we found only one study that replaced Bunge’s (1977) ontology with

Chisholm’s (1996) ontology to evaluate data modelling languages (Milton & Kazmierczak,

2004). However, we couldn't find any empirical evidence of evaluating Bunge’s (1977)

ontology in comparison with Chisholm’s (1996) ontology.

42

Another open research opportunity proposed by Wand and Weber (2002) concerns

empirically testing the impacts of potential weaknesses or strengths of conceptual modelling

grammars on users of scripts created by the grammars, as well as the usefulness of

ontologically based modelling rules.

Another important question that remains is which combinations of grammars best

support users who undertake conceptual modelling work. While there has been some work

regarding evaluating the ontological capabilities of multiple modelling grammars (Green et

al., 2007; Green et al., 2011), there is still no empirical evidence regarding which

combinations of grammars best support users in their design and analysis tasks even though

practical observations clearly indicate that IS professionals normally use multiple modelling

grammars in their design and analysis tasks (Dobing & Parsons, 2008).

4.4.2 Conceptual Modelling Methods

In all, 38 studies addressed research opportunities on conceptual modelling methods.

Studies in this area were primarily empirical, with 13 experimental studies, three design

science research, two practitioners’ interviews, one case study, one action research, one

prototype, and two studies that provided multiple empirical evidence using experiment and

interviews and case study with experiment. Fifteen studies chose a theoretical approach and

proposed methods and guidelines to improve conceptual modelling.


Research on conceptual modelling methods mostly focused on developing procedures

for using grammars and identifying instances of phenomena to model with the grammar.

Table 4.7 provides a summary of these studies.

Out of 38 studies on conceptual modelling method, nine papers investigated how

different methods lead to better conceptual modelling practices. Together, this research

indicates that methods such as the Resource-Event-Agent modelling patterns (Poels, 2011;

Poels, Maes, Gailly, & Paemeleire, 2011), ontologically motivated rules in modelling (Bera,

2012), reuse of analysis patterns augmented with learning mechanisms (Purao, Storey, & Han,

2003), or diagrammatic attention management (Dunn, Gerard, & Grabski, 2011) can lead to

improved conceptual modelling results, such that, for example, models developed following

these methods are easier to understand for users. Siau and Rossi (2011) reviewed evaluation


techniques for conceptual modelling methods and argued that different evaluation techniques

fit different purposes.

Table 4. 7 Papers on conceptual modelling method and type of evidence6

Code Focus # Type of

Evidence #

M1 Evaluating how well different methods allow users to elicit and model

critical domain knowledge. 9

Experiment 7

Action Research 1

Theoretical 1

M2 Developing procedures to assist users of a grammar in identifying and

classifying phenomena according to the grammar's constructs. 19

Experiment 4

Interview 2

Design Science 3

Proto Type 1

Theoretical 10

M3 Determining the beliefs and values that underlie different methods and

evaluating the consequences of these beliefs and values for practice. 1 Interview 1

M

Other Other 10

Case Study 2

Experiment 5

Theoretical 4

A second noted main focus was research on conceptual modelling methods that

developed procedures to assist users of grammars in identifying and classifying phenomena

according to the grammars’ constructs; such as developing cognitive principals to guide

classification (Parsons & Wand, 2008). The dominant idea emerging from research in this

category was that cognitive principles and ontological guidelines can assist users of a

grammar to identify and classify phenomena according to the grammar’s constructs (Bera,

2012; Bera et al., 2010; Evermann, 2005; Hadar & Soffer, 2006; Parsons & Wand, 2013).

A third dominant focus was the phenomena of diversity in conceptual modelling

(Ågerfalk & Eriksson, 2004). Researchers argued that using an ontology-based modelling

framework can help modellers to reduce variety in developed models (Hadar & Soffer, 2006).


Several scholars went beyond the research opportunities suggested by Wand and Weber

(2002) and suggested other methods aimed at improving conceptual modelling. One foci of

this work was the question of whether ontological guidelines, per se, can sufficiently cover

the problems of conceptual modelling. Clarke et al. (2016) proposed a logical perspective to

6 Note that two papers contained more than one type of empirical evidence and one paper focused on more than

one research opportunity. These papers were coded in each element, respectively.

44

improve the semantics of conceptual modelling and suggested using logical principles

together with ontological principles.

Researchers highlighted the classification of phenomena according to the grammars’

constructs and information loss when model users use developed class-based models. An

approach to overcome the effect of classification on information loss was proposed by

Lukyanenko, Parsons, and Wiersma (2014). The proposed approached signified the use of the

instance-and-attribute based modelling approach in developing models.

Another emergent dominant idea was to represent temporal dynamics of an application

using the constructs of a structural model. Allen and March (2003) argued that temporal

dynamics are semantic rather than structural and that the existing constructs in the ER model

are sufficient to represent them. The proposed method was based on representing events as

entities. Methods of mapping and transforming the captured requirements in conceptual

models to relational database and platforms were other areas highlighted by the reviewed

studies. An, Hu, and Song (2010) offered Round-Trip Engineering to maintain the

consistency of conceptual-relational mappings. Pardillo, Mazón, and Trujillo (2011) proposed

an approach to automate transforming the captured requirements into the data-warehousing

platform following model driven architecture and QVT (query/view/transformation)

transformations, and Rittgen (2006) suggested a language mapping framework to transform

from/between UML and DEMO. With regards to automated database design, Storey,

Goldstein, and Ding (2002) suggested using common-sense knowledge to improve the quality

of the designed database.

While research on conceptual modelling methods proposed various approaches to

improve conceptual modelling, Batra (2008) asked a question regarding which pattern should

be used in different circumstances. To answer this question, Batra (2008) evaluated eleven

data modelling patterns suggested in textbooks. The findings indicated that the use and

frequency of patterns were different based on the type of audiences, for example, practitioners

vs. students. The suggestion of the study was to use a combination of patterns to improve

effective training and use strategy.


4.4.3 Conceptual Modelling Scripts

Thirty-four out of the 105 considered papers focused on conceptual modelling scripts.

Notably, most of these 34 papers also fell into other categories, such as grammars and scripts.

Table 4.8 represents the distributions of papers across research opportunities related to

modelling scripts. Out of the 34 papers on conceptual modelling scripts, 25 papers were

experimental studies, one case study, one design science, one study’s evidence was from

leveraging the findings of the previous studies, two literature reviews, and four theoretical

papers.

Table 4. 8 Research on conceptual modelling scripts and type of evidence7

Code Focus # Type of Evidence #

S1 Evaluating competing scripts generated via the same grammar to

describe some phenomenon. 21 Experiment 20

Design Science 1

S2 Evaluating competing scripts generated via different grammars to

describe the same phenomenon. 3 Experiment 2

Leveraging previous data 1

S3 Evaluating different combinations of scripts to determine which

combination best supports the task at hand. 0

S4 Developing theory to predict and understand how humans use

scripts to accomplish various tasks. 5 Experiment 3

Theoretical 2

S

Other Other 7

Case Study 1

Experiment 2

Literature Review 2

Theoretical 2


Twenty-one out of 34 papers on conceptual modelling scripts evaluated competing

scripts generated via the same grammar to describe the same phenomena; only three papers

evaluated competing scripts generated using different grammars. With most conceptual

modelling scripts being the products of using modelling grammars and methods, most of these

papers also appeared in our discussion above on conceptual modelling grammars and

methods.

The main theme that emerged from studies on scripts was that most of the researchers

compared competing scripts against the existence of construct deficits and overload and

whether assigning ontological meaning to grammatical constructs leads to better conceptual

7 Note that two papers focus on more than one research opportunity. These papers were coded in each element,

respectively.

46

modelling scripts. The second main theme of these studies was evaluating scripts developed

using different grammars or methods.

The evaluations of scripts based on ontological factors provided arguments about the

effect of construct overload on the clarity of conceptual modelling scripts. Most of the studies

supported that the existence of construct overload decreases the clarity of scripts and therefore

decreases the performance of the users (Bowen et al., 2006; Milton et al., 2012; Parsons,

2011; Shanks et al., 2010; Shanks et al., 2008). However, some studies provided contrary

arguments. For example, Allen and March (2012) did not find significant evidence that the

existence of construct overload affects users’ performance. Similarly, results of Bera et al.’s

(2014) study showed that construct overload did not affect the performance of novice and

expert users; however, models without construct overload improved the performance of users

with moderate knowledge of the domain shown in the model. In this line, findings of the

study by Bowen et al. (2009) also challenged the arguments of the usefulness of ontological

clarity on users’ performance for all task types, especially in the case of high complexity and

increase in the size of scripts (models).

Research on conceptual modelling scripts also evaluated scripts that developed using

the same grammar but following different conceptual modelling methods. Bera (2012) argued

that users of models developed using ontological rules have less difficulty in achieving

problem-solving models, while in developing the models, modellers who followed the

ontological rules also have less difficulty. Some other methods where researchers indicated

that ontological rules led to higher quality conceptual modelling scripts were classification

rules (Parsons & Wand, 2008), decomposition principles (Burton-Jones & Meso, 2006),

Resource-Event-Agent modelling patterns (Poels, 2011; Poels et al., 2011), application-based

domain modelling approach (Reinhartz-Berger & Sturm, 2008), using diagrammatic attention

management (Dunn et al., 2011), and using gateway constructs (Recker, 2013).

In contrast to evaluating different scripts developed using the same grammar, only three

studies evaluated different scripts developed using different grammars. Figl, Mendling, et al.

(2013) evaluated the effect of notational deficiencies, viz., semiotic clarity deficiencies and

perceptual discriminability deficiencies on process models understandability by comparing

scripts developed using the grammars YAWL, EPC, BPMN, and UML. Their findings

showed that the existence of both types of deficiencies negatively affected the performance of


model users, while the performance was worse than the other scripts for users of the script

with severe notational deficiencies. Khatri et al. (2006) investigated the performance of users

of ER and EER models considering the individual differences based on participants’ IS and

domain knowledge. Their findings did not show the differences between users of EER in

comparison with ER model users.

Five studies developed theories to predict and understand how humans use scripts to

accomplish various tasks. One dominant theory argued that using high-quality information

and a different form improves users’ performance (Burton-Jones & Meso, 2008). The finding

of the study by Burton-Jones and Meso (2008) was that users of well-decomposed models in

combination with textual narratives had better performance than users without textual

narratives and users with poorly decomposed models.

Another argument was that users of models developed following an ontological

guideline had less cognitive difficulties compared to a user of models developed without

ontological guidelines (Bera, 2012). The proposed theory that emerged from the studies was

that ontological guidelines alleviate the cognitive difficulties of model users in understanding

the developed models, as well as the cognitive difficulties of modellers during model

development (Bera, 2012; Bera et al., 2010). These arguments indicate that users of models

consistent with the ontological guidelines have less difficulty in understanding and achieving

task goals.

In developing theories on how practitioners use scripts to achieve tasks goals, Parsons

and Wand (2013) argued that the principles of conceptual modelling, such as classification,

can also be used in other disciplines. They further argued that classification principles used on

conceptual modelling can also be used in natural science. The theory argued that principles of

inference and economy guide the choice of individual classes and collections of classes.


We found seven papers that did not fall into any categories on conceptual modelling

scripts. Two main ideas emerged from these studies: first, quality measures that relate to

conceptual modelling scripts; second, the use of different types of additional information in

support of conceptual modelling scripts.

48

In terms of quality measures, our above review shows that researchers mainly used

ontological expressiveness, semantic quality, and notational discriminability to evaluate

conceptual modelling scripts. In addition, Burton-Jones et al. (2009) suggested guidelines and

a framework to evaluate conceptual modelling through experiments; Siau (2004) used human

information processing theory and cognitive science to propose an approach for evaluating

different scripts considering their informational and computational equivalence. Krogstie et

al. (2006) developed a quality framework for process modelling to ensure that developed

models fulfil the needs of models users and the objectives that underlie the modelling project.

Bandara, Gable, and Rosemann (2005) proposed that there are contextual factors, such as

project-specific factors, together with modelling related factors that lead to effective process

modelling and offer successful measures for conceptual modelling in general.

In terms of additional information, two studies suggested that using textual narratives

together with conceptual models improves domain understanding (Burton-Jones & Meso,

2008; Gemino & Parker, 2009). This line of research has been based on the Multimedia

Theory of Learning (Mayer, 2005), and reaffirms the importance of models and texts in

developing understanding.

Opportunities proposed by Wand and Weber (2002) that have not been addressed

The most notable unanswered opportunities concern the lack of empirical investigation

into the use of multiple models. To date, no research has been conducted into combining and

evaluating multiple scripts to best support analysis and design tasks.

4.4.4 Conceptual Modelling Context

The papers on conceptual modelling context fell into three main sub-categories, viz.,

individual factors, task, and social agenda factors, as summarised in Table 4.9. In all, 25

papers fell into the conceptual modelling context, out of these 25 papers, 22 papers focused on

individual factors, eight papers focused on task-related factors, and five papers focused on

social agenda factors.8

8 Papers could address more than one sub-category, and some papers focussed on more than one research

opportunity in each sub-category.


Table 4. 9 Papers in conceptual modelling context and type of evidence9

Code Focus #

Type of

Evidence #

Co

nce

ptu

al M

odel

ling C

onte

xt

Indiv

idual

Dif

fere

nce

s

CI1 Development of knowledge-based tools to support conceptual

modelling. 5

Experiment 2

Case Study 1

Developing

Prototype 1

Design

Science 1

CI2 Predicting which cognitive and personality variables bear on a user's ability to undertake conceptual-

modelling work. 15

Experiment 9

Interview 2

Survey 1

Literature

review 1

Theoretical 3

CI3 Predicting and testing empirically which social skills affect the

outcomes of conceptual modelling tasks. 1

Literature

Review 1

Other Other 1 Case Study 1

Tas

k

CT Evaluating the strengths and weaknesses of conceptual modelling

grammars, methods, and scripts in the context of different tasks. 4

Survey 1

Experiment 1

Interview 1

Theoretical 1

Other Other 5

Case study 1

Interview 2

Experiment 1

Literature

Review 1

Theoretical 1

So

cial

Ag

enda

CS1 Understanding which values and beliefs underlie conceptual

modelling work in practice. 3 Interview 3

CS2 Determining the costs and benefits of adopting different values and beliefs when undertaking conceptual modelling work.

0

CS3 Articulating detailed conceptual modelling procedures that are

congruent with different beliefs and values. 0

CS4

Understanding how existing conceptual modelling grammars and

methods facilitate conceptual modelling work under different

values and beliefs.

0

Other Other 2 Experiment 1

Theoretical 1

Individual Differences Factors – Identified research that addressed opportunities

proposed by Wand and Weber (2002)

Human factors in conceptual modelling have gained widespread attention since 2002. In

this area of research, we reviewed previous literature to evaluate how well previous studies

covered research areas proposed in 2002.

9 Note that one paper contains more than one type of empirical evidence. This paper was coded in each element,

respectively.

50

First, five out of 22 studies on individual differences factors in conceptual modelling

focused on developing knowledge-based tools to support conceptual modelling. The main

theme arising from this stream of research was the importance of the use of collected and

learned knowledge in conceptual modelling. For instance, Purao et al. (2003) developed a

prototype to emulate the learning that occurs over time in analysis patterns. Storey et al.

(2002) developed a common-sense business reasoner containing general knowledge about the

world and a reasoning tool to apply this knowledge to a database design task. Similarly,

Koschmider, Song, and Reijers (2010) developed a recommendation-based process modelling

support system to receive recommendations through social features to improve the modelling

process, and as a result, the quality of the designed models.

Second, researchers also highlighted the importance of capturing all requirements and

the traceability of the system and argued that information systems should have the ability to

evolve as their environment changes. To achieve this objective, Loucopoulos and Kadir

(2008) developed tools to improve the traceability in information systems through the

Business Rules-Driven Object Oriented Design (BROOD) approach. In this vein of research,

and to be able to automatically transform the requirements captured in the early stages to the

entire data-warehousing platform, Pardillo et al. (2011) proposed an model-driven

architecture approach and query/view/transformation (QVT) transformations and developed

CASE support tools that support the approach.

The third important research opportunity was predicting which cognitive and

personality variables bear on a user's ability to undertake conceptual modelling work (Browne

& Parsons, 2012; Davern, Shaft, & Te'eni, 2012a, 2012b). Most of the research we reviewed

on individual differences factors in conceptual modelling fell into this category. The findings

of this research identified various factors such as domain knowledge (Bera et al., 2011; Khatri

et al., 2006), background, experience, modelling familiarity (Recker, 2010a; Recker et al.,

2010), cognitive load (Evermann, 2005; Figl, Mendling, et al., 2013), learning behaviour and

capability (Soffer & Hadar, 2007; VanderMeer & Dutta, 2009), understandability and

applicability of the selected ontology for IS professionals (Green & Rosemann, 2004), as well

as familiarity and cognitive load (Masri et al., 2008; Shanks et al., 2010).

The fourth main area of research was to predict and empirically test which social skills

affect the outcomes of conceptual modelling tasks. We only found one review paper on


human factors on conceptual modelling (Topi & Ramesh, 2002), which only discussed studies

published prior to 2002. The finding of this research indicates that researchers mostly focused

on issues relevant to communications between analysts and developers, while the issues

important to communication between analysts and users have apparently received little

attention.

Individual Differences Factors - Identified opportunities in addition to those proposed

by Wand and Weber (2002)

One relevant study was conducted on the conceptual modelling project level (Bandara et

al., 2005). These researchers argued that it is not only the issues and human factors of

analysts, developers, and users that are important, support from the managerial level also

plays an important role in the success of conceptual modelling.

Task Factors – Identified research that addressed opportunities proposed by Wand

and Weber (2002)

We reviewed eight papers that focused on task factors on conceptual modelling. One

paper used more than one empirical method. The main arguments of these articles were that

the differences in task settings (Recker, 2010a), the purpose of conceptual modelling (Green

& Rosemann, 2004; Recker et al., 2010), and different stakeholders involved in conceptual

modelling (Green & Rosemann, 2004) play an important role in the conceptual modelling

process. By contrast, evaluating the strengths and weaknesses of conceptual modelling

grammars, methods, and scripts in the context of different tasks was not the main focus of any

of the reviewed studies (Wand and Weber 2002).

Task Factors – Identified opportunities in addition to those proposed by Wand and

Weber (2002)

Several researchers identified the availability of tools for different tasks as an important

factor in conceptual modelling (Bandara et al., 2005; Recker, 2012). Other important task-

related factors identified in conceptual modelling were domain tangibility (Soffer & Hadar,

2007), the modelling grammar choice in dependence of a task (Bandara et al., 2005), and task

complexity in general (VanderMeer & Dutta, 2009).

52

Social Agenda Factors – Identified research that addressed opportunities proposed by

Wand and Weber (2002)

Only three studies focused on understanding the values and beliefs that underlie

conceptual modelling work in practice. One of the main arguments emerging from the studies

on conceptual modelling context was that the definitions of success may differ by the unit of

analysis (e.g., developer, project, organisation) and that the relationship among these

definitions is complex because a large number of variables affect the success of conceptual

modelling (Larsen, Niederman, Limayem, & Chan, 2009). In terms of the values and beliefs

impacting the outcome of conceptual modelling tasks that have been studied, Hadar and

Soffer (2006) argued that an individual’s perception and interpretation of reality, professional

experience, and perception of model quality may influence an individual’s performance.

Another study revealed that modelling conventions play an important role in the process of

conceptual modelling (Recker et al., 2010).

Social Agenda Factors – Identified opportunities in addition to those proposed by

Wand and Weber (2002)

We found two studies that examined opportunities in addition to those proposed by

Wand and Weber (2002). The first emerging idea is to use knowledge from social networks in

order to improve the quality of conceptual models. Koschmider et al. (2010) developed a

recommendation-based process modelling system that receives recommendations on the basis

of the previous usage of modelling fragments to complete or edit a formal business process

model.

The second emerging idea concerns environmental considerations during conceptual

modelling. Environmental consideration is one of the most important concerns in the current

decade and has gained widespread attention among researchers. He Zhang, Liu, and Li (2011)

extended goal-oriented modelling to model the rationality behind IT systems design, in

particular, how environment-related considerations come into play in such design decision

making.

Social Agenda Factors - Opportunities proposed by Wand and Weber (2002) that

have not been addressed

Not all of the research opportunities in conceptual modelling proposed by Wand and

Weber (2002) that relate to different values and beliefs have been addressed. The remaining


research opportunities are: 1) determining the costs and benefits of adopting different values

and beliefs when undertaking conceptual modelling work, 2) articulating detailed conceptual

modelling procedures that are congruent with different beliefs and values, and 3)

understanding how existing conceptual modelling grammars and methods facilitate

conceptual modelling work under different values and beliefs.

4.4.5 Identified articles that did not match the research framework by Wand and Weber

(2002) altogether

We identified 21 papers related to conceptual modelling that did not fall into any of the

categories or opportunities suggested by Wand and Weber (2002). Sixteen of these studies

were theoretical, one was a literature review, one a commentary, and three were empirical

(one case study, experiment, and one action research). From these, we identified four main

streams in addition to the research framework proposed by Wand and Weber (2002).

A first theme emerging from the studies that did not fit in the research opportunities

proposed by Wand and Weber (2002) was multidimensional conceptual modelling. Trujillo,

Luján-Mora, and Song (2004) proposed an approach for multidimensional modelling using

UML. In this approach, the authors provided a UML-compliant class graphical notation to

conceptualise the dynamic aspects and defined rules about how to use state and interaction

diagrams to model the behavioural aspects of a system (Trujillo et al. (2004). In line with this

research, Garrigós et al. (2012) proposed personalising multidimensional models for on-line

analytical processing to overcome the complexity of understanding large models. Finally,

research by de Albuquerque and Christ (2015) proposed a modelling method to explicitly

model different aspects of a system aiming to improve organisational flexibility.

Another theme emerging from these studies was the quality of knowledge captured and

represented in knowledge engineering. The researchers highlighted the need to separate

knowledge representation into human comprehension and computational reasoning (Chua,

Storey, & Chiang, 2012).

The third main argument was about the usefulness and role of ontologies in conceptual

modelling. Fonseca and Martin (2007) believed that ontologies and conceptual schemas

belong to two different epistemic levels. While ontologies and conceptual schemas are

complementary, they represent distinct aspects of conceptual modelling. Ontologies focus on

explanation and information integration grounded in assumptions about invariant conditions

54

that define the domain of interest, whereas conceptual schemas focus on linking the general

ontological categories with particular observations to be classified in IS. On the other hand,

Weber (2003) and Wand and Weber (2004) argued the usefulness of ontology and discussed

why ontological theories can be used to inform conceptual modelling practice, research, and

pedagogy.

The fourth emerging theme concerns model-driven architecture and different aspects of

model-driven architecture, such as security features in database and process design

(D'aubeterre, Singh, & Iyer, 2008; Fernández-Medina, Trujillo, & Piattini, 2007) and software

configuration and design patterns (Dreiling, Rosemann, van der Aalst, Heuser, & Schulz,

2006; Vergara, Linero, & Moreno, 2007; Wagelaar & Van Der Straeten, 2007).

4.5 GUIDING THE NEXT WAVE OF CONCEPTUAL MODELLING RESEARCH:

A NEW FRAMEWORK

Based on our literature review, we believe that Wand and Weber’s (2002) framework

was useful and necessary at its time. It organised key aspects of conceptual modelling

research to progress and assisted in ascertaining conceptual modelling’s place as a core

research stream in IS. The volume of literature published since 2002 also suggests that the

framework served its purpose of guiding the community of researchers.

However, in our own use of the framework for research and for the purpose of this

literature review, we identified several reasons as to why we believe that a new framework

may be more suitable to guide the next wave of conceptual modelling research than simply

following-up on the outstanding research opportunities identified above. Our main reasons are

the following:

First, Wand and Weber’s (2002) framework is script-centric; it places the creation of

modelling scripts (via grammars, methods and in a context) at the core of modelling activity.

This, for example, makes it difficult to accommodate cases where the modelling activity does

not give prominence to modelling scripts.

Second, the framework is focused on supporting IS development (via modelling).

While IS development is a major part of IS, the existing framework prohibits consideration of

the use of existing IS, interaction with the data provided through an IS (e.g., business

analytics), or indeed any impacts that stem from the use of IS (i.e., outcomes).


Third, the framework is coined by the tacit assumption that modelling is typically

undertaken by professional IS analysts, knowledgeable in appropriate methods and

grammars. Recently, however, the proliferation of content-producing technologies that may

support the creation of digital representations by ordinary people (e.g., Twitter’s hashtags),

raises questions about modelling performed by ordinary people that may be more creative and

spontaneous than the traditional process (Chang, 2010; Lukyanenko et al., 2017; V. Ramesh

& Browne, 1999).

Fourth, consistent with the decades of conceptual modelling research preceding the

framework where many modelling grammars and approaches have been proposed, the

framework emphasises the evaluation of existing grammars, potentially to the neglect of the

design of entirely novel modelling artefacts or approaches. However, the dramatic changes to

the information technology landscape for revisiting traditional design assumptions suggests

the development of novel conceptual modelling methods, grammars, and scripts. An already

debated instance in this context is the use of conceptual modelling for agile development (e.g.,

Erickson, Lyytinen, & Siau, 2005; Lukyanenko et al., 2015), to name just one example.

Fifth, the framework is technology-agnostic. With a steady availability of design

automation tools (Orlikowski, 1993) and the increasing prevalence of technologies with

inherent agency even without human interventions, the modelling of domains, existing or

future, is not necessarily a function of human conceptualisation or behaviour alone any

longer. Mining techniques that automatically construct process models from event logs are a

case in point (van der Aalst, 2011). This calls for consideration of technology support,

enablement, or even embodiment of conceptual modelling.

Sixth, the framework is static and does not explicitly consider feedback resulting from

the creation and use of modelling. This makes it difficult to accommodate multi-stage study

designs, such as action design research (Akhigbe & Lessard, 2016; Sein, Henfridsson, Purao,

Rossi, & Lindgren, 2011) involving modelling phenomena.

In summary, while the Wand and Weber (2002) framework remains reflective of

existing practice and has been useful to the academic discourse until now, it under-represents

the ever-widening spectrum of phenomena that can be supported by conceptual modelling.

56

Therefore, in what follows we propose a new framework with the objective of capturing both

traditional, as well as emerging opportunities.

Key to the new framework is the view that digital representation of reality – which lies

at the core of conceptual modelling research – is becoming a major societal force as

information technology increasingly entwines with all human activities (Leonardi, 2011).

Representations can be either formal or informal conceptualisations of user views and

information requirements, structure, and behaviour of information systems, personal, social,

and business processes, and existing information records. Representations can take the form

of diagrams (e.g., such as ER diagram), but can also include narratives, images, and other

multimedia forms. From a cognitive perspective, the representations we refer to are

considered external representations (J. Zhang, 1997); artefacts that exist outside of any one

individual’s mind and contain knowledge and structure about a domain.

As human reliance on IS for daily functions grows, people routinely reason and act

based on their perceptions of representations of reality stored in digital systems and

increasingly shun direct and traditional interactions. Floridi (2012) coined this on-going

process the “enveloping” of society by an ever-increasing digital layer. We believe conceptual

modelling research brings an important array of theories, tools, methods, and objects of

research to develop, support, and interpret modern digital representations. While

representation is a research object for many scientific disciplines (Hoyningen-Huene, 2013),

the IS conceptual modelling community has unique expertise investigating representations in

the context of information technology. We thus propose a new research agenda of

investigating representations to support the development and use of information and

information technologies. This agenda remains cognisant and incorporates all issues related to

conceptual modelling scripts, grammars, methods, and context that Wand and Weber’s

framework stipulated, but is substantially broader as it explicitly recognises the role of the

conceptual modelling community in supporting a wide range of human interactions with

information technologies. At the same time, it retains the core of the traditional framework, as

the issue of representation constituted a major part of research on conceptual modelling

scripts, grammars, and methods (Browne & Parsons, 2012; Burton-Jones & Grange, 2013;

Kent, 1978; Rai, 2017; Wand & Weber, 1995). Figure 4.3 demonstrates our view of this

framework.


Figure 4.3 A New Research Framework to support future conceptual modelling research

To illustrate the applicability of our new framework, consider several research

directions that follow from it.

First, while Wand and Weber’s framework was script-centric, our new framework does

not insist on this emphasis, which makes it easier to accommodate emerging forms of

representations. As the digital envelope expands, much of this process is spontaneous and

highly creative, through which novel forms of representation are born. Thus, many successful

systems (e.g., Facebook, Twitter) may not implement traditional modelling (e.g., ER

diagrams) or use traditional storage technology (e.g., relational), and their success paves the

way for novel modelling paradigms (e.g., agile modelling, noSQL databases). Many of these

emerging systems explicitly proceed without a modelling script, or use modelling in a

different way (e.g., for feasibility analysis or data interpretation) (Storey & Song, 2017). The

new framework calls for investigation of novel representational approaches and for

assumptions to be made when no script is involved (Kaur & Rani, 2013; Lukyanenko &

Parsons, 2013).

Second, while it remains important to study effective and appropriate representations to

support the development of new IS, with the growth of digital content, novel needs are

emerging. Repurposing data for unanticipated insights is at the heart of the increasing growth

58

of data mining, business analytics, and applied artificial intelligence (H. Chen, Chiang, &

Storey, 2012; Rai, 2017). Here, representations remain critical; however, their role changes –

they no longer guide IS development, but are needed to integrate, visualise, and interpret

massive volumes of heterogeneous data to make informed decisions. Furthermore, different

assumptions made when assembling information for the analytics process may result in

different model performance and predictive power, and may thus result in different actions

being taken.

Third, in moving beyond the conceptual modelling scripts, our new framework enables

exciting new synergies between conceptual modelling research and other research streams

that may be affected by the assumptions behind and the quality of the representations. This

includes studies that investigate the impact of new representations by ordinary users on

information quality, effective use, adoption, and more generally, IS success (Burton-Jones &

Grange, 2013; Lukyanenko & Parsons, 2014; Lukyanenko et al., 2014).

Fourth, as our new framework does not insist on the traditional modelling process, it

supports the emerging practice of information production by ordinary people. Currently, very

little is known about these more spontaneous kinds of models paving the way to an exciting

new direction for conceptual modelling research (Lukyanenko et al., 2017; Recker, 2015).

Fifth, our new framework explicitly recognises the need for ongoing design innovation

in response to technological change. For example, the requirements of open information

environments – where controls over information production are considerably weaker than in

traditional corporate settings, motivating the search for novel approaches to conceptual

modelling that is more adaptable, flexible, and open (P. P.-S. Chen, 2006; Liddle & Embley,

2007; Parsons & Wand, 2014). Likewise, the blooming practice of machine learning and

business analytics may require new forms of representations of data.

Sixth, the new framework proposes feedback as part of a research agenda. We explicitly

recognise that antecedents could influence other antecedents. For example, ontological

assumptions could influence grammars or creator’s capabilities may influence the method

employed in appropriating a grammar. Next, outcomes can have feedback loops to other

outcomes. Using a representation for communication about a domain could lead to better


domain understanding (Anglim, Milton, Rajapakse, & Weber, 2009; Geiger, 2010; Hoffer,

Ramesh, & Topi, 2012; Power, 2011).

Lastly, outcomes can also impact the antecedents to representation. For example, lack of

effective use, adoption, or quality could lead to a change in the creator’s capabilities as the

creator may learn or realise a better way to create future representations to mitigate the issues.

Explicit modelling of feedback in the new framework should provide impetus for more

research of this type.

Our new framework creates significant new opportunities for the conceptual modelling

community and substantially expands what counts as conceptual modelling research. It also

stands to bring different research communities that deal with digital representation (e.g.

information quality and conceptual modelling) into closer contact, promising more

opportunities for cross-pollination of ideas and interdisciplinary collaboration. Our new

framework strongly suggests that conceptual modelling research impacts a broad range of

issues related to information and information technology.

4.6 CONCLUSION

As a research field, conceptual modelling has matured into an established research area

of IS. Perhaps it is not regarded in the same manner as research on technology adoption and

the business value of technology; however, conceptual modelling stands as a cornerstone of

the research discipline.

Yet, the standing and reputation of conceptual modelling within the discipline is not

stable. As with any other field, conceptual modelling research is rightfully under constant

scrutiny in terms of its validity, applicability, relevance, and utility in the ever-changing

world. To cement the place as a research field within IS and surrounding disciplines, it will be

important to constantly review and revise our own research efforts on conceptual modelling.

To that end, in this paper we have taken two important steps. We examined the

influence and consequences of a seminal research framework in the field, and we provided a

new research framework that we believe offers a reinvigorated and exciting new perspective

on conceptual modelling research challenges and opportunities. In doing so, we have created

new pathways to research conceptual modelling that: (a) both relax and challenge our own

assumptions about what conceptual modelling is, and (b) move our research efforts towards

60

the fringes of the conceptual modelling paradigm, to areas where we are required to explore

unknown territory rather than confirm known principles. Our new framework makes an

important step in this direction by drawing attention to significant new opportunities for the

conceptual modelling community and substantially expanding our view of what counts as

conceptual modelling research. It also stands to bring different research communities that deal

with digital representations (e.g. information quality and conceptual modelling) into closer

contact, promising more opportunities for cross-pollination of ideas and interdisciplinary

collaboration. Our new framework strongly suggests that conceptual modelling research is

impacted by and impacts a broad range of issues related to information and information

technology.

In following the agenda set by our work, we may discover that conceptual modelling

has its limits. However, we will certainly increase our confidence in where, how, and why

conceptual modelling is effective and useful – and we may discover that conceptual modelling

has premises and promises that we are yet to foresee.

Chapter 5:Combined Use of Conceptual Models in Practice: An Exploratory Study 61

Chapter 5: Combined Use of Conceptual Models

in Practice: An Exploratory Study10

ABSTRACT

Conceptual models are fundamental to system analysis and design. However, the actual

usage of conceptual models remains poorly understood, in particular, how and why

practitioners would use multiple models in combination when undertaking their work. In this

paper, we explore the reported use of multiple conceptual models for system analysis and

design to determine the circumstances that lead professionals to use multiple models. We

uncover both semantic and pragmatic reasons that influence the choice and selection of

different models for system analysis and design tasks. Contrasting these findings to existing

ontological theories demonstrates that the extent and type of multiple model use is determined

not only by ontological factors, but also contextual factors that can override ontological

qualities, and in so doing bring forth desired qualities for users. We offer several novel

propositions about the implications of ontological theory that will be worth exploring in

future research.

Keywords: combined use, conceptual models, ontology, context, completeness, semi-

structured interviews, representation theory, ontological analysis, qualitative research

10 Published as: Jabbari Sabegh, M. A., & Recker, J. (2017). Combined Use of Conceptual Models in Practice:

An Exploratory Study. Journal of Database Management, 28(2), 56-88.

62

5.1 INTRODUCTION

System designers and analysts often begin their work by developing and using graphical

representations of relevant features of the domain under examination (Burton-Jones & Meso,

2006). These representations are called conceptual models (Wand & Weber, 2002). They play

a significant role in the early detection and correction of system development errors and assist

analysts to better communicate with stakeholders (Moody, 2005; Saghafi & Wand, 2014).

Much research on conceptual models and the grammars with which they are created has

been conducted over the past decades, often using ontological analysis (Wand & Weber,

1990a, 1993, 1995). Ontological analysis allows researchers to suggest how grammars for

conceptual modelling might be modified to be ontologically sound and how well developed

models that correspond to ontological guidelines are understood. There is a strong track

record of studies showing empirical support for these guidelines (Saghafi & Wand, 2014).

Yet, noticeably, this rich research tradition has focused almost entirely on the evaluation of

single grammars (such as UML, ERD, BPMN, and so forth) or single models (such as those

that are ontologically clear versus unclear).

One important insight from this stream of research is that no single available grammar

is ontologically complete (e.g., Irwin & Turk, 2005; Opdahl & Henderson-Sellers, 2002;

Recker et al., 2009; Wand & Weber, 1993; Weber, 1996). This situation implies that users

will never be able to create a single model that fully represents all of the relevant aspects of

the real-world phenomena they wish to have represented.

This situation is not necessarily problematic; in fact, incompleteness seems to be a

design choice for many popular grammars: UML, for example, provides fourteen different

grammars to describe structure, behaviour, and interactions of a system from a variety of

perspectives, each of which is, by necessity, incomplete (Rumbaugh, Jacobson, & Booch,

2004). Other longstanding methodologies, such as Multiview (Avison & Wood-Harper,

1986), have promoted the use of multiple models with different perspectives for close to thirty

years.

In this paper, we explore two questions that follow from this situation: How do analysts

and designers deal with the fact that any model they have available is not a complete

representation? How do they select from a variety of possibly available models given that


each will only offer some representation but never a complete one? The proposition that we

put forward is that they use multiple models in combination such that the completeness of the

representation of their relevant real-world phenomena can be maximised.

We are not the first to make this proposition. Theoretically, this proposition has been

explored, firstly, by Weber (1997) and Green (1996) who suggested two principles, maximum

ontological completeness and minimal ontological overlap, to explain why designers might

select different grammars for conceptual modelling. More recently, Recker (2014) suggested

theoretical arguments in a theory of faithful use of conceptual model combinations. Yet, what

remains notably absent is empirical knowledge about how practitioners work with multiple

models. We take this step in this paper and explore two broad research questions:

RQ1: Why are system analysts and designers using multiple conceptual models?

RQ2: How are system analysts and designers using multiple conceptual models?

Our contribution is primarily empirical: we set out to explore current practices in the

actual use of multiple conceptual models, which is surprisingly scarce in the literature. In

doing so, we also provide theoretical contributions: through our work, we identify a range of

contextual and contingency variables that impact on the use of conceptual models; and we

will show how these variables extend upon the understanding that can be generated through

ontological analysis alone; and in turn, how ontological theory might be improved.

We proceed as follows: First, we review prior ontological analysis and empirical studies

on conceptual modelling. The research method and how we designed our research is then

discussed. Section 5.4 reports the results in two parts, in line with our research questions.

Section 5.5 summarises the main contributions, outlines implications for the research and

practice, and reviews potential limitations of the work. Section 5.6 closes by drawing some

conclusions from our work.

5.2 BACKGROUND

Two streams of research informed our study: first, existing theoretical research on the

ontological analysis of conceptual modelling grammars and models; second, empirical

research on the use of conceptual models in practice. We discuss each, in turn below.

64

5.2.1 Ontological Analysis as a Theoretical Foundation of Conceptual Modelling

Research on conceptual modelling is an active research area in information systems

research (Burton-Jones et al., 2009). Much of this research draws on Wand and Weber’s

(1990a, 1993, 1995) representation theory as a means to evaluate, explain, and improve

conceptual modelling grammars and their usage (e.g., Gemino & Wand, 2005; Green et al.,

2011; Recker, 2010b; Recker et al., 2010).

Representation theory builds on the philosophical field of ontology (Bunge, 1977) to

inform evaluations of conceptual modelling grammars and the models (also called scripts)

that can be generated with them. The theory postulates that conceptual modellers are

concerned with building models of some relevant phenomena or domain in the real world

(Burton-Jones & Weber, 2014) and that ontological factors, such as the completeness and

clarity of a representation of real-world phenomena, can be used to predict the strengths and

weaknesses of conceptual modelling grammars and scripts, and ultimately conceptual

modelling success.

In essence, Wand and Weber’s theory suggests two main evaluation criteria: ontological

completeness and ontological clarity. Ontological completeness is measured by the degree of

construct deficit, that is, the extent to which grammatical constructs exist that allow users to

represent the entirety of the real-world phenomena they desire to be represented in a model.

Ontological clarity is measured by the degrees of construct overload (where a grammatical

construct can represent multiple aspects of a real-world phenomenon), construct redundancy

(where some aspects of a real-world phenomenon can be represented by multiple grammatical

constructs), and construct excess (where a grammatical construct does not represent any real-

world phenomena). Deficiencies in terms of ontological completeness and ontological clarity

are typically established through a procedure that has become known as ontological analysis

(Fettke & Loos, 2005; Gehlert & Esswein, 2005; Hadar & Soffer, 2006; Rosemann, Green, &

Indulska, 2004; Rosemann, Recker, Green, & Indulska, 2009; Wand & Weber, 1993; Hong

Zhang et al., 2007).

Ontological analysis based on representation theory has been used in various studies to

examine the predictions of ontological completeness and clarity on modellers’ ability to

develop clear and complete models of real-world phenomena using different modelling

grammars. Structured overviews of these studies are provided by Green et al. (2011), Burton-


Jones et al. (2009), and Recker et al. (2009), amongst others. A meta-analysis of empirical

studies to date shows that the theory is indeed able to make largely valid predictions about the

use of conceptual modelling in practice (Saghafi & Wand, 2014).

Two aspects of this stream of research are noteworthy. First, virtually all of the existing

research has been on single grammars or single models, examining questions such as whether

ontological deficiencies in a grammar lower perceptions of grammar usefulness (Recker et al.,

2011), or whether the existence of ontological deficiencies in a grammar inhibit users’ ability

to faithfully describe a particular real-world phenomenon in a model constructed with that

grammar (e.g., Bodart et al., 2001; Parsons, 2011; Shanks et al., 2008). Second, all

ontological analyses performed to date have shown that all current available grammars have

some level of construct deficit (see Table 5.1). In other words, no grammar, and therefore no

single conceptual model, is ever ontologically complete: some aspect of real-world

phenomena cannot be expressed. This state is intuitively reasonable: most grammars have

been developed with a clear purpose in mind. For example, data modelling grammars, such as

ERD, have been developed to model substance and form in the real world – but not possibility

and change (Burton-Jones & Weber, 2014). Process modelling grammars, by contrast, model

behaviour, change, and dynamics – but not things and their properties. Moreover, often not all

constructs provided via a grammar are used in any given model (Siau et al., 2005; zur

Muehlen & Recker, 2008). In other words, most models are even less complete than the

grammars with which they have been created.

Nevertheless, any information system that is being analysed or designed incorporates

both aspects: substance and form (Which data needs to be stored about objects? Which

attributes characterise these objects?), as well as dynamics and behaviour (What happens to

these objects? Which processes are supported by the system?). During analysis and design,

therefore, users are likely to desire a more complete representation than any one model can

provide. In this paper, we therefore ask: How do practitioners achieve the level of

representation they desire?

66

Table 5.1 Examples of studies on ontological completeness of modelling grammars

Reference Example of Grammars

Evaluated

Identified Construct Deficit

Irwin and Turk (2005) Use Case Modelling

Grammar

e.g., system structure, system

decomposition

Recker et al. (2010) BPMN e.g., history, system decomposition

Wand and Weber (1989, 1993);

Weber (1996)

Entity Relationship

grammar

e.g., transformation

Data Flow Diagram e.g., state law

Green and Rosemann (1999) ARIS e.g., decomposition, state law

Weber and Zhang (1996) NIAM e.g., event, history

Green et al. (2007) BPML e.g., things, state law

ebXMLBPSS e.g., kind, system decomposition

BPEL4WS e.g., history

WSCI e.g., things, state law

Recker et al. (2009) Petri net e.g., property, history

IDEF3 e.g., class, kind

Merise e.g., state, system

ISO TC87 e.g., class, state

ANSI Flowchart e.g., things, property

Opdahl and Henderson-Sellers

(2002)

UML e.g., conceivable state space, lawful event

space

5.2.2 Empirical Studies of Conceptual Modelling Practice

We reviewed the empirical literature with two goals in mind: first, is there evidence to

support the assumption that professionals use more than one model or one grammar in their

tasks? Second, if so, how do they go about achieving the level of representation they require?

We quickly realised that practitioners do indeed often use more than one conceptual

model in their design and analysis tasks (Davies et al., 2006; Dobing & Parsons, 2008; Fettke,

2009). For example, Recker (2012) reported that over 30% of surveyed process modellers

access additional grammars when modelling business processes. Green et al. (2011) showed

that 80% of users of modelling tool environments select and use multiple grammars in

combination when engaging in conceptual modelling. Dobing and Parsons (2008) reported

that 90% of UML users employ at least two different UML grammars in at least one-third of

their projects.

We also realised that the literature is scarce with regards to the reasons for how and why

users use multiple grammars or models. Weber (1997) argued that if analysts or designers

encounter ontological incompleteness, they will somehow have to augment their model(s).

One plausible strategy is to use additional grammars or models, as indicated in some studies

(Green et al., 2011; Recker, 2012). Other reported workarounds include annotations and


colour highlighting (Recker et al., 2010), the reliance on organisational conventions (Samuel

et al., 2015), or the augmentation of models with textual descriptions (Gemino & Parker,

2009).

Another finding from our review was that empirical studies on multiple conceptual

models use are scarce, albeit their results are promising. We found three empirical studies on

multiple models (Gemino & Parker, 2009; J. Kim et al., 2000; Siau & Lee, 2004). The

findings of these three studies all implied that using multiple models results in better

performance than using a single model.

Thus, our conclusion from this review is that: a) multiple model use is an empirically

established phenomenon, with b) ascribed but not conclusively demonstrated theoretical

benefits, and c) that substantive knowledge is lacking with regards to at least three questions

concerning why users would use multiple models and how:

1) How do users realise that one model may not be sufficient for the tasks they set out

to complete?

2) Can the rationale for multiple model use be linked to ontological factors, such as

the completeness and clarity of representation?

3) Which contingencies, such as conventions or other workarounds, exist that

influence the decisions users make?

We now report on our exploratory study that aims to provide some answers to these

questions.

5.3 RESEARCH METHOD

5.3.1 Design

To answer our two broad research questions, we decided on an exploratory, qualitative

research design. Specifically, we opted for semi-structured qualitative interviews. Semi-

structured interviews are a main data collection tool in qualitative research (Yin, 2009). In our

case, they enabled us to examine our phenomenon of interest from the model users’

perspective (Benbasat et al., 1987; Myers & Newman, 2007; Yin, 2009).

To ensure reliability and validity, we mainly followed and applied the guidelines of

leading positivist qualitative researchers (Shanks, 2002; Yin, 2009). We used the extant

68

literature on empirical and ontological research on conceptual modelling to build a semi-

structured protocol to guide our interviews. The protocol is available in Appendix B. We

designed the protocol such that we could obtain information on the extent of and reasons for

the use of multiple models (Dobing & Parsons, 2008; Fettke, 2009; Green et al., 2011), the

contextual factors involved in the use of the conceptual models (Recker, 2010a, 2012; Wand

& Weber, 2002), and the potential impact of ontological deficiencies on the use of multiple

models (Recker et al., 2010; Recker et al., 2011).

The use of the semi-structured interview protocol helped us to be consistent in

structuring and conducting the data collection processes. It thereby assisted us in maintaining

the reliability of the collected data. For example, it assisted us with clear terminology; in the

literature several relevant terms have various definitions (e.g., a conceptual model), as do

factors that possibly influence the use of conceptual models (such as familiarity with different

types of models, existing domain knowledge, ontological factors). Our protocol allowed us to

use relevant terms understandably and consistently. With these considerations in mind, our

protocol consisted of four parts:

1. We started by clarifying terms used during the interview, such as conceptual

models (graphical representations vs other schemas such as XML schemas or meta-

models), interpretation (human-readable models vs machine-readable models), and

others.

2. We collected demographic information and performed control checks to be sure

that the interviewees were indeed users of conceptual models and whether they had

different types of conceptual models available for use.

3. In the main part of the interview, we asked questions regarding the interviewees’

use of conceptual models. For example, we asked how often practitioners used

conceptual models, whether they used multiple types of conceptual models, why

they used different models, and what types of models they used for which purposes

and tasks.

4. Finally, we collected information about the familiarity of participants with different

types of conceptual models to be able to control for potential preferential biases

(Figl & Recker, 2014) in choosing or using conceptual models. This part also


contained questions for individuals who reportedly did not use combinations of

conceptual models, which was useful to explore boundary conditions.

Throughout all interviews, we followed the interview protocol closely to increase the

reliability of data and to ensure that the same questions were asked of every interviewee.

5.3.2 Participants and Procedures

The population of interest for our study consisted of practitioners in systems analysis

and design roles that used already created conceptual models for their work tasks (as opposed

to modelers who would create such models). We used purposeful sampling techniques for

participant selection for this study instead of random sampling (Eisenhardt, 1989).

To approach potential participants, we initially recruited UML practitioners from

LinkedIn groups, such as UML professionals, Business Analyst Professionals, Software

Developers, UML, UML Lovers, and Executable UML. We began with this focus for two

main reasons. First, UML is reported to be one of the most widely used conceptual modelling

methods and a de facto standard during information systems development (Davies et al.,

2006; Fettke, 2009). Second, UML provides a wide variety of model diagrams, each of which

conveys different information (Dobing & Parsons, 2008), which increases the likelihood that

UML practitioners engage with multiple types of conceptual models during their work.

However, we did not limit our sample to only UML users. We also approached other

systems analysis and design practitioners by searching for LinkedIn professionals who were

endorsed for skills in requirement analysis, system analysis or development, software design,

engineering, architecture, analysis, or object oriented design. Through this sampling

procedure we also recruited practitioners with expertise in EPC or SysML, for example.

To offset potential bias stemming from particular modelling roles (Recker, 2010a) and

to accumulate a broader view on the actual use of conceptual models in practice, we recruited

interviewees from different companies and with different roles. The variety of organisational

roles in our interview sample was important for us to investigate conceptual model use from

both technical and business aspects.

Using these procedures, we collected interview data between December 2015 and May

2016 from 15 individuals, across eight countries, incorporating various roles in their

respective business environments (e.g., system/business analyst, system engineer, software

70

architect). We conducted our interviews mostly through Skype, as most of our participants

were from countries outside of Australia. Interviews with locally available participants were

conducted in their office or preferred meeting place. All interviews were audio recorded and

transcribed by external transcribers upon reception. Each interview lasted between 45-90

minutes. Overall, we obtained over 18 hours of interview data, resulting in 372 pages of

interview transcriptions for analysis. Table 5.2 provides basic demographic information. The

reported experiences classified our interviewees as highly experienced professionals.

Table 5.2 Demographic information

Role and Responsibilities # of Interviewees Experience in years Mean Min Max

Business Analyst/consultant 6 Experience in Business 23.74 7 38

Software Architect 3

System Engineer 2 Experience in IT 21.8 7 35

System Architect 2 Experience as a System

designer/analyst 15.3 7 25

Software Developer 2 Experience in Conceptual

Modelling 15.2 5 25

Total 15

We conducted interviews until we reached a sample size we deemed appropriate and

substantial for our analysis, on the basis of guidelines by Guest, Bunce, and Johnson (2006)

and Francis et al. (2010). Specifically, we tracked the derivation of new knowledge from each

interview. We did this by interviewing, transcribing, and analysing each interview in

sequence, and plotting new codes emerging from each interview analysis and the

accumulation of codes for all interviews (Figure 5.1) until the trends indicated data saturation.

Figure 5.1 shows that we achieved saturation after approximately thirteen interviews; thus, we

ceased data collection after completing our 15th interview.

5.3.3 Data Analysis

We analysed the transcribed the data using NVivo 10. We used multiple coding

techniques commonly associated with the grounded theory method (Urquhart, 2013;

Urquhart, Lehmann, & Myers, 2010). Specifically, during open coding we analysed each

interview line-by-line to identify concepts or key ideas potentially related to our research

questions. Table 5.3 provides several examples of how we guided our coding through the

open codes collected initially.


Figure 5.1 Interview sampling and theoretical saturation

Table 5.3 Examples of open codes collected

“Nugget” from interview transcript Initial open code

“So, an information system is vastly more than the paradigm of process only or data

only, it’s a whole lot of interrelated considerations.”

Size of domain

“I don’t think there’s a single model to rule them all, no. I think within domains I

expect different model types, greater or lesser.”

Deficit in model

“Every single diagram does not give you an accurate description of the system, it's by

the combination of the diagrams that you get the complete description.”

Complementary use of

models

“We use BPMN for all [of] the process models, and that's mandated, we must use

that.”

Mandated by

organisation

“It is basically repetition. You are basically repeating yourself over and over in three different dimensions that are saying the same things.”

Redundant information

We then used selective coding techniques to categorise and assemble the identified

concepts to see whether we could match the identified concepts to existing theoretical patterns

based on our understanding of the literature (such as ontological qualities), or alternatively to

define new patterns grounded in our interpretation of the data. Figure 5.2 illustrates how we

tracked and categorised codes in our attempt to explain why practitioners used different types

of models, and how they selected the types of models.

72

* Numbers indicate the frequency of code across all interviews.

Figure 5.2 Illustration of coding procedure

5.4 FINDINGS

We discuss our findings in two main steps, in alignment with our research questions.

We start by exploring our data regarding why practitioners use multiple types of models in

combination. We then explore the factors were unearthed that influence selection and use of

different types of models in combination.


5.4.1 Why practitioners use multiple types of models

Not surprisingly, all of our interviewees (15 out of 15) reportedly used more than one

type of models in their design and analysis tasks. One interviewee reported: “… a lot of the

time I’m interested in using [different] models as a view onto the domain from different points

of view and then having links behind the scenes between those representations.” In fact, using

multiple types of models seemed to be usual for practitioners; an interviewee noted: “I need

two types of diagrams, at least two, one structural that shows the physical topology of my

domain […] and the dynamic view of it.”

Table 5.4 summarises the types of models reportedly used by at least two interviewees.

The results are reasonably consistent with other studies (Dobing & Parsons, 2008; Grossman,

Aronson, & McCarthy, 2005) even though we focused on model users as opposed to model

designers and we did not only focus on UML diagrams but also other model types (e.g.,

BPMN). Moreover, our results show a higher usage for activity diagrams, state machines

diagrams, and BPMN diagrams compared to previous studies. Still, we learned that, as

expected, professionals use UML selectively, but never completely (i.e., all available model

types).

Table 5.4 Most widely used models

Type of Models Number of

Interviewees

Purpose of Use (Number of Interviewees)

Class Diagram 13 Capture and define requirements (10), Collaborate with developers and engineers (5), Documentation (1), and Database design (3)

Use Case Diagram 8 Capture and define requirements (8), Collaborate with business people

and end users (5), and Scoping the project and estimations (1)

Activity Diagram 8

Collaborate with business people and end users (3), Collaborate with

developers and engineers (1), Documentation (3), and Basic

development design (2)

State Machine

Diagram 7

Collaborate with developers and engineers (7) and Documentation (5)

BPMN 6

Collaborate with developers and engineers (5), Documentation (5),

Change Management (1), Migrate to a New System (1), and Validate

Processes (1)

Sequence Diagram 5 Collaborate with developers and engineers (4), Documentation (3),

and Validate Processes (1)

Deployment Diagram 3 System Configuration (2) and Aggregation of system parts (2)

Component Diagram 2 Collaborate with business people and end users (1), Aggregation of

system parts (1)

Use Case Scenario 2 Capture and define requirements (1) and Documentation (1)

Entity Relationship

Diagram 2

Database design (2)

Object Role Model 2 Collaborate with developers and engineers (1) and Scoping out

information and data requirements (1)

74

Table 5.4 also summarises the main purposes for which practitioners reported use of the

different models. Note that a model could be used for multiple purposes. For example,

practitioners mostly stated that they employed use case diagrams for both capturing and

defining requirements and also for communicating with business people and end users.

As presented in Table 5.4, practitioners used different types of models for different

purposes. The most notable result is that practitioners used various types of models to

collaborate and communicate with users vs. analysts. For example, class diagrams were

mostly used to collaborate with developers and engineers, while use case diagrams were

mostly used to collaborate with business people and end users.

These findings are consistent with earlier studies. For example, Topi and Ramesh

(2002) noted differences in model-based communications between analysts and clients (i.e.,

end users) versus that between analysts and developers. They argued that more expressive

models can convey more information to model readers; however, this would be more useful to

analysts than clients. Our data likewise suggests that practitioners often trade-off complexity

of models to promote simplicity. An analyst cited: “When you go to, let’s say, a process

owner, [...], you can’t show her a sequence diagram or a class diagram. She simply won’t

understand […]. But you can show her something like an activity diagram, or use case

diagram. So, it depends on your communication partner.”

Reasons for the use of multiple models were more varied. At the outset, multiple types

of models were not always and consistently used by practitioners in their system design and

analysis tasks. Instead, practitioners chose sets of multiple models based on several conditions

of their tasks and initiatives. These included: 1) the size and complexity of the domain, 2)

involvement of different stakeholders in the projects, and 3) incompleteness of any particular

model. We explore each factor in turn below.

Size and complexity of the domain

One of the first findings that became apparent in our study was that system designers

and analysts may not need to use graphical representations of a system at all to gain an

understanding of the domain. When dealing with a small and simple domain, they simply use

a single model, if at all. Practitioners reported that this often provided them with enough

information to help them to achieve their tasks goals. A small and simple domain was


characterised as one involving only a few interacting elements with stable and clear cause-

effect relationships. For example, a “loan payment” system was described as having a limited

number of elements with direct and stable relationships in comparison with a system with a

large number of interconnected components with non-linear relationships, such as banking.

As one system analyst commented: “So, a small project, I might get by with one class

diagram […]. For a bigger project, I might want to describe certain key processes that really

do matter […], using different diagrams.”

We also learned that for small and simple project development processes, conceptual

modelling may not only not be required, but could also slow down the development process.

However, for initiatives involving systems in big and complex domains, interviewees

consistently noted that it would be challenging to understand the domain through the use of a

single model only. One practitioner noted “[…] because our system is far more complex than

either data or process [model]”; while another mentioned “[the] banking domain is way too

complex to represent on one type of diagram”.

We learned that practitioners do indeed use conceptual models as specific abstractions

of a real-world domain from a particular perspective. When a system to be represented is

complex, multiple interrelated models were used to represent different aspects of the system,

much as suggested in the study by J. Kim et al. (2000). A practitioner commented: “Yes, in

order to depict everything that happens in a wire transfer, for example, I’d need static

diagrams, component and class diagrams, but also behavioural diagrams, like activity.”

Practitioners had their own way of classifying abstraction perspectives, often guided by

available textbooks that provided simple classifications, for example, categorisations into

behavioural and structural (Booch, Rumbaugh, & Jacobson, 2005). Figure 5.3 summarises the

most often named model types associated with a particular systems design perspective. It

shows that practitioners mainly used class diagrams or component diagrams to depict the

structural perspective of a domain, while activity diagrams or state machine diagrams were

used to describe the dynamic perspective of a domain. As one system analyst noted: “I need a

state machine to capture what’s going on inside it, and again if it really is a big project, class

diagrams will be good to describe certain key low-level aspects of the system.”

76

Figure 5.3 Example of diagram use to represent different aspects of a system

We further learned from our data that practitioners felt that understanding a complex

domain through the use of a single diagram alone would be challenging. A practitioner

commented: “Where that starts to be a real challenge, is where that system is very complex,

as it starts to include more and more aspects….” In fact, our interviewees consistently

commented that they often dealt with large and complex systems that included a large set of

interrelated aspects. In these cases, using different models in combination improved their

domain understanding. A user commented on why they used a combination of models: “Well,

the short answer is ideally you want to do everything. You want to get the best understanding

of a system as possible.”

Involvement of different stakeholders

Different stakeholders are involved in systems analysis and design, all of which require

an understanding of the relevant problem domain in some form or other (Darke & Shanks,

1997). In the context of our research, and akin to (Topi & Ramesh, 2002), our data suggested

that two categories of stakeholders are relevant in terms of the use of multiple models:

stakeholders as systems users and stakeholders involved in systems analysis and design.

Stakeholders as systems users occupied roles such as process owners, technical personnel, and

lay end users. Stakeholders involved in system analysis and design were managers, systems

analysts, and developers. Importantly, we learned that each of these types of stakeholders may

require different models. A system analyst commented: “My communication partners

currently are, as I said, line management, lower management, technical personnel in the


bank, analysts, and my own developers. And for all, each group, I need different sets of

diagrams.”

We further learned that practitioners use different types of models that are

understandable by different stakeholders in order to be able to communicate with all of the

stakeholders involved in the project. They do so because individuals involved in the project

glean a different level of understanding from different models. Out of our 15 interviewees, six

mentioned that they used multiple, different types of models when they wished to be able to

communicate with an entire project team beyond the technical staff responsible for analysis

and design. A system analyst commented: “When you go to a lady who basically just ticks off

issues in an issue management system, you can’t show her [a detailed diagram]. She simply

won’t understand, and within ten seconds her eyes will glaze over, and you’ve lost her.”

Another practitioner stated: “If you go and seek software engineers and you turn up with

something a bit too simplistic, they wouldn't take you seriously.”

Our data also suggests that not all stakeholders in a project desire to obtain the same

level of understanding about the details of a system. Managers and decision makers normally

wish for an overview representation of the entire system. According to our interviewees,

developers and analysts want to obtain a deep level of understanding about the system or

domain. A practitioner commented: “When you are talking to business people, maybe you

don’t like to use state diagrams because it’s too technical in most cases. […] but when talking

to business people they want to see simple diagrams, and they want to see what standard

value is.”

Incompleteness of single models

In probing for reasons why practitioners use multiple models, we also learned that, as

predicted by ontological analysis, ontological deficits of a model provided some key

explanations. Eighty percent of interviewees frequently used phrases such as “expressive

power”, “deficit”, or “ability to capture information”, all of which allude to the existence of

ontological deficit or representational incompleteness of a model. As one interviewee noted:

“Sometimes there is no language that represents some concepts in the domain.” A second

interviewee commented, “I mean, I don’t think you have a modelling technique that

succinctly captures issues in all of the areas in which you are trying to conceptualise the

system.” A third practitioner argued that he used different types of models: “Basically

78

because I can express certain aspects of the whole system [only] with specialised kinds of

diagrams.”

5.4.2 How practitioners decide on which models to use

Above we reported that all interviewees indicated that they used multiple models, at

least two, in most of their design and development processes. Reasons for doing so included

stakeholders, domain size, and complexity and incompleteness. This behaviour raises the

question of how practitioners select what types of models to use in their design and analysis

tasks. In this section, we explore the reported rationales that determined the selection of types

of models. We structure them broadly into semantic reasons – selection rationales grounded

in the capacity of models to represent a domain, and pragmatic reasons – selection rationales

grounded in the socio-technical context of conceptual model use.

Semantic Reason 1: Select models that increase the completeness of a representation

The most eminent reason for the selection of multiple models noted by our interviewees

was their desire to maximise the completeness of representation. To do so, almost 80% of

interviewees mentioned that they use additional models to that end. One analyst noted, “Well,

the reason you need [more than one model] is because each one of those diagrams only

depicts certain properties of the system”. Another practitioner noted, “In order to depict the

system completely, or more fully probably, so […] you need to use multiple diagrams to

depict it.” This suggests that system designers and analysts select those models in

combinations that are complementary to each other and enable them to achieve a maximal

level of ontological completeness.

The main way that practitioners attempted to achieve completeness was to select models

that featured alternative perspectives, from structure to behaviour, for instance. A user

commented: “Every single diagram does not give you an accurate description of the system,

it's by the combination of the diagrams that you get the complete description. So, I have to

look at my system from different angles. The structural angle, the behavioural…”

However, some practitioners selected models that described the same or similar

perspective as another, but provided more detailed information about the requirements within

a specific aspect. For example, some mentioned that if they were particularly concerned about

the behavioural view of the domain, they selected those models that provided more


information about the behaviour of the domain. For example, a system designer commented:

“So, the areas that I’m not comfortable with I tend to model, and say if it’s behaviour I’ll be

leaning on the dynamic models more, so the activity diagrams, the state transition or the third

one.” However, if their main concern was about information and structure of the domain they

selected the types of models that provided them with more detail in that view. Another system

designer noted: “The basic system design is by looking into the use cases and a collaboration

which tells you “Okay, I have an idea, this and that must be done, and I kind of need business

[uptakes] doing certain things and having certain attributes”. So then you have a basic class

model, and this class model is again linked to the use cases from where you derive it and the

collaboration.”

This data suggests that on the one hand the selection of the models is based on which

model gives more information, but on the other hand, also depends on which types of

information a user requires. In other words, selection also relates to the purpose of the task,

which we explore further below.

Semantic Reason 2: Select models with little overlap to minimise cognitive demands

The second rationale in the selection of models we found was that practitioners tried to

avoid overlapping models because they add unnecessary complexity. A practitioner noted:

“I’m trying to avoid a situation where I was using a mix of functional representations. You

know, I'm trying to stay with only one sort of representation in the functional view or in the

information view or whatever. Otherwise, it starts to get messy.” A second analyst noted:

“Do I allow competing model types that attempt to describe the same sorts of things using

different formats? […] Absolutely not, it is a disastrous practice, it always ends badly, it

always causes problems.”

When practitioners referred to terms such as “overlap”, “competing”, or “redundant

models”, they also consistently referred to terms such as “problems”, “complexity”, or

“cognitive effort” that such selections lead to. In other words, practitioners frequently

commented that high overlap between models would increase the cognitive demands on those

trying to use the models. For example: “[for] a general user community three different models

with three different conventions all attempting to describe the same sort of thing but

describing it in different ways – that’s a pretty big cognitive task for your average user to

have to get through.”

80

We also learned from three practitioners that cooperating with different stakeholders

may lead to an increase in overlapping models if and when those stakeholders favour

particular models and representations. A system analyst mentioned: “So, I’ve done some

database work where the software development has used UML class diagrams to represent

the logical data model but then there’s been a database expert who’s used an entity

relationship model from the same project to also represent the database, and generally

speaking we try to integrate these things and if we can use a single language, we use a single

language, because otherwise it just adds complexity.”

Semantic Reason 3: Select models that establish correspondence between them

We also learned that the extent of overlap mattered. Two practitioners commented that

in selecting models, they select those models that have some correspondence between them.

Correspondence in this context refers to the existence of semantically related or equivalent

elements that appear in multiple models (such as a construct representing “the customer” in

both a process and a data model). Model users believed that this correspondence between

models helped them to link models together. A practitioner indicated: “I mean, there are what

I call 'touch points' between the different paradigms there because you will have things that

are referred to in a business process functional view which are the artefacts which also

appear in the information log. There are these touch points between the different views. But

that's the value in having […] the diagrams representing a model because you can then start

and do checking between the different views.”

While six interviewees commented that overlap between models may increase

confusion and increase complexity, the existence of some level of correspondence between

models was also noted as a useful means to help users to better link different models and in

turn improve the understandability of the information.

Only one of our interviewees stated that she attempted to keep the total number of

models as minimal as possible because correspondences also present a burden. She believed

that too many models would make it too difficult for readers to assimilate the information: if

the number of models increases, readers need to find links between models and map them to

their own mental model of the phenomenon.


Pragmatic Reason 1: Select models based on organisational regulations and

standards

What types of models were selected also depended on organisational settings. Most

organisations define some internal regulations in a set of modelling conventions (Delfmann,

et al., 2015; Rosemann, 2006; Samuel, et al., 2015) to facilitate analysis and design of

information systems, as noted by a system designer: “It [use of models] depends on the

standards and the template that companies define.” Most comments showed that the selection

and use of conceptual models fairly strictly follows these predefined regulations:

“Organisations will rarely just leave it to an individual project team or an individual to just

arbitrarily choose whatever technique. They typically have these quality standards which they

set…”

We found that 60% of the practitioners we interviewed were indeed governed in their

use of models through organisational conventions. While these take different formats, the

most common objective of these guidelines was harmonisation. An interviewee commented,

“We have standards based on how we define for the sake of consistency.”

In terms of contents of such conventions, we learned that most of the organisations had

predefined regulations regarding not only on what types of models should be used but also

how these models should be represented. An analyst mentioned, “They create not only

standards of which techniques to use, but most importantly […] what conventions in these

techniques you should use.” This involved predetermination of which types of models were

available to use. For example, a practitioner commented “Yeah, in our case we use UML. We

use BPMN 1.1. We use activity diagrams. We use class diagrams. These are the agreed upon.

We don’t use ERD and we don’t really use context diagrams. So, those are the agreed upon.”

Moreover, the organisational conventions in use specified the types of model constructs that

should be used in specific occasions. A practitioner commented, “They have the standard on

[….] what specific constructs you should use and when you should use them and what degrees

of modelling apply to them and how you should describe [them].”

In summary, we learned that organisational regulations, if they exist, can overrule or

mitigate some of the semantic rationales for selection rooted in the models themselves (such

as completeness or overlap). For example, some organisations explicitly required

documenting a system in types of models that overlap with each other: We do use it right now

82

because it is an agreed on the requirement, which is a lot of work. But asking the question of

relevance. Is it always required? No. We are actually challenging that; we’re saying “No,

this is a lot of work”. It is basically repetition. You are basically repeating yourself over and

over…”

Pragmatic Reason 2: Select models based on involved stakeholders

Involvement of different stakeholders not only leads model users to use different types

of conceptual models but also affects the types of models IS professionals choose to use. An

analyst commented: “So, I think that the selection of a modelling language should be

influenced by your target audiences that you want to communicate with.” Different audiences

may require different types of models because not all audiences desire the same type of

information. A practitioner commented: “You've got the very high-level executive decision

makers, especially the ones who shine off of all the money and what they want is different

from the IT technical people who will actually implement the system.”

This situation also implies that practitioners may use the same types of models with

different stakeholders, but with different levels of details because not all stakeholders seek to

gain the same level of understanding of the details of the system. One practitioner mentioned:

“I may use use-case diagram with technical people, but then I would use it differently”, while

another practitioner stated: “So, who I want to say it to influences the type of diagram that I

choose and saying it concisely means that I want to restrict to an absolute minimum. The

things in that diagram should just include the things that I needed to include in order to

communicate whatever it was I wanted to say.”

We learned that practitioners select different types of models based on their audience’s

targeted level of understanding from different diagrams. A practitioner noted: “So you’re

trying to get a feeling for what type of diagram your communication partner would or could

intuitively understand and you choose that one. That’s why there’s no problem using a

sequence diagram or a class diagram with a developer. Most of them understand it. But if we

talk […] let’s say with a line manager, you’re basically reduced to component diagrams, and

for the behavioural part, you’d be limited to activity diagrams. […] So, I use activity

diagrams with them based on their level of understanding.”


We also learned that practitioners choose types of diagrams based on the target audience

skill level and familiarity with models. A system analyst commented: “With one of them, he’s

a senior experienced developer. He immediately understands something like a sequence

diagram, for example. With one of them, he’s […] very talented, but also very junior. So, I

can basically only use class diagrams or component diagrams.”

Finally, we uncovered that stakeholders sometimes had their own preferences and liked

to use specific types of models due to different purposes or because their previous projects

involved specific types of models, that is, they had already used those models. For example, a

system analyst noted “Sometimes the customer will already have whatever request to use

particular types of models. That might be because they are already using them for a

particular purpose or for historical reasons and things like that.”

Pragmatic Reason 3: Select models based on adoption levels

A third pragmatic factor that affects choosing types of models to be used is the overall

diffusion of the models in industry. We found that widely used models were preferred in

many system analysis and design tasks, for two main reasons: first because it is more likely

that stakeholders involved in the projects are familiar with more popular models. An analyst

commented: “[…] the other main driver is the fact that they're the most popular. So, they're

good enough and they're the most popular, that means […] you're more likely to find someone

who knows UML and BPMN than someone using something a bit more esoteric.” Second, for

documenting purposes, it is more likely that system designers use the most up-to-date

modelling grammars to document a system for the purpose of traceability. “But more recently

I think the industry has been dominated by UML […] I could use the data flow diagram, I

could use their equivalent of the state transition diagram […]. But, I tend not to use those

because what customers tend to want is some kind of more modernisation.”

Pragmatic Reason 4: Select models based on task goals

We also unearthed that the specific purposes of tasks affect the selection of different

types of models. Practitioners mentioned that not all types of models can be used for the same

purposes. One designer commented: “Yeah, I think there is, certainly, techniques that are

useful for particular purposes. I think that’s why those techniques continue to be long

standing.” Almost 80 percent of the practitioners we interviewed mentioned that they select

models that provide the best “fit” to the purpose of the task they are trying to complete. One

84

system engineer commented “Which diagrams you use is dependent on […] what the problem

is or the characteristics of the system. […] So, it depends really on what you’re trying to

understand or what you’re trying to specify.”

We also learned that practitioners select types of models based on the characteristics of

the domain and what level of understanding they are trying to glean from that domain. A

systems developer commented: “So, it might be that the problem requires more structural

modelling than behavioural modelling or more behavioural modelling than structural

modelling and it really depends on the nature of the problem. And so also it may be that

you’re dealing with a system that has no interesting states but does have interesting message

sequencing, or it could be the reverse.” Another practitioner stated: “If I am mostly interested

in the process and possible data flow in that process and the decisions during the process […]

then I would tend to want to use an activity diagram. On the other hand, if I was more

interested in my system being in a number of states, what I want to focus on is that it is very

driven and depending on receiving events it will transition between those states. Then, of

course, I would be more interested in using a state machine diagram.”

Pragmatic Reason 5: Select models based on tool support

Tool support was a fifth pragmatic factor we identified as a model selection rationale.

When different types of conceptual models were available, 40% of interviewees mentioned

that they preferred to use models that were supported by appropriate tools available to them.

A developer commented on how she selected models as: “It depends on what tool support

there is.” For example, another developer mentioned that “[I use state machines] because I

have a tool that translates my state machines into code.”

We also learned that the availability of tools can be in compliance with the

organisational standards. A developer noted: “So it is really critical to set standards not only

in what techniques you use, what tools you use and what conventions you set.” Another

practitioner commented: “Tools have to comply with the standards. […] Each team needs to

have the tool and the management that says okay…”

5.5 DISCUSSION

We explored how practitioners reportedly use conceptual models during systems design

and analysis, with a focus on why and how they use different types of models in combination.


Fairly immediately we found that indeed multiple types of models are used in today’s system

development and analysis, which confirms earlier reports (Davies et al., 2006; Dobing &

Parsons, 2008; Fettke, 2009). Our main objective then turned to exploring why IS

professionals use multiple models and how they select different types of models.

We first explored factors that lead IS professionals to use multiple models. Several of

the findings that we noted, in and by themselves, have been noted previously. For example,

our data analyses suggests that IS professionals use multiple models because 1) no one model

is a complete representation of a domain (e.g., Green et al., 2007; Irwin & Turk, 2005; Recker

et al., 2009; Siau, 2010); 2) complex and large domains are much broader than what can be

presented in a single model (e.g., J. Kim et al., 2000); and 3) different stakeholders are

typically involved in a project, with different levels of understanding and familiarity with

different models and posing different requirements towards a model (e.g., Darke & Shanks,

1997; Topi & Ramesh, 2002).

While many of these findings on their own are consistent and corroborative of earlier

studies, it is the interaction of these factors that is interesting and novel. Our view of the

model that describes the different rationales for using multiple models in combination during

systems analysis and design is visualised in Figure 5.4. The model suggests that the extent

and type of multiple model use is determined by two primary factors: the size and complexity

of the real-world domain, which the existing or intended representation is meant to support;

and the provided representation by any one conceptual model, in terms of the perspective

provided on the domain and the level of incompleteness this model has. The model further

proposes that both extent and type of multiple model use, given one existent model will be

moderated by the requirements of the involved stakeholders, in terms of the level of

understanding they seek from the models (e.g., surface versus deep) and the extent to which

they are familiar with the provided and any additional model representation.

86

Figure 5.4 The role of different factors that determine the use of multiple models

These two sets of rationales in our model provide two important extensions to the

literature:

1. The model emergent from our data is consistent with both Wand and Weber’s

(1990a, 1993) representation theory and their research framework on conceptual

modelling (Wand & Weber, 2002); and provides a synthesis of both: the model

suggests that the use of multiple models can be explained by both conceptual

modelling script and context factors, with both direct and mediated relationships

between them.

2. The model also extends the main premises of representation theory in itself: it

proposes that not only the incompleteness of single models is a key determinant

of the use of multiple models, but also that the supported perspective of the

models (e.g., structural vs behavioural) is a main direct determinant. Rosemann

Conceptual Modelling Context

Relevant stakeholders- required type of understanding

- familiarity with models

Relevant real-world domain- size

- complexity

Conceptual Modelling Script

Conceptual model- ontological incompleteness

- supported perspective

Use of multiple models- extent- type

necessitates

increases likelihood

influences choices


and Green (2002) suggested that such perspectives can be identified in the

ontological constructs underlying Wand and Weber’s theory (e.g., things and

their properties, states assumed by things, and so forth), which aligns with our

interview data.

Our second main objective was to explore how model users choose which types of

models to use. Figure 5.5 shows our view of the emergent model forthcoming from our

interpretation of the findings that describes the selection rationale.

Figure 5.5 The role of different factors that determine the selection of different types of models

Much like the reasons for using multiple models, our findings indicate that, broadly,

two sets of factors, about the semantics of the models as script combinations, as well as the

pragmatic contexts in which they are used, influence the selection. Again we note how the

script combination factors match concepts proposed in the different extensions of Wand and

Weber’s representation theory (Green, 1996; Recker, 2014; Weber, 1997); yet, again we also

discovered a new concept: that of correspondence. Correspondence implies that while

Pragmatic Reasons

Conceptual Modelling Context

Organizational regulations and standards- conventions

- construct specifications

Adoption- diffusion

- popularity

Semantic Reasons

Conceptual Modelling Script Combination

Overlap- redundant constructs- similar perspectives

Selection of models- extent- type

requires

overrides

varies Correspondence- model linkages

- visual cues

Completeness- additional representations

- varied perspectives

increasesdecreases

minimizes

Relevant stakeholders- required type of

understanding- cognitive demand

Task- objectives

- requirements

Tool Support- supported grammars- translation support

influences choices

constrains

88

practitioners avoid overlapping models, the existence of semantically related elements in pairs

of models influences the selection of these models.

Again, we also identified the strong prevalence of context, which has, much like

pragmatics, largely been ignored in most of the theoretical and empirical work (Bera et al.,

2014; Burton-Jones et al., 2009; Recker, 2007). We identified factors, such as organisational

conventions, tool support, and adoption levels of models, that clearly influenced the selection

of the types of models IS professionals use in their design and analysis tasks. In themselves,

these findings are not new; similar findings had been reported in Delfmann et al. (2015);

Recker et al. (2010); Rosemann (2006); and Samuel et al. (2015). For example, Recker et al.

(2010) identified tool support and organisational conventions that influence how analysts use

BPMN grammar. These factors were also prominent in our data, and in addition to these

factors, we exposed the factor of “adoption level” as also influencing the selection of types of

models. Furthermore, consistent with Samuel et al. (2015), we identified that objectives and

requirements of the task at hand, together with relevant stakeholder requirements, influence

the choice of different models to use.

Beyond the identification of these factors, our dominant novel finding is that the

relationships between attributes of the modelling context and the modelling script

(combination) are far from trivial: for example, organisational conventions may direct model

users to choose models that overlap. These organisational regulations may increase

ontological overlap, whereas in principle, they could be designed to mitigate it instead.

Together, the findings of our exploratory study and assumptions proposed by our two

theoretical models provide novel theoretical logic and demand further research to

operationalise and test our conjectures. Whereas our research was, deliberately, exploratory

and qualitative, in our view, the follow-up research should be falsificationist and quantitative.

5.5.1 Implications for ontological research on conceptual modelling

We identified three main implications for ontological research on conceptual modelling.

First, we have shown that ontological factors are among the main drivers of using multiple

models and selecting types of models. While this empirical knowledge is novel, it falls within

theoretical expectations. It corroborates the relevance of ontological qualities in the actual use

of conceptual models in system analysis and design tasks. One novel, empirically grounded


extension of our work is in the interaction between completeness and the overlap of models:

while, perhaps as expected, ontological incompleteness is often a starting point for the use of

multiple models, the interaction between completeness and overlap in the selection of models

is not always clear-cut: it would appear that in dependence of the context of use, overlap

between models (and the correspondence it provides) may sometimes bring forth desired

qualities for users.

Second, we also found that not only ontological qualities are important in using and

selecting multiple types of models; contextual and individual difference factors also play

important roles on why and how IS professionals use conceptual models. Importantly, again,

we identified not only main effects but also important interaction effects between context and

script factors. The main factors we found are not necessarily novel; however, the interaction

effects we discovered are novel and stimulating: sometimes, context influences the impact of

script qualities; sometimes it overrides them. This was most evident in the role of

organisational conventions and individual differences (Khatri et al., 2006; Recker et al.,

2010): depending on regulations and intended audiences, IS professionals sometimes choose

to entirely compromise or even reverse ontological qualities (both in terms of overlap and

completeness). For the ontological research streams on conceptual modelling, our study not

only suggests the adage that “context matters”, but also offers novel propositions about the

existence and type of interaction effects that are worth exploring.

A third implication concerns the ways in which our study can be extended: we focused

on two overlooked aspects that fall into the pre-usage stage of conceptual modelling: the

rationales for why and how users select multiple models. These rationales determine the

choices users make in light of the tasks they set out to complete. During these tasks, they will

be required to interpret the chosen models to glean real-world understanding from them and to

assist them in meeting their task goals. Therefore, an important extension to our work would

be field research that examines whether the identified factors and proposed effects also inform

the development of domain understanding from multiple models (Gemino & Wand, 2003)

and overall task performance of users and the extent to which they found the models useful

for their tasks (Maes & Poels, 2007).

90

5.5.2 Implications for empirical research on conceptual modelling

Our research adds to the empirical body of knowledge by exploring conceptual

modelling context and scripts factors as influential factors affecting the use of multiple types

of conceptual models. Previous studies have indicated that IS professionals normally use

multiple types of models; however, it is unknown how and why they are using different types

of models. We uncovered that there are important context factors that lead IS professionals to

use multiple models and also select what types of models they wanted to use. There is much

that can be done to continue this research. Some of the identified factors, such as individual

differences (Khatri et al., 2006), organisational conventions, tool support, and adoption levels

(Delfmann et al., 2015; Recker et al., 2010; Rosemann, 2006; Samuel et al., 2015), or task

goals (Figl & Recker, 2014), have previously been considered in studies of the use of single

models or grammars. Opportunities now exist to examine the role and interactions of these

factors on using different types of models. For example, future work could attempt to qualify

the relative importance of our identified factors in different contexts or usage stages. For

example, one replication of our study could explore different usage stages (e.g., model

interpretation, model application, or model creation stages) by using a similar research design.

A second way to replicate our study would be to explore differences in the study population,

for example, by interviewing both novice and expert model users (Batra & Davis, 1992;

Shanks, 1997). A third meaningful continuation would flow from the exploration of

particularly large and complex domains, to explore how systems analysis and design on the

basis of multiple models tries to mitigate challenges related to volume, emergence, and

complexity. Domains such as mission-critical systems (Pettit, 2008; Pettit & Street, 2005),

information systems used in geology or ecology (Goodchild, 2012), or in building information

management (Merschbrock & Munkvold, 2012) might be good settings for such

investigations.

5.5.3 Implications for the practice of conceptual modelling

Several practical implications flow from our work: one, our findings inform

components that would be important to be included in conceptual modelling conventions and

guidelines: What types of models should be used to meet task purposes, to satisfy different

stakeholder requirements, and that dictate desired levels of completeness and clarity of

representation?


Two, our study provides practical guidance for IS professionals for identifying

appropriate models for use. Specially, our findings suggest that in dealing with large and

complex domains, multiple types of models may improve domain understanding and inform

that those types of models should be selected that not only best fit the purpose of task, but

also meet different stakeholder requirements.

Three, another implication of this study is in regards to managerial decisions concerning

tool selections. Our findings uncovered that IS professionals prefer to use those models that

they are supported by tools available in their organisations. This suggests a managerial

direction for decision makers to invest in appropriate supporting tools to achieve the best

results from conceptual modelling.

Four, we found that organisational conventions may lead IS professionals to use

overlapping models, whilst, prima facie, users prefer to avoid using models that have high

overlap. These findings suggest that it is an important managerial decision to identify

appropriate types and contents of models to be included in such conventions. Decision makers

should (re-)design organisational conventions in such a way that not only provides system

designers and analysts with appropriate types of models for a given task, but also enables

them to provide a complete representation of the domain without overlapping information.

5.5.4 Limitations

We followed an inductive approach to determine how conceptual models are used in

today’s system analysis and design practice. This choice of research design brings forth

several limitations common to exploratory, qualitative, and inductive research (Shanks, 2002;

Yin, 2009), which we were aware of and mitigated where possible.

First, as is normal for interviews, it was time consuming to approach participants and

conduct the interviews. We continued the interviews until we reached appropriate saturation,

as explained; nevertheless, our absolute sample size of fifteen interviewees is arguably low.

Second, the quality of data from interviews is dependent on the capability of the

interviewer and performing reliable interviews. To manage this, we conducted five pilot tests

before the actual interviews, and in order to avoid bias in the questions, the interviewer

followed the designed protocol closely. Moreover, following the designed protocol helped us

to collect reliable data.

92

Third, our sample was drawn mostly from practitioners who were using UML in their

system design and analysis tasks. Our choice was motivated by the observation that UML

users were most likely exposed to a variety of grammars and models, whereas other types of

practitioners, for example, data or process models, may simply not have had exposure to

contemplate using multiple models. However, to increase the generalisability of our findings,

we also interviewed practitioners who were using other conceptual modelling techniques. We

also specifically recruited practitioners with different backgrounds, roles, and skills to

maximise the generalisability of our findings.

Fourth, we did not interview practitioners who did not use conceptual models in their

analysis and design tasks. Our findings are therefore susceptible to positive confirmation bias.

However, this was not by design: we recruited any practitioner involved in systems analysis

and design with some knowledge of UML or other models, independent of whether they used

it or not. It just so happened that all of our interviewees did in fact have experience in using

conceptual models in their work.

Finally, we did not consider using multiple conceptual models for a particular purpose

or in specific phases of system design and development process. As the purpose of tasks may

change during the life cycle of system development, and different stakeholders may be

involved in different phases of system development, practitioners may need to use different

types of models at different times. Moreover, there may be other or different factors

influencing this decision at different times. For example, it is conceivable that different phases

of the systems development life cycle may require models with different levels of

completeness and overlap. Future studies could expand our study by considering this

contingency.

5.6 CONCLUSION

Burton-Jones et al. (2009) highlighted the important role of the interaction between

semantics and pragmatics when understanding conceptual models. They noted a dearth of

research on this topic, as well as, more broadly, a need for “more theoretical work and more

exploratory studies of conceptual modelling in practice” (p. 514). In this paper, we have

attempted to provide both: an empirical account on the use of (multiple) conceptual models in

practice as reported by practitioners and the development of important extensions and


contingencies to existing theory in this field, mainly representation theory and the associated

ontological analysis.

We believe the value of our research lies particularly in its generativity and fertility: our

analyses and findings are generative because they lead us to ask several questions that

challenge the common assumptions about conceptual modelling practice and theory: for

example, one question our research asks is: Can we use existing theory about the

representational semantics of conceptual models to better formalise the perspectives these

models arguably (and perhaps intuitively) provide to practitioners?

Our results are also fertile because they provide clear impetus for theoretical

advancement and development: for instance, we demonstrated how constructs that describe

the modelling context (e.g., task factors, conventions or tools) interact with factors that

describe the semantics of a set of conceptual models (e.g., overlap and completeness). The

propositions from our emergent models can be used both to refine and extend the core

premises of representation theory and its extensions for multiple grammars and models

(Green et al., 2011; Recker, 2014) and to evaluate contingencies in the empirical domains in

which the theory predictions purportedly hold. These are important avenues to continue the

active and ongoing dialogue in our field.

94

Chapter 6: An Experimental Study to Test and

Extend the Theory of Faithful Use of

Conceptual Model Combinations11

ABSTRACT

Conceptual models are used to graphically capture the requirements of an information

system. Research on conceptual modelling has focused on improving the modelling process.

The vast majority of studies have focused on evaluation of single grammars or models.

However, practitioners typically use multiple, and different, types of models in system

analysis and design tasks. Recker (2014) proposed a theory of the faithful use of conceptual

model combinations. The theory develops propositions to explain the model selection, domain

understanding, and perceived usefulness of model combinations. On the basis of a laboratory

experiment with 131 students, this research presents the first dedicated empirical evaluation of

the propositions of Recker’s (2014) theory. Our results of the testing of the three propositions

lead to the conclusion that the level of combined ontological completeness and ontological

overlap impacts users’ selection of model combinations, as well as user’s understanding

across measures of comprehension and deep-level understanding. However, the impact on

users’ retention ability and perceived usefulness of model combinations was not significant.

Our research informs implications for research and practice by providing empirical evaluation

of the merits of the theory of faithful use of conceptual model combinations and guideline

development to better use of model combinations for different purposes.

Keywords: Combined Ontological Completeness, Ontological Overlap, Theory

Testing, Experiment Design, Ontology, Faithful Use

11 An Experimental Study to Test and Extend the Theory of Faithful Use of Conceptual Model Combinations, To

be submitted to MIS Quarterly

An initial version of the paper was published as: Jabbari Sabegh, M. A., Recker, J., & Green, P. (2016).

Designing Experiments to Test the Theory of Combined Ontological Coverage. Paper presented at the

International Conference on Information Systems, Dublin, Ireland

Chapter 6:An Experimental Study to Test and Extend the Theory of Faithful Use of Conceptual Model Combinations 95

6.1 INTRODUCTION

A key activity in systems analysis and design is to develop and use representations,

often graphical, of relevant features of the domain under examination (Burton-Jones & Meso,

2006). These representations are called conceptual models. Conceptual models are developed

using modelling grammars (Wand & Weber, 2002), that is, sets of constructs and rules that

show how to combine the constructs to model real-world domains. Studies about conceptual

modelling abound (Gemino & Wand, 2004), and they have investigated a wide variety of

aspects. However, they have almost exclusively focused on single models or single grammars,

for example, how the use of a model can be improved by additional notation elements such as

colours (Masri et al., 2008), or how much domain understanding can be created from reading

a model (Bera et al., 2014). By contrast, research on the use of multiple models is very sparse.

Results of empirical survey research, on the other hand, show that IS professionals typically

do not use just one conceptual modelling grammar, let alone one conceptual model in their

analysis and design tasks (Dobing & Parsons, 2008; Fettke, 2009). In fact, multiple types of

models, often developed using different grammars, are usually used in combination. For

example, UML provides grammars to design many different types of models, such as class

diagrams and sequence diagrams; and practitioner surveys show that many of these model

types are used together (Dobing & Parsons, 2008).

Recker (2014) proposed a theory of faithful use of conceptual model combinations to

predict and explain the use of multiple models in combination. This theory provides

propositions to explain and predict the selection of models, generation of domain

understanding from multiple models, and perceived usefulness of models in combination.

This research is the first dedicated empirical study with the aim of evaluating the propositions

of the theory of faithful use of conceptual model combinations.

The objectives of this paper are therefore: (1) to investigate how users select a model

combination from a set of available models to achieve given task goals, (2) to examine how

much domain understanding can be generated from a model combination based on different

levels of completeness and clarity, and (3) what the roles of combined ontological

completeness and ontological overlap are on perceived usefulness of a model combination. To

that end, we conducted an experimental study with 131 information systems students to test

and extend the theory of faithful use of conceptual model combinations.

96

The rest of this paper is structured as follows. In Section 6.2, we review relevant

literature about the use of conceptual models by system analysts and designers. In Section 6.3,

we briefly recall the propositions of the theory of faithful use of conceptual model

combinations. We then describe the research method, the design of the experiment, materials,

and how we operationalised the levels of combined ontological completeness and ontological

overlap. Section 6.5 reports the hypothesis development. Next, in Section 6.6, we explain the

results of data analysis. In Section 6.7, the discussions of the findings and the key

contributions of these findings are presented. The chapter ends with the conclusion of this

study.

6.2 USE OF MULTIPLE CONCEPTUAL MODELS

Conceptual models are used for a wide variety of purposes, such as communication,

improving domain understanding, designing and improving processes, documenting or

analysing requirements, and designing and developing software or databases (Davies et al.,

2006; Siau, 2004; Wand & Weber, 2002). Much research has been carried out to understand

the capabilities of conceptual models to achieve these objectives. Reviews of this literature

have been provided by Wand and Weber (2002), Burton-Jones et al. (2009), Recker et al.

(2009), and Green et al. (2011), amongst others.

Across this literature, we noted that, with the exception of a limited number of studies

on using multiple grammars (e.g., Green et al., 2007; Green et al., 2011) or models (e.g.,

Gemino & Parker, 2009; J. Kim et al., 2000; Siau & Lee, 2004), much of the research on

conceptual models focuses on the use of a single grammar or a single model (Bera &

Evermann, 2014; Bodart et al., 2001; Bowen et al., 2006; Gemino & Wand, 2005; Parsons,

2011; Recker, 2013). The dominant findings of research on conceptual modelling grammars is

that any one grammar, and as a result, any single model, suffers from some level of construct

deficit (Irwin & Turk, 2005; Recker et al., 2010; Recker et al., 2009; Hong Zhang et al.,

2007).

Weber (1997) argued that if analysts or designers encounter ontological incompleteness,

they will somehow have to augment their model(s). One plausible strategy is to use additional

grammars or models. For instance, Green et al. (2011) demonstrated how grammars with

different representational capabilities can complement each other. The findings of these


studies show that modellers use multiple grammars to overcome construct deficits in any one

individual grammar. Studies on multiple models likewise indicate that using additional

models results in better performance than using a single model (Gemino & Parker, 2009; J.

Kim et al., 2000; Siau & Lee, 2004).

Practitioner surveys indicate that indeed, more than one conceptual model is often used

in most of the systems design and analysis tasks (Davies et al., 2006; Dobing & Parsons,

2008; Fettke, 2009). For example, Recker (2012) reported that over 30% of surveyed process

modellers access additional grammars when modelling business processes. Green et al. (2011)

showed that 80% of users of modelling tool environments select and use multiple grammars

in combination when engaging in conceptual modelling. Dobing and Parsons (2008) reported

that 90% of UML users employ at least two different UML grammars in at least one-third of

their projects.

Overall, these arguments lead us to conclude that: first, system analysts and designers

do use different types of grammars to create different types of models. Second, using multiple

grammars or multiple models may lead to better modelling outcomes. Both conclusions are

intuitive: as targeted systems become more complex, we need more than a single model to

represent them (J. Kim et al., 2000). Moreover, different types of models seem to represent

different aspects of a problem domain, for instance, use case diagrams and class diagrams

(Siau & Lee, 2004). Thus, multiple model use is apparently quite common in practice;

however, we still do not have an understanding regarding how and why that is the case.

6.3 THE THEORY OF FAITHFUL USE OF CONCEPTUAL MODEL

COMBINATIONS

Recker’s (2014) theory of faithful use of conceptual model combinations set out to

provide an explanation for the faithful use of multiple models by analysts and designers. The

primary conjecture of the theory is that faithful use of a combination of conceptual models

will be dependent on two factors: the models’ combined ontological completeness (level of

representational coverage a set of models provides about some real-world phenomenon) and

ontological overlap (level of redundant representations across a set of models) in representing

the focal real-world phenomena. Figure 6.1 provides a visual representation of these two key

factors.

98

Figure 6.1 Illustration of Ontological Completeness and Overlap of Two Conceptual Models (Recker, 2014)

The basic rationale behind this conjecture is that: first, users will select multiple models

to maximise their coverage of some relevant real-world phenomena (Green, 1996). This is

because ontological evaluations of modelling grammars show that any single modelling

grammar suffers from construct deficit (Green et al., 2011; Irwin & Turk, 2005; Siau, 2010).

In other words, no one grammar can be used to model all potentially relevant real-world

phenomena. Moreover, the actual level of completeness of a conceptual model is often less

than its potential level of completeness, because often not all constructs provided via a

grammar are used in any given model (Siau et al., 2005; zur Muehlen & Recker, 2008).

Therefore, users will use multiple models in combination to overcome ontological deficits in a

single model and to maximise combined ontological completeness through multiple models.

Second, users will also seek to minimise overlap in the representation to keep the coverage as

simple and clear as possible (Wand & Weber, 1990a, 1990b, 1993). On the basis of these two

main factors, the theory suggests three propositions that detail the faithful use of multiple

models:

Model selection: model users will select a combination of conceptual models with the

aim to reach maximal ontological completeness; however, they will select additional

models only until a particular level of overlap is reached that they perceive to be

bearable.

Completeness of


of model A

Completeness of


of model B

Resulting

ontological overlap

between models A

and B

Remaining construct

deficit of combined

representation


Domain understanding: the level of understanding users can generate from reading

models will be increased when additional models present more elements of a domain;

however, the level of understanding that can be generated will be moderated by the

level of ontological overlap between models.

Perceived usefulness: Perceptions of the usefulness of model combinations will

increase when the ontological completeness of the achieved representation increases;

however, combinations of models with high ontological overlap between the models

will be evaluated as less useful.

The theory addresses an important and ecologically valid aspect of the use of conceptual

models in practice. The propositions of the theory provide foundations to examine which

combinations of models are faithful for use. Following the guideline proposed by the theory,

in the next section, we describe how we designed and conducted an empirical study to test the

propositions of the theory.

6.4 RESEARCH METHOD

6.4.1 Preliminaries

An experiment is a suitable research method to evaluate propositions of the theory of

faithful use of conceptual model combinations. This is because the theory has not been tested;

therefore, a primary objective is to maximise internal validity (Calder et al., 1981). The theory

suggests three sets of propositions about selection, domain understanding, and usefulness.

Selection is a preferential task, whilst generation of domain understanding is an experiential

task and usefulness perceptions are formed reflectively post hoc. The differences in the nature

of each set of the propositions raise issues with respect to designing an experimental study to

test the propositions effectively. Therefore, a two-part experiment is required: an experimental

design to examine model selection following another experimental design to investigate the

development of domain understanding through using model combinations. A post hoc test

following the second part can then evaluate perceptions of usefulness.

6.4.2 Experimental Design

We designed a two-phase experiment: a free simulation experiment and a 2x2 between-

groups experiment followed by a post hoc test. Each phase is discussed below:

100

In the first phase of the experiment, participants were asked to select conceptual models

for an upcoming model use task (which required developing a deep-level understanding of the

modelled domain). Therefore, in this phase, the experimental design corresponds to a type of

free simulation experiment. In this type of design, no experimental treatment is provided.

Participants were given four types of conceptual models for a domain, and they were asked to

select a combination of models out of a given set of models that would assist them in an

upcoming problem solving task. Participants were free to behave within the required

boundaries of the study (i.e., the prescribed tasks) and they were asked to make decisions and

choices as they saw fit (Figl & Recker, 2014). To measure the selection, we captured: (1) the

types of models participants selected, and (2) the number of models participants added to their

model combinations prior to engaging in the task. We employed two independent variables:

the level of combined ontological completeness the additional model added to the

combination, and the level of ontological overlap the additional model added to the

combination. The dependent variable in this phase was a binary variable that indicated

whether the difference between the level of combined ontological completeness and the level

of ontological overlap of the selected model combination was more than any other possible

combinations.12

In the second phase, we examined domain understanding and perceived usefulness of

model combinations. We implemented a 2x2 between-groups experimental design with two

between-group factors, combined ontological completeness and ontological overlap, each of

which had two levels: high and low. The purpose of having two levels of combined

ontological completeness and overlap is to determine whether different levels of combined

ontological completeness and overlap affect understandability and perceived usefulness of

model combinations in different ways. The two factors implemented by designing four

combinations of two conceptual models that vary in their level of combined ontological

completeness and ontological overlap, similar to the example given by (Recker, 2014). For

instance, in our study, one such combination could be a class diagram with a state machine

diagram (high combined ontological completeness and low ontological overlap). An example

for low combined ontological completeness and high ontological overlap would be a class

12 Note that the independent variables and the dependent variable will be based on the selected second model and

third model.


diagram and a use case diagram. To measure the dependent variable, generation of domain

understanding, we employed three measures often used in conceptual modelling experiments,

viz., comprehension test, problem-solving test, and Cloze test.

6.4.3 Materials

To be able to develop and test hypotheses derived from the propositions of the theory,

we needed to apply the propositions to a set of conceptual models of some relevant real-world

phenomena. However, there is a possibility of a learning effect during a two-part experiment.

To avoid the potential learning effect, we developed two sets of models, one set to investigate

model selection and one set to evaluate domain understanding. In developing the conceptual

models, we chose a textbook case simply because the textbook featured a wide selection of

different models for this scenario.

The sets of models we selected are described below, and we then explain how we

manipulated the experimental treatments of combined ontological completeness and

ontological overlap using these models.

Deciding on Case and Models

We selected two different cases: the High Peak Bicycles case, (i.e., a bicycle rental

business that included concepts such as customer, rent, and return) and the Library

Management System case (i.e., a library in a university that had concepts such as students,

borrowing, returning, books, and articles). We adopted both cases from Whiteley (2013, pp.

228-263). We selected these cases for two reasons: first, we could use the instructions

explained in the textbook to develop different types of models to represent different aspects of

the domains; second, these are well-established domains that have also been used in related

prior studies (e.g., Bera et al., 2014) and in IS education.

For each domain we decided to develop four conceptual models, each developed with a

different UML grammar. Figure 6.2 presents these models for the Library Management

System, the models for the High Peak Bicycles case are available in Appendix C. We chose

UML diagrams for several reasons. First, UML is a widely used conceptual modelling method

with widespread adoption in practice and with extensive coverage in IS education (Davies et

al., 2006; Fettke, 2009). Second, UML provides a wide variety of models, each of which

conveys different information (Dobing & Parsons, 2008). Third, a substantial body of

102

research has analysed the relevant grammatical constructs of UML based on Wand and

Weber’s (1993) ontological theory (e.g., Evermann & Wand, 2005, 2006; Opdahl &

Henderson-Sellers, 2002; Siau, 2010). In turn, it was likely that the models would be different

in their coverage of real-world phenomena, and we could draw on established ontological

analyses to evaluate the levels of combined ontological completeness and overlap.

Post-Graduate Student

Borrow Books

Return Books

Pay Fines

Request Research JournalsLibrarian

Maintain Records

Manage Collection

Source New Books

Shelve Books

Calculate Fines

Under-Graduate Student

User

User

-Student ID*

-userName

-userInfo

Undergraduate

student

-Student ID

-Name

-DoB

-Major

-Faculty

-Address

Postgraduate

student

-Student ID

-Name

-DoB

-Major

-Faculty

-Address

-Thesis Topic

-Thesis supervisor

Transaction

-transactionID

Reservation

-dateReserve

-dateAvailable

-studentID*

Copy

-bookID*

-copyNumber

-copyAvailability

1..1

1..*

Book

-ISBN

-bookTitle

-bookAuthor

-Year

-Edition

-Topic

-Category

-Keywords

-Description

0..*

1..1

-bookID

-bookReservation

Loan

-copyNumber*

-dateOut

-dateBack

0..*

1..1

1..1

1..*

Author

-biography

-birthdate

-name

Research Journals

-journalTitle

-publisher

-volume

-issue

0..1

0..*

-author 1..*

1..*

1..*

1..*

Collection

-Name of collection

-relevant student major(s)

-number of books in

collection

0..*

1..*

Bill

-billNumber

-StudentID*

-billDate

-fineAmount0..4

1..1

+createBill()

+updateBill()

+createUser()

+checkUser()

+cancelUser()

+payFine()

+returnBook() +createReservation()

+getReservation()

+checkAvailableDate()

+createLoan()

+getLoan()

+completeDateBack()

+createJournal()

+createAuthor()

+createCopy()

+addBook()

+searchBook()

+displayBookDetails

+updateBookDetails()

+createCollection()

+createTransaction()

+getTransaction()

0..1

0..1

+updateAvailability()

-copyNumber*

+checkUser()

Library Management System

Transaction User

Request an ItemCheck the Request

Calculate Fine

Check Availability

Validate User

[Ret

urn

Book

]

[Request Book]

[Req

uest

Jour

nal A

rtic

le]

[Not

Ava

ilabl

e]

Inform User

[Available or Hired]

[Ava

ilabl

e]

Request Loan

[On

loan

]

Request Reservation

Validate User

Validate User

Make Reservation

Issue Book

Change Book Status

[Not

Val

id o

r Exc

eed

loan

or b

ill]

[Not

Val

id o

r Exc

eed

loan

or b

ill]

Submit Request

[Sto

p th

e or

der]

[Continue to Order]

Authorise Journal Access

[Pos

t-gr

ade

Stud

ent]

[Not Valid]Shelve Book

[Fin

e ==

0]

[Fin

e >

0]

Shelve Book Issue Bill

Change User Status

Check User Status

Check User Status

[Valid & No exceed loan and bill]

[Valid & No exceed loan and bill]

b: Bill

[unpaid]

Save as Debt

Pay the Bill

Change Book Status

b: Bill

[paid]

Change User Status

Change Book Status

Idle Request Submitted Item Searched

User Authorised

User Not Authorised

Book Status Updated

Journal AccessedFine Calculated

Bill Issued

su

bm

it re

qu

est

ch

eck

re

qu

est

[Req=Search]/search item

[Found]/check user validity

[Not Found]

[User valid]

[User not Valid]

[Req==Article]/access article

[Req==Return]/calculate fine

[Fine>0]/issue bill

[Fine==0]/update book status

update book status

[Req==Reserve]/update book status

[Req==Loan]/update book status

Figure 6.2 Four Conceptual Models for the Library Management System Case (Whiteley, 2013, pp. 228-263)

We developed a use case diagram, a class diagram, an activity diagram, and a state

machine diagram for each case based on three reasons:


1. We chose a use case diagram, class diagram, activity diagram, and state machine

diagram because together they specify a systems’ function, structure, and behaviour

(Booch, Rumbaugh, & Jacobson, 2010; Burton-Jones & Meso, 2006).

2. The results of our semi-structured interviews indicate that these four diagrams are

the most widely used UML diagrams IS professionals use in practice (Jabbari

Sabegh & Recker, 2017).

3. We selected these diagrams based on their level of combined ontological

completeness and ontological overlap. For example, there wouldn’t be a meaningful

difference between the levels of combined ontological completeness of a

combination of a use case diagram with a sequence diagram compared to the

combination of a sequence diagram with a class diagram. That is, we could not

explain the effect of our treatment if there would be a significant difference

between the performance of the users of combinations of models with the same

level of combined ontological completeness.

6.4.4 Establishing Levels of Combined Ontological Completeness and Ontological

Overlap

To manipulate our treatment variables, we first analysed of the representations offered

by any one model in a given combination of models. First, we performed an interpretation

mapping. That is, we mapped the constructs represented in each model to ontological

constructs that describe things in the real-world (Bunge, 1977; Wand & Weber, 1993). We

then conducted overlap analysis to establish the levels of combined ontological completeness

and ontological overlap for any possible combinations of models.

Interpretation mapping

In carrying out interpretation mappings, we followed established procedures for the

ontological analysis of models, as demonstrated by Recker (2014) and detailed by others (e.g.,

Rosemann et al., 2004; Rosemann et al., 2009). We also followed the multi-coder mapping

procedure, as described in Recker et al. (2010), to establish the reliability of this mapping. We

followed three steps:

104

1. All three researchers individually read the constructs represented in each model

and then interpreted and mapped the ontological constructs to constructs

represented in the models (Recker et al., 2010). In conducting the interpretation

mapping we relied upon previous literature (Wand & Weber, 1993, p. 221).

UML grammars have been evaluated by researchers (e.g., Evermann & Wand,

2001; Evermann & Wand, 2005, 2006; Irwin & Turk, 2005; Opdahl &

Henderson-Sellers, 2002). For each model, we identified grammar constructs

represented in the model. We then evaluated the meaning ascribed to each

construct and the mapping rationale to map it to the corresponding ontological

construct. We did so because not all constructs provided via a grammar are

presented in the models13

(Siau et al., 2005; zur Muehlen & Recker, 2008). For

example, the constructs “system” or “extend” are not represented in any of the

use case diagrams for High Peak Bicycles and Library Management System.

2. The researchers then met to discuss the interpretations of the representation

mapping.

3. Finally, the second revised version of the interpretation mappings was discussed

and refined in several meetings with all of the researchers until a 100%

agreement on mappings was reached.

The results of these processes are represented in Table 6.1 and details of the ontological

evaluation of the conceptual models are presented in Appendix C.

Overlap Analysis

After we implemented interpretation mapping for each of the eight models developed

for both cases, we performed an overlap analysis (Green et al., 2007). To evaluate the level of

combined ontological completeness of any possible combination of models, we counted those

constructs of the models in a combination that corresponded to different ontological

constructs.

13 Moreover, interpretation mappings are not always a 1:1 correspondence of ontological to grammatical

constructs. The details of the mappings are presented in Appendix C.


Similarly, we measured the level of ontological overlap of any possible combination of

models by counting those constructs of models in a combination that could be mapped to the

same ontological construct. Table 6.2 shows the results of overlap analysis for the pairwise

combinations of models for cases. Table 6.3 shows the level of combined ontological

completeness and ontological overlap of possible combinations of three models and the

combination of all four models for the High Peak Bicycles case.

Table 6.1 Ontological evaluation of conceptual model constructs - Library Management

System and High Peak Bicycles

Ontological construct Class Use Case Activity State Machine

Thing

Class *(*) *(*) *(*)

Composition Thing *

Property in General *(*)

Hereditary Property *

Binding Mutual property *(*) *(*)

State * *(*)

State Law *(*) *

Stable State *(*)

History

Event *(*)

External Event

Transformation *(*) *(*) *(*) *(*)

Lawful Transformation *(*)

Stability Condition *(*) *(*)

Coupling *(*)

Acts on

System *(*)

Sum out of 17 8 (6) 3 (3) 8 (6) 4 (4)

The corresponding mappings of models represented in the High Peak Bicycles case are presented in brackets.

Table 6.2 Overlap Analysis

High Peak Bicycle Case

Diagram Type Use Case

Diagram Class Diagram Activity Diagram

State Machine

Diagram

Use Case Diagram 3 2 1

Class Diagram 6 2 1

Activity Diagram 7 10 2

State Machine Diagram 6 9 8



Diagram Class Diagram Activity Diagram

State Machine

Diagram


Class Diagram 8 3 1


State Machine Diagram 6 11 9 Combined ontological completeness of the combinations is given in the dark grey cells below the diagonal; ontological overlap is given in the light grey cells above

106

Table 6.3 MOC and MOO for combinations of three and four models - High-Peak Bicycles

Case

Use Case

Diagram

Use Case

Diagram

Use Case

Diagram Class Diagram

Use Case

Diagram

Class Diagram Class Diagram Activity Diagram Activity Diagram Class Diagram

Activity Diagram State Machine

Diagram

State Machine

Diagram

State Machine

Diagram Activity Diagram

State Machine

Diagram

MOC 10 9 9 12 12

MOO 3 3 3 3 4

6.4.5 Measures

It is difficult to test domain understanding directly. For this reason, we used three

measures – models comprehension test, problem-solving questions, and a Cloze test – to

assess domain understanding. The comprehension test was used to determine a users’ ability

to glean information from the models. The performance on the comprehension test was

measured by the number of correct answers given to eight comprehension questions. The

problem-solving test was used to examine using knowledge gained from reading the

combination of models to solve a problem not directly answerable from models. A typical

problem-solving question defines a problem in the domain, for example, an element that is not

working (Burton-Jones & Meso, 2008). We devised three problem-solving tasks. Participants

were asked to: 1) describe the main functionality of the system, 2) identify a problem in the

system and describe how it occurred, and 3) to provide a solution for a problem based on their

understanding. The performance on the problem-solving test was measured by the total

number of correct answers given to the three problem-solving questions (Gemino & Wand,

2003, 2005). After the problem-solving test, the models were removed. In the Cloze test,

participants were given a passage about the domain to read; however, some of the words in

the passage were missing. Thus, the Cloze test was about participants’ ability to recall

information gained from the models to fill in the blanks of a textual description of the domain

(Masri et al., 2008). The performance on the Cloze test was measured by the number of

blanks filled in with correct answers in the Cloze test (Burton-Jones & Meso, 2006; Recker &

Dreiling, 2011).

After completing the second phase of the experiment, a post hoc test was designed to

measure the perceived usefulness of model combinations to complete the assigned tasks. To


measure participants perceived usefulness of model combinations we adopted measurement

items from Recker and Rosemann (2010b). The measurement items for the dependent

variables are presented in Table 6.4.

Table 6.4 Measurement instrument for dependent variables

Measurement Items for the Dependent Variable - Model Selection

MOC and MOO

= 1, if (combined ontological completeness - ontological overlap) is maximum compared to

any other possible model combinations

= 0, otherwise

Measurement Items for the Dependent Variable - Comprehension

1. Are there any limitations for a student to extend a loan?

2. Can a post-graduate student borrow more than six research journals at the same time?

3. Can a user of a library be an employee of the university? 4. Do students need to pay fines to access research journals?

5. Can a student request books that are loaned by someone else?

6. Are there a limited number of copies of each book in the library?

7. Can an undergraduate student borrow a book and a research journal at the same time?

8. Do students need to pay fines for a delay in returning books?

Measurement Items for the Dependent Variable- Problem Solving

1. Explain the main functionality of the library management system. Write down as many supported activities

as you can.

2. Under what circumstances would a student fail to borrow an item s/he needs? Write down as many possible

reasons as you can think of.

3. How can the library management system be improved to ensure that students in each major can have access

to the relevant books and journals they need? Write down as many possible solutions that should be

considered.

Measurement of the Performance on Cloze Test

The library management system is designed to track books and journal articles in the library of a university. It

also provides functions to …(1)… and control orders and loans, and calculate fines for …(2)… . Users of library

management system are either …(3)… or …(4)… .

The library has books in a wide variety of areas and topics. Normally, there are multiple …(5)… of a particular

book in the library. The library also provides access to research journals.

…(6)… do not have access to research journals . However, books in the library are categorised in specific

…(7)… to be sure that the library provides relevant books for students in different majors.

Users can search items using title, authors name, topic and any keywords related to their majors. In fact, …(8)…

can search items in the library, but only …(9)… can borrow books or have access to journal articles.

The results of a search for a book can be threefold: … (10) …, …(11)…, or …(12)…. If a book is on loan by

someone else, a user can still …(13)… the book. There are no limitations on how many …(14)… a …(15)…

student can have access at the same time.

Measurement Items of Perceived Usefulness

On a 1-7 scale (“Strongly disagree” to “Strongly agree”)

1. Overall, I found the combination of diagrams useful for the given tasks.

2. I found the combination of diagrams useful for achieving the purpose of the given tasks.

3. The combination of diagrams helped me in meeting the objectives of the given tasks.

Responses to the problem-solving questions and the cloze test were coded14

by two

independent graduate students. We prepared an extensive coding scheme that described the

coding procedure and provided examples of correct answers for the questions. The first author

14 For details on coding of the participants’ responses to problem solving questions and the Cloze test, please see

Appendix C.

108

explained the coding procedure to both coders. Before starting to code the responses, the

coders coded a sample of responses for both problem solving-questions and the cloze test.

Upon completion of the sample coding, the author and the coders discussed the rationale and

coding to ensure that the coders fully understood the nuances of the responses and how they

were to be coded (Khatri et al., 2006). The coders then independently coded the participants’

responses. After the coding procedure was completed, both coders and the first author met

and compared and discussed their coding to try to arrive at a consensus regarding the coding;

they then changed their coding accordingly. This process was repeated until an acceptable

reliability score was achieved. The inter-coder reliability was assessed using both raw

agreement and the kappa statistic (Cohen, 1988). After multiple rounds of coding, the raw

agreement achieved for the problem solving was 0.83, and the kappa statistic was 0.81 (i.e.,

81% agreement between coders). The cloze test was much more objective. The raw agreement

for the Cloze test was 0.92 while the kappa statistic was 0.91 (i.e., 91% agreement between

coders). The kappa value of more than 0.81 is almost perfect. However, the coders and the

first author met and discussed and refined their coding until a 100% agreement was obtained

between the two coders.

6.4.6 Covariates

To eradicate potential exogenous factors and evaluate rivalling hypotheses throughout

both phases of the experiment, we also measured several control variables, namely individual

difference factors, and cognitive load (Figl, Mendling, et al., 2013; Khatri et al., 2006; Masri

et al., 2008). We operationalised individual differences by examining participants’ prior

experience in conceptual modelling and familiarity with the conceptual models used in the

experiments. The rationale was to ensure that participants had experience of using relevant

conceptual models and to evaluate their levels of experience (Figl & Recker, 2014).

Familiarity with models was also used as a potential exogenous variable, which could affect

the usage of a model (Figl & Recker, 2014; Recker, 2010a) because participants with high

familiarity may have found using a combination of conceptual models easier than participants

with low familiarity, because of inherent preferences. To that end, we adopted the model

familiarity test from Figl and Recker (2014). Likewise, we measured perceived experienced

cognitive load during the tasks (Marcus, Cooper, & Sweller, 1996). We did so because a

complex combination of models (Figl, Recker, et al., 2013) and high ontological overlap


(Moody, 2009) is likely to increase cognitive load and may adversely affect the development

of understanding. We measured experienced cognitive load using a subjective rating scale

(Figl, Mendling, et al., 2013) after the second part of the experiment. The measurement

instruments developed to measure concepts of relevance to our study are available in

Appendix C.

6.4.7 Participants

The target population for the experiments comprised users of conceptual models. The

participants of this study were 131 undergraduate students in a large university in Australia

who completed courses related to systems analysis and design and conceptual modelling of

real-world domains. Students were regarded as reasonable proxies of junior working experts

as a sample for this study, first, because prior studies have found little difference between

students and junior professionals (Arisholm & Sjøberg, 2004). Second, this research was

relevant to the students who passed UML related courses because they had covered the

requirements of analysis, communication, system design, and modelling concepts. Finally,

using students is relatively common in conceptual modelling research (e.g., Bera et al., 2014).

Participation was voluntary, and remuneration and incentives were provided for

participants in the study. The experiment was conducted using an online experimentation

system in computer labs between September and November 2016 and was monitored to

ensure that individuals completed the experiment independently.

6.4.8 Procedures

Participants started with pre-tests that collected users’ demographic information and

UML familiarity levels. A short tutorial then introduced the basics of a use case diagram,

class diagram, state machine diagram, and activity diagram. No time limitation was placed on

the tutorial and performing the tasks. Figure 6.3 presents an overview of the experimental

procedures.

110

Pre-test

Modeling Familiarity

UML Diagrams Familiarity

Domain Familiarity

Tutorials

Use Case Diagram, Class

Diagram, State Machine

Diagram, Activity Diagram

Model Selection

High Peak Bicycles

Comprehension


Problem Solving


Cloze


Post hoc test

Perceived Usefulness


Remove models

Fir

st

ph

ase

Se

con

d p

hase

Figure 6.3 Overview of experimental procedure

First Phase

The first part of the experiment began with the free simulation experiment designed for

model selection. Participants were then instructed about the upcoming task and asked to make

selection decisions for this task, similar to the study by (Figl & Recker, 2014). The given task

was twofold: a) to identify reasons for a problem in the domain, and b) to suggest solutions

for a problem in the domain. We believe that if a user is asked a problem-solving question

that requires a deep understanding, he or she is more likely to examine different models

(Burton-Jones & Meso, 2008). Participants were provided with a use case diagram, class

diagram, state machine diagram, and an activity diagram presenting the High Peak Bicycle

case. Participants knew the types of models, but they could only see the models once selected.

This ensured that participants only used the models that they selected, and allowed us to

control for potential learning effects from unselected models. Through multiple pilot tests, we

learned that if participants were presented with all of the diagrams, they would still glean

information from the models they did not add to their selection.

Participants were free to start with any model to answer the problem-solving questions.

Participants could continue to add models to their selections if they felt they needed more

models to be able to answer the questions. However, the selection of models was stepwise. In

each step, participants were asked to either complete the task if they felt that the selected


model/s provided enough information to accomplish the task or to select additional models if

they needed to do so. In the latter case, the problem-solving questions and the remaining

model types were displayed, and participants were asked to choose one additional model.

These steps were repeated if participants continued to prefer to select a third or fourth model.

After the fourth model, participants were asked whether they felt they could complete the task

using all four models or if they were not sure they could answer the questions.

Participants were asked to select one model in each step and continue to add models in

the following steps. This stepwise procedure was implemented to observe the users’ selection

procedure of models. It also enabled us to examine whether participants compromised the

level of combined ontological completeness. Participants might not aim to have a

combination of models with a higher level of combined ontological completeness when the

addition of more models increases ontological overlap.

At the end of this phase, participants were asked to rate the extent to which they relied

on the information in each model they selected, their own knowledge, and the extent to which

they made assumptions in answering the two problem-solving questions. Participants were

also asked to identify which model/s they would like to remove if they had an opportunity to

remove a model, or which models they felt were not useful in answering the questions. These

measures allowed us to gather data for control checks and to measure cognitive engagement

with the models.

Second Phase

For the second phase of the experiment, participants were randomly assigned to four

groups. Table 6.5 presents the distribution of the participants across the treatment groups.

Each group had between 31 and 34 participants.

Table 6.5 Distribution of participants in each group

Combined Ontological Completeness

Low High

Ontological Overlap

Low

Use Case +

State Machine

(N=34)

Class +

State Machine

(N=33)

High

Class +

Use Case

(N=31)

Class +

Activity

(N=33)

112

Each group was given a combination of two models with different levels of combined

ontological completeness and ontological overlap (high or low). Each participant completed

three tasks (models comprehension task, problem-solving task, and Cloze test). After the

problem-solving test, the models were removed. The sequence of tasks was fixed to ensure

the internal validity of the experiment (Gemino & Wand, 2005). For all tasks, participants in

each group were given the same combination of two models representing the library

management system. Similar to the first phase, in the second phase, we used four UML

models (use case diagram, class diagram, activity diagram, and state machine diagram)

describing the Library Management System case. The models were described in Section 6.4.1.

Post Hoc Test

After the Cloze test, participants were asked to answer the post hoc test about the

perceived usefulness of the model combinations. Participants finished the experiment by

answering the questions about cognitive load.

6.5 HYPOTHESIS DEVELOPMENT

We developed our hypotheses in three parts following the model selection, domain

understanding, and perceived usefulness propositions. We begin with the selection

proposition.

6.5.1 Selection of Model Combination

The use of conceptual models occurs as part of a particular task (Recker, 2014) and a

decision on which model or model combination to use is characterised by task goals. A

selection decision about which model to use is influenced by performance outcome

expectations (Compeau, Higgins, & Huff, 1999). On the assumption that several models about

a real-world domain are available, the selection proposition concerns which models would be

selected by users to maximise the level of domain understanding they can generate to

complete an upcoming task (Recker, 2014). Recker’s (2014) theory suggests model users will

select a set of models from a number of available models to increase the ontological

completeness of representation of some focal real-world phenomena because any one model

will have construct deficits. Users will do so because they have a desire to have a more

complete representation that provides all relevant information about a real-world domain that

users will need.


Users make decisions about which model to use based on an evaluation of the potential

value of alternatives (Thompson, 1967), which leads them to achieve upcoming task goals.

The outcome expectations of using conceptual models is to generate a complete and clear

understanding of real-world phenomena (Weber, 1997). Users create a mental model of real-

world phenomena based on the information directly presented in the models (Gemino &

Wand, 2003). The outcome expectation of using models is to create understanding of the

domain that requires users to develop a more complete mental model (Masri et al., 2008) and

a more complete mental model demands more information. This reasoning suggests that,

given one starting model out of the four presented diagram types, users will select an

additional model in which the selected model provides users with more information compared

to the alternatives. Therefore, given a set of models, users will select additional models for

task completion such that the ontological completeness of the model combination is

maximised. For our experimental materials, we propose that users would select an additional

model for use according to the following rules:

H1: Given any model participants start with, they will select an additional model/s

that add/s maximum ontological completeness to the model combination.

By increasing the level of ontological completeness of the representation of a domain,

more information will be available for users to integrate a mental model of more aspects of

the represented real-world phenomenon. On the other hand, through selecting additional

models, the ontological overlap between models may also increase (Green et al., 2007).

Increasing ontological overlap between models will decrease the clarity of representation of

the real-world phenomena because several representations are in the set of models that

represent the same real-world phenomenon (Wand & Weber, 1993). It is likely that high

levels of ontological overlap will lead to misunderstanding, because users need to identify

overlapping constructs and reconcile their meaning.

The increase in the level of ontological overlap will also increase the complexity of the

representation of the domain (Gemino & Wand, 2005). If we consider a simple representation

of a domain as a representation without ontological overlap, based on Occam’s razor

(Domingos, 1999), which argues that simplicity is a goal itself, model users will avoid adding

more ontological overlap to the representation of the domain by selecting additional models.

However, the outcome expectation of model use is to create a complete understanding of a

114

domain, therefore adding more information, increasing ontological completeness, will

improve domain understanding. Therefore, we argue that model users will also follow the law

of parsimony: the least number of different models with the most explanatory power to

represent the real-world domain of interest will be used in combination. Based on these

reasonings, we argue that model users will use additional models to reach maximal level of

completeness only if adding one more model will increase the level of combined

completeness and is accompanied by a minimal level of increase in the overlap.

We also argue that developing a mental model is limited to the capability of individuals’

working memory. That is, an increase in ontological overlap between models will require

additional reasoning to develop understanding and increase the cognitive load (Sweller &

Chandler, 1994). The burden on the limited capacity of working memory can be reduced by

using models that present multiple constructs that can be interpreted as a single element (Paas,

Tuovinen, Tabbers, & Van Gerven, 2003; Soffer, Wand, & Kaner, 2015); therefore, they

require further integration to construct a mental model that increases cognitive load and

reduces efficiency in creating a clear and accurate understanding. Thus, we argue that model

users will not select an additional model if adding another model only increases the

ontological overlap.15

In the case of our materials, we therefore believe the following logic

would be used.

H2: Given any model participants start with, they will select an additional

model/s that add/s a minimum ontological overlap to the model combination.

6.5.2 Development of Domain Understanding

Recker’s (2014) theory argues that the level of domain understanding that can be

generated from the reading of a model combination will be a function of the combined

ontological completeness of the model combination. In fact, one of the primary objectives of

using conceptual models is to create understanding of some aspects of the social and physical

world around us (Gemino & Wand, 2004). Any model-based task requires constructing

knowledge from the information presented in the model and integrating that knowledge with

15 Recker’s (2014) theory also argues that users will select additional models until they reach a bearable level of

ontological overlap. The bearable level of ontological overlap is constrained by users’ processing capability. If

this capability is maximised, users will not select additional models even if adding another model would increase

combined ontological completeness. However, users processing capability is volatile and contextual; therefore,

we have no basis to speculate ex ante what the bearable threshold could be; thus, we cannot offer a hypothesis on

this element of the proposition.


the mental model developed by using the model. A combination of models will provide more

representation elements that convey meaning about the phenomena in a domain if and when

the combination has higher ontological completeness than any one model alone. More

information is then available for assimilation into the users’ mental model about all relevant

aspects of focal real-world phenomena (Recker, 2014). This combination should improve

domain understanding.

Moreover, interpretation of a model for domain understanding is a cognitive process

(Bodart et al., 2001; Gemino & Wand, 2005). The existence of a construct deficit in any

single conceptual model results in a lack of relevant information presented in the model.

Based on cognitive load theory, the way information is presented, that is, lack of information

in a domain presentation, can increase extraneous cognitive load (Kirschner, 2002). A

combination of models with high ontological completeness will not only reduce extraneous

cognitive load, but also present different forms of information (Burton-Jones & Meso, 2008)

(e.g., structural and behavioural). The availability of different forms of information may

promote germane cognitive load, and as a result facilitated learning and understanding. We

propose that model users who read a combination of models with high levels of combined

ontological completeness will generate higher levels of domain understanding than users who

read a combination of models with a low level of combined ontological completeness; thus

we developed the following hypotheses:

H3a: Users of the combination of models with a high combined ontological

completeness will achieve a higher score in the comprehension test compared to

the users of the combination of models with a low combined ontological

completeness.

H3b: Users of the combination of models with a high combined ontological

completeness will achieve a higher score in the problem-solving test compared to

the users of the combination of models with a low combined ontological

completeness.

H3c: Users of the combination of models with a high combined ontological

completeness will achieve a higher score in the Cloze test compared to the users

of the combination of models with a low combined ontological completeness.

Recker’s (2014) theory of faithful use of conceptual model combinations proposes that

an increase in the level of ontological overlap in a model combination will impede the

positive effect of combined ontological completeness. This is because, while a combination of

models with a high level of combined ontological completeness and a high level of

116

ontological overlap will provide more information than a single model, this achievement often

comes at the cost of decreased clarity if and when additional models overlap partially (i.e.,

they share at least partially redundant information with an existing model). In such a situation,

reading combinations of models with ontological overlap will require additional cognitive

effort to develop domain understanding, because users will need to identify and discriminate

elements that appear in multiple models and that convey the same meaning about some real-

world phenomena. Thus, this may increase the extraneous cognitive load experienced by

model users (Gemino & Wand, 2005). In turn, users will have less cognitive capacity

available to receive and integrate relevant information from models. We propose that the

positive impact of combined ontological completeness of model combinations on users’

ability to generate domain understanding would be decreased by the extent of ontological

overlap in the combination of models; thus leading to the following hypotheses:

H4a: Users of the combination of models with a high ontological overlap will

achieve a lower score in the comprehension test compared to the users of the

combination of models with a low ontological overlap.

H4b: Users of the combination of models with a high ontological overlap will

achieve a lower score in the problem-solving test compared to the users of the

combination of models with a low ontological overlap.

H4c: Users of the combination of models with a high ontological overlap will

achieve a lower score in the Cloze test compared to the users of the combination of

models with a low ontological overlap.

6.5.3 Perceived usefulness of model combinations

Finally, we examined the perceived usefulness of multiple models for a given task. The

ontological completeness of a modelling grammar has a direct effect on perception about the

usefulness of the grammar (Recker et al., 2011). We argue that similar mechanisms will affect

the perceived usefulness of conceptual models (Maes & Poels, 2007). Perceived usefulness

refers to the extent that model users believe a model combination helps them to achieve task

goals (Davis, 1989; Maes & Poels, 2007; Recker et al., 2011). We assumed that one of the

primary objectives of using models is to create complete and clear domain understanding

(Recker, 2014). Construct deficit that exists in a single model results in the lack of relevant

information in which a model reader seeks to develop a mental model about the domain.

Similarly, a model combination with a low level of combined ontological completeness

provides a representation with some level of relevant information about the domain.


Therefore, model readers require additional cognitive demand that will reduce their

understanding.

Model combinations with high levels of combined ontological completeness will be

perceived as more useful (Recker, 2014) because high combined ontological completeness

provides more information, and therefore greater ability for users to manifest relevant aspects

of real-world phenomena, which we call the direct information effect; thus leading to the

following hypothesis:

H5: Users of the combination of models with a high combined ontological

completeness will perceive the combination as more useful than the users of the

combination of models with a low combined ontological completeness.

Yet, in model interpretation tasks, a combination of models with high combined

ontological completeness and high ontological overlap will be evaluated as less useful,

because the representation with ontological overlap will undermine the gains in

representational coverage, in turn making the model combination less useful than a less

complete but less overlapped combination of models (Recker, 2014). This is because a high

level of ontological overlap in the model combination decreases the clarity (Wand & Weber,

1993) and increases the complexity of the representation of the domain (Gemino & Wand,

2005). The additional added complexity of the representation with ontological overlap will

increase the cognitive effort of interpretation of the information presented in the models

(Sweller, 1994). The high cognitive load will decrease the user’s capacity for cognitive

processing of information (Mayer, 1989). The high cognitive effort will undermine the

perception of usefulness (Recker, 2010a); we therefore hypothesised that:

H6: Users of the combination of models with a high ontological overlap will

perceive the combination as less useful than the users of the combination of the

models with a low ontological overlap.

6.6 DATA ANALYSIS

We analysed our data in four steps. First, we checked the results of the reliability and

homogeneity of variances. We then tested the model selection hypotheses. In the third step,

we tested the domain understanding hypotheses, and in the fourth step, we analysed the

perceived usefulness hypotheses.

118

6.6.1 Preliminary Tests

Our data analyses began with screening the data for its conformance with the

assumptions of the designed tests. We started by assessing the reliability and validity of the

Likert-type measures of the pre-test modelling familiarity measures, cognitive load, and post

hoc test, perceived usefulness measures through an exploratory factor analysis implemented

in IBM SPSS Statistics 22 (Tabchnick & Fidell, 2007). The results of factor analyses

indicated that one measurement item of the cognitive load needed to be eliminated due to low

loading (0.48). Therefore, we used three measurement items that appropriately captured the

cognitive load in engaging with the experimental tasks. All measurement items for modelling

familiarity and perceived usefulness showed an acceptable factor loading of more than 0.65

(Fornell & Larcker, 1981). Table 6.6 shows the average factor loadings and Cronbach’s alpha

for the measurement items of the three constructs. Appendix C summarises the factor loading

for each measurement items.

Table 6.6 Reliability and validity analysis of scaled measures

No. Items Average. F. Load Cronbach's Alpha

Modelling Familiarity 4 .774 .778

Perceived Usefulness 3 .860 .822

Cognitive Load 3 .837 .806

The homogeneity of variances assumption was verified using the Levene test. Table 6.7

shows the results. Levene statistics for each of the dependent measures suggest that the

hypothesis of equal variances was not rejected across any of the variables at the 0.05 level.

Table 6.7 Test of homogeneity of variances for dependent measures

Measure Levene Statistic df1 df2 Sig.

Comprehension .050 3 127 .985

Cloze Test 1.108 3 127 .348

Problem Solving .669 3 127 .572

Perceived Usefulness .433 3 127 .730

Cognitive Load 1.173 3 127 .323

6.6.2 Hypothesis Testing: Model Selection

Selection of a second model

One hundred and seven (82%) of 131 participants selected more than one type of

models. Only twenty-four of 131 participants selected one model to achieve the goals of the

upcoming task in the model selection experiment. Table 6.8 summarises this information.


Table 6.8 Frequency of participants and the number of models in combination

One model Two models Three Models Four Models All

24 31 41 35 131

18.32% 18.32% 23.66% 31.3% 100

Table 6.9 shows the descriptive statistics of the number of participants and the types of

models they selected as the first and second choice. The most widely selected model as a first

choice was the activity diagram. The activity diagram was also the most widely selected

model as the second choice. The least frequently selected model was the class diagram. Table

6.10 shows the mean and standard deviation of the constructs investigated in the model

selection. On average, users selected a second model that added almost three levels of

ontological completeness to the representational coverage of the model combination.

Table 6.9 Frequency and descriptive statistics for second model selection

Started Model # (%) Second Model #(total)

Model A) Use Case Diagram 47 (35.88)

Model B) Class Diagram 10 (19)

Model C) State Machine Diagram 9 (22)

Model D) Activity Diagram 22 (36)

None 6 (24)

Model B) Class Diagram 16 (12.21)

Model A) Use Case Diagram 5 (30)



None 3 (24)

Model C) State Machine Diagram

19 (14.5)




None 3 (24)

Model D) Activity Diagram 49 (37.40)




None 12 (24)

Table 6.10 Mean and standard deviation of constructs for model selection

Constructs Mean SD

Domain Familiarity 3.43 1.38

Modelling Familiarity 4.1939 .911

Familiarity with Selected Diagram .41 .49

Level of Completeness Added by the Second Model 2.84 1.23

Level of Overlap Added by the Second Model 1.95 .57

120

The correlation between constructs is presented in Table 6.11. The results of the

correlation matrix indicate that the selection of the second model had a statistically significant

positive correlation with the level of combined ontological completeness added by the model,

while it also had a statistically significant but negative correlation with the level of

ontological overlap added by the model.

Table 6.11 Correlation matrix for selection of second model

Constructs 1 2 3 4 5 6

1 Domain Familiarity 1

2 Modelling Familiarity 0.174 1

3 Familiarity with the Selected Model -0.150 0.041 1

4 Level of Completeness Added by the Second

Model -0.159 -0.86 0.077 1

5 Level of Overlap Added by the Second Model 0.026 -0.14 .202* -0.131 1

6 MOC and MOO -0.049 -0.012 -0.066 .716** -.355** 1

We conducted logistic regression analysis to test the predictors of the selection of the

second model using the level of combined ontological completeness and the level of

ontological overlap added by the second model, familiarity with the selected model,

modelling familiarity, and familiarity with the domain as covariates.

The results of model fit, as presented in Table 6.12, suggest that between 59.3 percent

and 79.4 percent of the variation in selecting the second model can be explained by the set of

independent variables (Pallant, 2007). We ran hierarchical logistic regression analysis with

and without the ontological variables. Based on the results presented in Table 6.12, the ability

of the model to predict the selection of the second model was higher when we used

ontological factors. The Wald test (Pallant, 2007) also supports that the level of completeness

added by the second model (β = 4.006, p = 0.00) and the level of overlap added by the second

model (β = - 4.217, p = 0.014) explains the selection of the model to achieve the maximal

combined ontological completeness and minimal ontological overlap. In line with Hypotheses

1 and 2, the results indicate that the more ontological completeness the second model added to

the combination, the more likely it was that the users would select the model. However, the

negative beta value for the level of ontological overlap the model added to the combination,

indicating that the more ontological overlap the model added, the less likely a user was to

select the model to add to the combination.


Table 6.12 Hierarchical logistic regression for selection of second model

B S.E. Wald Sig. Exp (B)

Model One

Domain Familiarity -.090 .146 .378 .538 .914

Modelling Familiarity .005 .219 .001 .982 1.005

Familiarity with Selected Model -.311 .404 .595 .441 .733

Constant .206 .995 .043 .836 1.229

Model Two


Modelling Familiarity .736 .427 2.969 .085 2.087

Familiarity with Selected Model -.516 .768 .451 .502 .597

Level of Completeness Added by the Second Model 4.006 1.127 12.631 .000 54.930

Level of Overlap Added by the Second Model -4.217 1.713 6.056 .014 .015

Constant -8.232 3.197 6.628 .010 .000

Model One Model Two

Chi-square 2.99 3.85

-2 Log Likelihood 146.34 50.93

Cox & Snell R Square 0.008 0.593

Nagelkerke R Square 0.011 0.794

Selection of a third model

Based on the data presented in Table 6.8, 76 participants selected a third or fourth

model to add to the combination. We evaluated the determinant factors for selection of the

third model. Table 6.13 presents the correlation between the factors. Similar to the selection

of the second model, the correlation matrix indicates that the selection of the third model had

a statistically significant positive correlation with the level of combined completeness added

by the third model.

Table 6.13 Correlation matrix for the selection of the third model

Constructs 1 2 3 4 5 6

1 Domain Familiarity 1

2 Modelling Familiarity 0.142 1

3 Familiarity with Selected Diagram 0.001 0.173 1

4 Level of Completeness Added by the Third Model -0.031 -0.115 0.142 1

5 Level of Overlap Added by the Third Model 0.010 -0.003 -0.024 -.266* 1

6 MOC and MOO -0.094 -0.14 0.192 .628** 0.205 1 *. Correlation is significant at the 0.05 level (2-tailed). **. Correlation is significant at the 0.01 level (2-tailed).

To test how the level of combined ontological completeness and ontological overlap

added by the third model could predict the selection of the model, we conducted hierarchical

logistic regression. As presented in Table 6.14, and similar to the selection of the second

122

model, the model with ontological factors was stronger in explaining the variations in the

selection of the third model (dependent variable: MOC and MOO of the combination of three

models).

Table 6.14 Hierarchical logistic regression for the third model selection

B S.E. Wald Sig. Exp(B)

Model One


Modelling Familiarity -.434 .270 2.592 .107 .648

Familiarity with Selected Model 1.268 .601 4.454 .035 3.554

Constant 2.254 1.223 3.399 .065 9.529

Model Two


Modelling Familiarity -.179 .566 .100 .752 .836

Familiarity with Selected Model 1.819 1.155 2.481 .115 6.168

Level of Completeness Added by the Third Model 4.113 1.188 11.975 .001 61.108

Level of Overlap Added by the Third Model 5.182 2.089 6.154 .013 178.086

Constant -.179 .334 .287 .592 .836

Model One Model Two

Chi-square 6.84 6.35

-2 Log Likelihood 97.74 32.99

Cox & Snell R Square 0.072 0.596

Nagelkerke R Square 0.097 0.808

Table 6.14 also shows the results of logistic regression to assess the impacts of

covariates and the treatment variables on the selection of the third model. Based on the results

of the Wald test, both the level of combined ontological completeness (β = 4.113, p = 0.001)

and the level of ontological overlap (β = 5.182, p = 0.013) the model added to the

combination had a statistically significant effect on the selection of the third model. However,

unlike the selection of the second model, the level of ontological overlap added by the third

model had a positive effect on the selection of the model. This may be due to the fact that in

selecting a third model, only four options out of 24 possible remaining options existed where

model users could select a model without ontological overlap. For any other options, the third

model added at least one level to the ontological overlap of the model combinations.

Therefore, we conducted a post hoc analysis to investigate the selection of more than two

models in more detail.


Post hoc test

To better understand the behaviour of participants in model selection, we looked at

participants’ responses to two questions: first, which models they relied on in answering the

questions, and second, whether they would like to remove any diagram/s from the

combination they had already selected. Table 6.15 shows the number of models participants

selected and the number of models they used in combination to answer the problem-solving

questions.

Table 6.15 Number of models selected and number of models they used in combination to

answer the questions

No. of models

selected

No. of

participants

No. of models used

in combination

No. of

Participants

Most frequently used

model combination

Three models 41

3 9 use case, class and activity

2 25 activity and use case

1 7 activity

Four models 35

4 2 + 7 all four models

3 17 use case, state, activity

2 6 activity and state/use case

1 3 activity

As presented in Table 6.15, only 35 participants used three or four models in

combination. Seventy-six percent of the participants who selected a third or fourth model

would have liked to remove at least one of the models they had selected.

Out of 41 participants, only nine participants who selected the third model used all three

models in combination to answer the questions, while 32 participants would have liked to

remove some of the models. Overall, out of 131 participants, 26 participants used three

models in combination to answer the questions, and 73.3% of participants used two or less

models in combination. Another dominant finding was that only two participants who

selected four models used all four models to answer the questions and seven mentioned that

they were not sure whether they could answer the questions using all four models. If we

remove those seven participants, 93% of the participants who selected the fourth model used

three or less models in combination to answer the questions. Table 6.16 shows the number of

users and types of models selected and dropped.

124

Table 6.16 Frequency of types of models

Types of Models Number of Users

Selected Dropped Used

Use Case 99 26 73

Class 67 25 42

State Machine 73 21 52

Activity 110 4 106

These findings indicate that participants would have liked to remove some models from

the combination, and as a result decrease the combined ontological completeness and also the

ontological overlap.

6.6.2 Hypothesis Testing: Domain Understanding

We began by screening the data collected in the second part of the experiment. Table

6.18 presents descriptive statistics and Table 6.19 shows the correlation matrix. We ran an

ANOVA test to compare the means for modelling familiarity, diagram familiarity, and

domain familiarity across groups. The results show that there were no statistically significant

differences across groups in terms of the level of the familiarity with modelling, diagrams,

and domains (see Table 6.17). These results indicate that the participants were effectively

randomised across treatments.

Table 6.18 shows that the standard deviations of the means for modelling, diagrams,

and domains familiarity across groups were almost more than one unit. This indicates that

there was sufficient variation in the level of familiarity within the groups. From the

correlation matrix, we noted that performance on the comprehension test and ontological

overlap had a statistically significant correlation. We also noted that cognitive load had a

statistically significant correlation with modelling familiarity (at the 0.01 level) and with

UML diagrams familiarity (at the 0.05 level). We then evaluated the performance of

participants in domain understanding. Responses to the comprehension test were evaluated

based on the number of correct answers to the eight multiple choice questions.

Table 6.17 ANOVA test to compare familiarity difference between groups

Sum of Squares df Mean Square F Sig.

Familiarity Self Rate Library 1.682 3 .561 .350 .789

Familiarity Self Rate 5.203 3 1.734 1.967 .122

Familiarity Diagrams Test Correct Answers 7.113 3 2.371 1.278 .285

Chapter 6:An Experimental Study to Test and Extend the Theory of Faithful Use of Conceptual Model Combinations

125

Table 6.18 Descriptive statistics

Constructs Modelling

Familiarity

Diagram

Familiarity

Domain

Familiarity

(Library)

Cognitive

Load Comprehension

Problem

Solving Cloze

Perceived

Usefulness

Scale 1-7 Count. 1-7 1-7 Count. Count. Count 1-7

Group A

M 4.48 2.42 4.09 4.46 3.73 8.79 6.76 4.97

SD 0.81 1.56 1.21 0.92 1.42 2.33 3.61 1.23

Group B

M 3.92 1.81 3.77 4.19 4.32 8.35 7.29 4.9

SD 1.03 1.3 0.96 0.77 1.58 2.78 2.89 1.16

Group C

M 4.14 1.91 3.88 4.48 3.5 7.53 7.47 5

SD 0.99 1.19 1.36 0.66 1.44 3.35 3.07 1.09

Group D

M 4.18 2.06 3.94 4.37 4.18 7.42 7.58 5.37

SD 0.91 1.37 1.46 0.85 1.33 3.06 3.79 0.99

126

Table 6.19 Construct correlations

1 2 3 4 5 6 7 8 9 10

1 Combined Ontological Completeness 1

2 Ontological Overlap .023 1

3 Modelling Familiarity .156 -.137 1

4 Diagram Familiarity .140 -.083 .326** 1

5 Domain Familiarity (Library) .074 -.050 .131 -.033 1

6 Performance on Comprehension Test .021 .218* -.097 .044 -.116 1

7 Performance on Problem Solving Test .031 -.047 -.029 .029 .080 .229** 1

8 Performance on Cloze Test -.033 .048 .009 .137 -.002 .298** .352** 1

9 Perceived Usefulness .097 .072 -.017 .071 -.037 .164 .182* .186* 1

10 Cognitive Load .072 -.075 -.312** -.173* .014 -.099 -.100 -.242** .138 1

*. Correlation is significant at the 0.05 level (2-tailed).

**. Correlation is significant at the 0.01 level (2-tailed).


Comprehension

To test hypotheses 3a and 4a, we ran ANCOVA analysis with combined ontological

completeness and ontological overlap as factors, modelling familiarity, diagram familiarity,

and cognitive load as covariates, and comprehension performance as the dependent measure.

Table 6.20 shows the results.

Table 6.20 Results for model comprehension

Difference among groups: Means (Std. Deviation)

Combined ontological Completeness

Low High

Ontological Overlap

Low 3.50

(1.44) 3.73

(1.42)

High 4.32

(1.58)

4.18

(1.33)

ANCOVA

Source df F Sig. Partial

2

Corrected Model 7 1.602 .141 .084

Intercept 1 30.814 .000 .200

Modelling Familiarity 1 1.604 .208 .013

Domain Familiarity 1 1.045 .309 .008

UML Diagrams Familiarity 1 .572 .451 .005

Cognitive Load 1 1.435 .233 .012

Combined Ontological Completeness 1 .198 .657 .002

Ontological Overlap 1 4.837 .030 .038

Combined Ontological Completeness * Ontological

Overlap 1 0.492 .484 .004

R Squared = .084 (Adjusted R Squared = .031)

As Table 6.20 shows, comprehension performance was driven by ontological overlap

rather than combined ontological completeness. Partial eta square value of 0.038 for

ontological overlap indicates that there was a small effect size based on Cohen (1988)

guidelines.

In line with Hypothesis 3a, the results indicate that when there was low ontological

overlap an increase in the level of combined ontological completeness improved model users’

performance in the comprehension test; however, the effect was not statistically significant.

While the positive impact of the combined ontological completeness decreased when

there was high ontological overlap in the model combinations, contrary to the Hypothesis 4a,

users of the combinations of models with high overlap still performed better compared to the

128

users of the model combinations with low overlap. The difference between groups was not

statistically significant.

Problem-Solving

To test Hypotheses 3b and 4b, we ran an ANCOVA test with combined ontological


and cognitive load as covariates, and problem-solving performance as the dependent measure.

Table 6.21 shows the results.

Table 6.21 Results for problem-solving



Low High

Ontological Overlap

Low 7.53

(3.35)

8.79

(2.33)

High 8.35

(2.78)

7.42

(3.06)

ANCOVA


2


Intercept 1 24.459 .000 .166


Domain Familiarity 1 .988 .322 .008

UML Diagrams Familiarity 1 .059 .809 .000



Ontological Overlap 1 .542 .463 .004


Overlap 1 4.468 .037 .035


The results indicate that combined ontological completeness and ontological overlap did

not have statistically significant effects on a user’s performance in the problem-solving test.

However, the results of ANCOVA show a statistically significant interaction effect between

combined ontological completeness and ontological overlap. Partial eta square values of

0.001, 0.002, and 0.035 respectively for ontological overlap, combined ontological

completeness, and the interaction effect indicate the small effect size based on Cohen’s (1988)

guidelines.


Therefore, we conducted an additional simple effect analysis to identify the origins of

the interaction effect (Elliott & Woodward, 2007). Table 6.22 summarises the results of the

simple effect analysis.

Table 6.22 Simple effect analysis for problem-solving performance

Combined Ontological

Completeness

Ontological Overlap

Mean Difference

Std.

Error Sig.

Low Low (I) High(J)

-.697 .732 .343

High Low (I) High(J) 1.464 .728 .047

Ontological Overlap Combined Ontological Completeness

Mean Difference Std. Error Sig.

Low Low (I) High(J)

-1.350 .728 .066

High Low (I) High(J) .811 .739 .275

The results presented in Table 6.22 show that when there was a high combined

ontological completeness, users of the model combination with a low ontological overlap

performed better than the users of the model combination with high ontological overlap (mean

difference = 1.464, p = 0.047). The results are in line with Hypothesis 4b.

In line with Hypothesis 3b, the findings indicate that when there was a low ontological

overlap, an increase in combined ontological completeness improved model users’

performance by almost 10 percent. However, there was no statistically significant difference

between groups (mean difference = -1.350, p = 0.066).

Similarly, to analyse the differences of the means of performance on the Cloze test

across groups, we ran ANCOVA analysis with combined ontological completeness and

ontological overlap as factors, modelling familiarity, diagram familiarity, and cognitive load

as covariates, and performance in the Cloze test as the dependent measure. Table 6.23

presents the means and standard deviations across groups and the ANCOVA test.

130

Table 6.23 Results for Cloze test



Low High

Ontological Overlap

Low 7.47

(3.07) 6.76

(3.61)

High 7.29

(2.89)

7.58

(3.79)

ANCOVA


2


Intercept 1 24.233 .000 .165



UML Diagrams Familiarity 1 2.246 .136 .018





Overlap 1 .964 .328 .008


As the results indicate, combined ontological completeness and ontological overlap did

not have statistically significant effects on model users’ performance in the Cloze test.

However, cognitive load appeared to be a significant determinant of the performance on the

Cloze test.

In contrast with Hypothesis 3c, the results show that when there was low ontological

overlap, an increase in the level of combined ontological completeness decreased the retention

ability of model users. However, the effect was still not statistically significant.

The findings of the ANCOVA test reveal that when there was high ontological overlap,

an increase in the combined ontological completeness improved model users’ performance in

the Cloze test. The users of the model combination with high ontological overlap and high

combined completeness performed better on the Cloze test compared to the users of a model

combination with high combined ontological completeness but low ontological overlap.

These results are in contrast with what Hypothesis 4c predicted, with no statistically

significant effect of ontological overlap.

Figure 6.4 shows the estimated marginal means of model users’ performances on

comprehension, problem-solving, and the Cloze test and illustrates the direction of the effects


of the combined ontological completeness and ontological overlap on domain understanding.

To be able to compare the marginal means, we normalised the graphs by dividing the mean of

the performance on each task by the maximum number of the correct answers provided for

each task to achieve a percentage score ranging from 0 to 1.

The results show that the effect of combined ontological completeness and ontological

overlap varied for each task. For both comprehension and problem-solving tasks, when there

was a low level of ontological overlap, an increase in combined ontological completeness

improved participants’ performance. However, this effect was not statistically significant on

participants’ performance in both tests. The direction of this effect on participants’

performance in the Cloze test was in contrast to both comprehension and the problem-solving

tests; however, the effect was not statistically significant.

As Figure 6.4 shows, an increase in ontological overlap had a major impact on

participants’ performance in the comprehension test. In contrast, an increase in ontological

overlap decreased participants’ performance in the problem-solving test. In particular, there

was a statistically significant difference between the performance of users of a model

combination with high ontological completeness and high ontological overlap and the users of

a model combination with low ontological overlap and high ontological completeness.

However, for the Cloze test, an increase in the level of ontological overlap improved users’

performance when the level of combined ontological completeness of models was high.

132

Covariates appearing in the model were evaluated at the following values: Domain Familiarity = 3.9237, Diagram Familiarity = 2.05, Modelling

Familiarity = 4.1870, Cognitive Load = 4.1170

Figure 6.4 Estimated Marginal Means of Performance on Domain Understanding

0.4

0.45

0.5

0.55

0.6

Low High


Low

High

Pe

rfo

rm

an

ce

on

Co

mp

re

he

ns

ion

Te

st Ontological

Overlap

0.4

0.45

0.5

0.55

0.6

Low High


Low

High

Ontological Overlap

Pe

rfo

rm

an

ce

on

Pro

ble

m S

olv

ing

T

es

t

0.4

0.45

0.5

0.55

0.6

Low High


Low

High

Ontological Overlap

Pe

rfo

rm

an

ce

on

C

lo

ze

Tes

t


To analyse Hypotheses 5 and 6, we ran ANCOVA analysis with combined ontological


and cognitive load as covariates, and perceived usefulness as the dependent measure. Table

6.24 illustrates the results.

Table 6.24 Results for perceived usefulness



Low High

Ontological Overlap

Low 5.00

(1.09) 4.97

(1.23)

High 4.90

(1.16)

5.37

(0.99)

ANCOVA


2


Intercept 1 28.607 .000 .189

Modelling Familiarity 1 .006 .936 .000


UML Diagrams Familiarity 1 1.072 .302 .009





Overlap 1 1.667 .199 .013


The results did not show any statistically significant effect of combined ontological

completeness and ontological overlap on perceived usefulness. Users of the model

combination with high combined ontological completeness and low ontological overlap

perceived the model combination as less useful than the users of the model combination with

low combined ontological completeness and low ontological overlap. While the effect of

combined ontological completeness was not statistically significant, the results do not support

Hypothesis 5. Figure 6.5 demonstrates that between the users of the model combinations that

had a high combined ontological completeness, users of a combination of models with high

ontological overlap perceived the combination as more useful than the users of the

combination with low ontological overlap. While the effect of ontological overlap was not

statistically significant, this result is in contrast with the expectation of Hypothesis 6.

134

Covariates appearing in the model were evaluated at the following values: Domain Familiarity = 3.9237,

Diagram Familiarity = 2.05, Modelling Familiarity = 4.1870, Cognitive Load = 4.1170

Figure 6.5 Estimated Marginal Means of Perceived Usefulness

6.7 DISCUSSION

6.7.1 Summary of Findings

Table 6.25 summarises the insights gained from the data analysis.

Table 6.25 Summary of hypothesis testing

Hypotheses Results Interpretation

Mo

del

Sel

ecti

on

H1 Given any model participants

start with, they will select an

additional model/s that add/s

maximum ontological

completeness to the model

combination.

Combined ontological completeness had

a statistically significant positive effect

on model selection.

The hypothesis is

supported.

H2 Given any model participants

start with, they will select an

additional model/s that add/s

minimum ontological overlap

to the model combination.

Ontological overlap had a statistically

significant negative effect on the

selection of a second model. Seventy-six

participants who selected three or more

models did not use at least one of the

models in combination.

The direction of the

effect is as

expected; however,

the hypothesis is

not supported.

Dom

ain U

nder

stan

din

g

H3a Users of the combination of models with a high combined

ontological completeness will

achieve a higher score in the

comprehension test compared

to the users of the combination

of models with a low combined

ontological completeness.

The group that had a model combination with high combined ontological

completeness and low ontological overlap

performed better on the comprehension

test compared to the group that had a

model combination with low combined

ontological completeness and low

ontological overlap; however, the

difference did not reach statistical

significance.

The direction of the effect is as

expected; however,

the hypothesis is

not supported.

H4a Users of the combination of

models with a high ontological

overlap will achieve a lower score in the comprehension test

compared to the users of the

When there is high combined ontological

completeness, users of the model

combination with high overlap performed better on the comprehension test

compared to users of the model


effect is not as

expected and the hypothesis is not

supported.

0.6

0.65

0.7

0.75

0.8

Low High


Low

High

Perc

eive

d U

sefu

lnes

sOntological

Overlap


combination of models with a

low ontological overlap.

combination with low ontological

overlap; however, the difference did not

reach statistical significance.

H3b Users of the combination of

models with a high combined



problem-solving test compared

to the users of the combination

of models with a low combined


Users of a model combination with high

combined ontological completeness and

low ontological overlap performed better

in the problem solving test compared to

users of a model combination with low


low ontological overlap; however, the


significance.

The direction of

effect is as

expected; however,

the hypothesis is

not supported.

H4b Users of the combination of

models with a high ontological overlap will achieve a lower

score in the problem-solving

test compared to the users of

the combination of models with

a low ontological overlap.

Model users of a model combination with

high combined ontological completeness and high ontological overlap achieved a

lower score in the problem solving test

compared to users of a model

combination with high combined

ontological completeness and low

ontological overlap, and the difference

reached statistical significance.

The hypothesis is

supported.

H3c Users of the combination of




Cloze test compared to the users of the combination of

models with a low combined




low ontological overlap achieved a lower

score on the Cloze test compared to users

of a combination with low combined ontological completeness and low

ontological overlap; however, the


significance.


effect is not as

expected and the

hypothesis is not

supported.

H4c Users of the combination of


overlap will achieve a lower

score in the Cloze test

compared to the users of the

combination of models with a


Model users of a model combination with

high combined ontological completeness

and high ontological overlap performed

better on the Cloze test compared to users

of a model combination with high




effect is not as

expected and the

hypothesis is not

supported.

Per

ceiv

ed U

sefu

lnes

s

H5 Users of the combination of


ontological completeness will perceive the combination as

more useful than the users of

the combination of models with

a low combined ontological

completeness.



low ontological overlap perceived the combination less useful compared to

users of a model combination with low


low ontological overlap; however, the


significance.


effect is not as

expected and the hypothesis is not

supported.

H6 Users of the combination of


overlap will perceive the

combination as less useful than

the users of the combination of

the models with a low ontological overlap.



low ontological overlap perceived the

combination less useful compared to

users of a model combination with high

combined ontological completeness and high ontological overlap; however, the


significance


effect is not as

expected and the

hypothesis is not

supported.

136

With respect to the model selection, our results were reasonably clear. Our findings are

in line with Hypothesis 1, indicating that model users added an additional model if the model

maximised the combined ontological completeness of the model combination. Specifically,

we found that the level of combined ontological completeness an additional model added to

the combination had a statistically significant positive impact on the selection of the model.

This finding is significant in that it confirms that model users desire a complete representation

of a domain (Recker, 2014), which also underlines the importance of design consideration

with regards to types of models. The results were also in line with Hypothesis 2, when model

users selected the second model to add to the model combination. The findings indicate that

the level of ontological overlap added to the model combination had a statistically significant

negative impact on the selection of the model. However, the hypothesis was not supported

when model users selected a third model to add to the model combination. Because this result

was a surprise, we set out to determine a possible explanation. A deep investigation of the

collected data indicated that more than 76% of the participants who picked more than two

models did not use all of the models in combination. Indeed, they would have liked to remove

some of them from the combination. By removing some models, they decreased the level of

ontological overlap; however, they also decreased the level of combined ontological

completeness. These results indicate that model users preferred less complete with less

overlap combinations than a more complete but also with more overlap combination. In line

with the prediction of the theory of faithful use of conceptual model combinations, the

selection of which model to use in combination was governed by the principles of clarity over

completeness (Recker, 2014). Some studies have also indicated that model users may trade-

off completeness to achieve an easier model to use (Bowen et al., 2009; Siau & Lee, 2004; zur

Muehlen et al., 2007).

With respect to domain understanding, our results were equivocal. To evaluate the

understanding gained from model combinations, we used three measures: a model

comprehension test, a problem-solving test to measure the deep-level domain understanding,

and a Cloze test to measure surface-level domain understanding by evaluating the ability of

participants to reproduce (or recall) absorbed information (Bera et al., 2014; Bodart et al.,

2001; Recker & Dreiling, 2011; Recker et al., 2014; Shanks et al., 2010).


In line with Hypotheses 3a and 3b, we found that when there was low ontological

overlap, an increase in combined ontological completeness improved both comprehension

performance and problem-solving performance; however, the effect was not statistically

significant. In contrast to Hypothesis 4a, our data analysis shows that ontological overlap had

a statistically significant and positive impact on participants’ comprehension performance.

However, in line with Hypothesis 4b, we found a statistically significant negative impact of

ontological overlap on problem-solving performance when participants used combinations of

models with high combined ontological completeness.

These findings are interesting because ontological overlap had a major positive impact

on model users’ performance on the model comprehension test. On the other hand,

ontological overlap had a statistically significant but negative effect on problem-solving

performance. One possible reason for these findings is the difference between the nature of

the model comprehension test and the problem-solving test. The model comprehension test

measures a surface understanding of the model combinations, while the problem-solving test

measures a deep understanding of the domain. Surface-level understanding requires users to

answer the questions based on the information plucked from their working memory (Mayer,

1989), and involves surface processing of the information without interpretation, reasoning, or

adding new elements (Offir, Lev, & Bezalel, 2008). Deep understanding involves deep

processing of information (Mayer, 1989, 2005), connecting the facts and concepts and

integrating the ideas with prior knowledge and collected information plucked from long-term

memory.

A model combination with high ontological overlap contains repeated constructs

representing the same element of a real-world domain. Studies on learning preferences

indicate that repetition could have a positive effect on learning (Bromage & Mayer, 1986;

Mayer, 1983). Repeating the information requires only surface processing without quotes or

interpretation. Therefore, the repetition of constructs in the model combination may help

participants to better link the elements of the domain represented in the models and create

higher surface level understanding.

While learning preferences hypotheses argue that repetition can improve learning

performance, the cognitive theory of multimedia learning, on the other hand, argues that

human working memory has a limited capacity. Repeated information would require

138

additional capacity out of the limited capacity of human working memory (Mayer, 2009, p.

126) and create extraneous cognitive load (Sweller, 1994; Sweller & Chandler, 1994). Given

that a deep understanding requires a deep cognitive process of handling of information

(Mayer, 1989, 2005), a more complete representation of a domain provides users with more

information that entails more cognitive demand to absorb information for problem-solving.

Therefore, additional extraneous cognitive load, introduced by the ontological overlap, will

greatly shrink the capacity of the human working memory to acquire information and generate

deep-level understanding.

In terms of Hypotheses 3c and 4c, our results show no evidence of significant

differences between treatment groups for the recall test (Cloze test). A lack of significant

differences between all treatment groups for the Cloze test may result from the fact that

participants responded to the Cloze test without having access to the models, in contrast to the

comprehension and the problem-solving test. Our findings also indicate that there was no

significant difference across treatment groups for the perceived usefulness of the model

combinations. However, the directions of the potential effects of combined ontological

completeness and ontological overlap on the recall test were similar to that of perceived

usefulness. While any explanation of these unexpected results is speculative, it is worth

noting that the perceived usefulness questions were immediately after the Cloze test. Thus,

there may have been a carry-over effect from the previous task.

6.7.2 Theoretical Implications

Our research provides the first dedicated empirical evaluation of the theory of faithful

use of conceptual model combinations. Similar to the argument of the theory, we identified

that combined ontological completeness and ontological overlap can explain the faithful use

of conceptual model combinations. In particular, our findings indicate that both combined

ontological completeness and ontological overlap are major determinants of users’ selection

of a model combination. Similar to the prediction of the theory, we also found that model

users compromised combined ontological completeness over ontological clarity. Our findings

indicate that users would have liked to remove some models from the combination they had

selected. A possible explanation is that the model combination reached the maximum bearable

level of ontological overlap. However, in this study, we could not offer a measurement to

evaluate the bearable level of overlap.


Our findings also extend the theory in a number of ways. First, we found that the effect

of combined ontological completeness and ontological overlap varies based on different task

settings. Accordingly, different attributes of different tasks should be considered when

evaluating the importance of combined ontological completeness and ontological overlap.

Studies on task setting indicate that based on the type of task, the information seeking

behaviour and use of information may vary (Byström & Järvelin, 1995; Campbell, 1988). Our

results corroborate this argument: overlap helps when model users are involved with tasks

that require surface-level understanding, whereas overlap has a negative effect when users are

involved with tasks that require deep-level understanding.

Second, we have determined that there is an interaction effect between ontological

overlap and combined ontological completeness. Recker’s (2014) theory suggested that the

level of understanding would be moderated by the level of ontological overlap. Our findings

indicate that the negative effect of ontological overlap overcomes the positive impacts of

ontological completeness. However, our findings indicate that when there is low ontological

overlap, an increase in combined ontological completeness improves model users’ problem-

solving performance.

These results can be interpreted by drawing on principles from the cognitive theory of

multimedia learning: this theory suggests that learning has three outcomes, no learning, rote

learning, and meaningful learning (Mayer, 2009, p. 21). No learning occurs when participants

have poor performance on both a retention test and a transfer test. Rote learning occurs when

participants only have a good performance on a retention test, and meaningful learning occurs

when participants have good performance on both a retention test and a transfer test. Our

results indicate that ontological overlap improves rote learning. However, when there was low

ontological overlap, an increase in combined ontological completeness improved participants’

performance on both the comprehension test (retention test) and the problem-solving test

(transfer test). These results indicate that a model combination with low ontological overlap

but high combined ontological completeness leads to meaningful learning.

6.7.3 Practical Implications

Our findings inform a largely neglected aspect of conceptual modelling practice – the

use of multiple types of conceptual models in combination. Two main practical implications

140

flow from this work. First, the findings of this study provide practical guidance that can assist

conceptual modelling practitioners to select and use appropriate combinations of conceptual

models for different systems analysis and design tasks. This study provides evidence that the

primacy of ontological completeness and clarity varies for different tasks. Therefore, model

end users should be aware of the essence and objectives of tasks and whether tasks require a

parsimonious representation or a complete representation of the real world.

Second, while our study focused on model interpretation rather than model design, we

believe that our findings also provide practical implications for model designing practices.

This is because conceptual modelling process – either the design or use of models – are

ultimately dependent on how well individuals can understand the domain presented by the

models (e.g., Burton-Jones & Meso, 2008; Recker et al., 2014). Our findings show that

different levels of completeness and overlap are useful for different levels of understanding.

Therefore, model designers should design partially redundant representations or complete

representations depending on the purpose of the design of the conceptual models. These

findings inform model designers with regards to which types of models should be designed to

effectively support an upcoming task.

6.7.4 Limitations

Several limitations pertain to our study. First, the experimental procedures tasks, and

models used in our experiment may not faithfully reflect the scale of real-world modelling in

practice. This is common for experimental studies (e.g., Bera et al., 2014; Figl, Mendling, et

al., 2013). Experiments are artificial situations mainly focused on controlling variables and

maximising internal validity (Calder et al., 1981).

Second, we used a sample of students rather than a sample of practitioners. Despite

concerns regarding the limitations of using students as a sample (Compeau, Marcolin, Kelley,

& Higgins, 2012), we believe that our sample was still suitable for our study for several

reasons. First, our experiment was designed to test the faithful use of conceptual model

combinations. The use of conceptual models was professionally and personally relevant for

our students because they had enrolled in conceptual modelling and systems analysis and

design courses. Second, in the context of our study, students are regarded as reasonable

proxies of junior working experts. For instance, Arisholm and Sjøberg (2004) found little


difference between students and junior professionals in understanding UML diagrams and

maintaining Java programs.

Third, we used combinations of UML diagrams in our experiment. However, our

findings from the interviews (Jabbari Sabegh & Recker, 2017) and some previous studies

indicate that IS professionals may integrate UML diagrams with other types of models, such

as ERD or BPMN (e.g., Petre, 2013). This limits the generalisability of our findings; however,

our primary objective in the experimental study was to control our variables and increase our

internal validity. We only used UML diagrams to avoid the potential extraneous effect of

individuals’ different levels of familiarity with UML diagrams compared to other types of

models.

6.8 CONCLUSION

Through this research, we have contributed to conceptual modelling research by

providing an empirical evaluation of the theory of faithful use of conceptual model

combinations. Our findings show the important role of completeness and clarity of model

combinations on model selection and domain understanding. These findings provide an

understanding of the use of conceptual models in practice, and in turn, can guide model end

users in their decisions when choosing to use conceptual models and inform executive

decision makers with regards to their investment to implement conceptual modelling

initiatives. In summation, our study informs the use of model combinations in practice and

may ultimately lead to more effective and efficient use of conceptual models in information

systems analysis and design tasks.

142

Part C: Retrospective

Chapter 7: Discussion

7.1 SYNTHESIS OF FINDINGS FROM STUDIES

This section summarises the findings of the three studies and discusses the implications

of the findings. The importance of conceptual modelling in information systems analysis and

design, a lack of knowledge about the use of multiple models in practice, and the constantly

changing and developing world motivated the first study of this dissertation. Study 1

answered the first research question: “What is the current state of knowledge of conceptual

modelling in IS?” To address this research question, Study 1 reviewed the current research

efforts in conceptual modelling and provided a revised research framework that suggests new

opportunities for research and stresses the need for ongoing innovation in conceptual

modelling in response to the emerging practices in IS.

Study 1 reviewed studies published in top IS journals and examined the extent to which

the current state of knowledge in conceptual modelling has addressed the challenges and

opportunities underlined by a formative and influential framework proposed by Wand and

Weber (2002). The results disclosed that some of the opportunities noted in 2002 have been

substantially addressed; however, some aspects of conceptual modelling remain entirely

overlooked. The findings of the first study also revealed several innovative studies that did

address some aspects that were not highlighted by the initial framework proposed by Wand

and Weber (2002). Study 1 also provided a new research framework that draws the attention

of conceptual modelling research to the interplay between digital representations and

outcomes and stimulates a new, relevant wave of conceptual modelling research. The findings

from a review of the literature are detailed in Chapter 4. One of the most notable unanswered

opportunities concerns the lack of empirical investigation on the use of multiple models. To

date, little research has been conducted to evaluate multiple models and investigate how a

model combination can support system analysis and design tasks.

Chapter 7:Discussion 143

Motivated by the practical observations and the lack of knowledge in IS literature about

the use of multiple models, the second study of this dissertation set out to answer the second

research question: “Why do systems analysts and designers use multiple conceptual models?”

To address this research question, Study 2 explored the reported use of multiple conceptual

models for system analysis and design tasks. Study 2 provided empirical evidence to address

the concerns highlighted in Study 1 about the lack of research in regards to the use of model

combinations. It provided qualitative support for the argument of previous studies (Dobing &

Parsons, 2008; Fettke, 2009; Whittle et al., 2014) that practitioners do indeed use multiple

models in system analysis and design tasks. The findings uncovered three main reasons why

practitioners use multiple types of models. These reasons are: 1) the size and complexity of

the domain, 2) involvement of different stakeholders in the projects, and 3) incompleteness of

any particular model.

As with the predictions of previous ontological analyses (e.g., Green et al., 2007;

Recker et al., 2009; Siau, 2010), incompleteness of single models led IS professionals to use

multiple models. In line with the findings of J. Kim et al. (2000), the finding of Study 2

indicated that practitioners use multiple models to understand different aspects of the domain,

especially when the domain is large and complex. Another reason why professionals use

multiple models is to meet the requirements of different stakeholders involved in the project,

such as project owners, end-users, or system analysts and designers. This finding is also in

line with previous studies (e.g., Darke & Shanks, 1997; Topi & Ramesh, 2002).

The findings of Study 2 also proposed theoretical relationships between these factors.

The results indicated that both the size and complexity of the domain and incompleteness of

single models are the primary determinants of how IS professionals use different types of

models combination. However, the extent and type of models they select will be moderated

by the requirements of the involved stakeholders. These relationships are consistent with

Wand and Weber’s (1990a, 1993) representation theory, while they also extend the main

premises of the theory. The findings uncovered that in addition to the ontological capabilities

of a model, the supported perspective of the model is also a key determinant of the use of

models. In line with Wand and Weber’s (2002) framework on conceptual modelling, scripts

(incompleteness of a single model) and context (size and complexity of domain and

144

involvement of different stakeholders) explain why IS professionals use different types of

conceptual models in combination.

Study 2 also explored the semantic and pragmatic reasons that influence the choice and

selection of different models for system analysis and design tasks. The semantic reasons

indicated that practitioners use multiple models in combination where: 1) they increase the

completeness of representation, 2) have low overlap, and 3) there is correspondence between

them.

The semantic reasons are in line with the propositions of the theory of faithful use of

conceptual model combinations (Recker, 2014), and other extensions of ontological theories

(Green, 1996; Weber, 1997). The findings indicate that IS professionals select a combination

of models that increase the completeness of the representation while also attempting to avoid

overlapping models, because overlap adds unnecessary complexity. However, Study 2

discovered a new concept: that of correspondence. Correspondence implies that while

practitioners avoid overlapping models, the existence of semantically related elements in pairs

of models influences the selection of these models.

The pragmatic reasons indicate that IS professionals use different types of models in

combination that: 1) are required by organisational regulations and standards, 2) are required

for different types of stakeholders involved in the project, 3) are widely used in industry, 4)

support the task goals, and 5) are supported by the available tools. In and of themselves, these

findings are also consistent with previous studies (e.g., Delfmann et al., 2015; Recker et al.,

2010; Rosemann, 2006; Samuel et al., 2015). However, the dominant finding of Study 2 is the

relationships between pragmatic factors, semantic factors, and the use of model combinations.

The findings indicate that pragmatic reasons might overrule semantic reasons. For instance,

similar to the findings of Green et al. (2011) on the use of multiple grammars, the findings of

Study 2 also indicate that organisational conventions may direct model users to choose

models that overlap. These findings are detailed in Chapter 5.

Motivated by the principles of the theory of faithful use of conceptual model

combinations, Study 3 set out to test the propositions of the theory (Recker, 2014) and

provide a deep understanding and empirical insights into the combined use of conceptual

models. Study 3 was a theory-testing study aimed at addressing the third research question:


“What characteristics of models affect users’ selection, understanding, and perceived

usefulness of model combinations?” To address this research question, Study 3 tested the

propositions of the theory of faithful use of conceptual model combinations through a

laboratory experiment.

The findings of Study 3 partially confirmed the selection proposition of the theory of

faithful use of conceptual model combinations. In line with the semantic reasons revealed in

Study 2, and conforming to the predictions of the theory (Recker, 2014), the findings of Study

3 indicate that the level of completeness a model added to the model combination had a

statistically significant positive effect on user’s selection of the model. These findings also

indicate that the level of ontological overlap a model added to the model combination had a

statistically significant negative effect on users’ selection of the second model. While the

tolerable level of ontological overlap could not be measured (Recker, 2014), the findings of

Study 3 indicate that participants would have liked to remove some models from the

combination they had selected to decrease the overlap, which also comes at the cost of

decreasing the completeness of the model combination. Results indicate that the use of

multiple models in combination can be governed by the principles of clarity over

completeness. Figl and Recker (2014) found that users’ preference for different representation

formats varies dependent on task purpose. The findings of Study 3 indicate that, for an

upcoming problem-solving task, participants selected models that increased the completeness

of the representation; however, they did not select additional models if the model increased

the ontological overlap of the model combination.

The findings of Study 3 reveal the different role of ontological overlap in different task

settings. These findings provide empirical evidence to support the theoretical relationship

between pragmatic reasons and semantic reasons, as proposed in Study 2. The results of Study

3 confirm that pragmatic reasons may overrule the ontological factors depending on the

context of use. The findings indicate that ontological overlap had a statistically significant

positive impact on users’ performance in models comprehension. However, an increase in

ontological overlap had a statistically significant negative effect on users’ problem-solving

performance. The propositions of the theory of faithful use of conceptual model combinations

argue that an increase in ontological overlap will decrease the understandability of the model

combination. The proposed reason is because ontological overlap between models decreases

146

the clarity of the model combination and increases the cognitive load. However, the findings

of Study 3 were equivocal. In the surface-level understanding of model combinations, the

ontological overlap between models helped users to better comprehend the elements presented

in the models. This finding can be interpreted by drawing on the learning preferences

hypothesis and surface-level understanding. Learning preferences hypotheses argue that

repetition can improve learning performance (Bromage & Mayer, 1986; Mayer, 1983).

Surface processing of information involves repeating the information without interpretation

and adding new elements (Offir et al., 2008). It appears that the existence of ontological

overlap between models would assist users to link the elements presented in the models.

On the other hand, in line with the predictions of the theory of faithful use of conceptual

model combinations, the findings of Study 3 reveal that ontological overlap decreased users’

performance in the problem-solving test. It appears that the problem-solving test requires a

deep-level understanding. Given that possibility, repetition would increase extraneous

cognitive load (Sweller, 1994; Sweller & Chandler, 1994) and human working memory has a

limited capacity (Mayer, 1989, 2005), high ontological overlap in a model combination will

greatly shrink the capacity of the human working memory to acquire information and

decrease its ability to generate deep-level understanding. These findings are detailed in

Chapter 6.

In summary, Studies 2 and 3 addressed the research opportunities highlighted in Study 1

and investigated the use of model combinations and examined the factors that define the best

combination of models. The findings of these studies corroborate the predictions of

ontological theories (Recker, 2014) and confirm the importance of pragmatic factors, as has

been noted in previous studies (e.g., Burton-Jones et al., 2009; Green et al., 2011; Recker et

al., 2010; Rosemann, 2006; Samuel et al., 2015). However, Study 2 also identified the

important interaction effects between context and script factors and proposed that sometimes

context influences the impact of script qualities and may override them. Experimental results

support these findings and indicate that the role of ontological overlap varies depending on

task settings. The findings of Study 2 and 3 are consistent with the framework proposed by

Wand and Weber (2002), and indicate that both the pragmatic context and semantic factors

explain the use of model combinations. In addition, the findings of Studies 2 and 3 also

correspond to the new framework proposed in Study 1. The findings provide empirical


support for the view that antecedents (i.e., context) can influence other antecedents (i.e.,

ontological qualities) and identified the role of these factors in the use of representation (i.e.,

combinations of models representing a domain) and the outcome (i.e., domain understanding).

7.2 IMPLICATIONS

7.2.1 Implications for Research

This thesis provides several implications for research. These implications are outlined in

each study and briefly summarised again here. First, Study 1 provided a review of research on

conceptual modelling in IS. Study 1 applied the framework proposed by Wand and Weber

(2002) to investigate the extent to which the research opportunities proposed by the

framework have been considered by the extant research on conceptual modelling. The

findings of the review highlighted the areas that require further research and indicated that

some of the research areas are entirely neglected in IS research. For example, further in-depth

investigations are required to understand the role of social agenda factors on conceptual

modelling. The study underlined the contributions of past research and can serve as a guide

for future research.

Second, Study 1 proposed new research opportunities and identified some path-breaking

studies, such as multidimensional modelling and systems flexibility (e.g., de Albuquerque &

Christ, 2015); or relevant aspects of model-driven architecture, such as security features (e.g.,

D'aubeterre et al., 2008). Study 1 also provided a new framework that draws the attention of

conceptual modelling research to unknown aspects of conceptual modelling (Recker, 2015);

emerging needs, such effective use of big data (Rai, 2017); and a new wave of modelling

paradigms, such as agile modelling or approaches when no script is involved (Lukyanenko &

Parsons, 2013). The proposed framework creates significant new opportunities for the

conceptual modelling community and substantially expands what counts as conceptual

modelling research.

Third, the second study explored the use of multiple conceptual models in practice.

While previous studies have indicated that IS professionals normally use multiple types of

models (Dobing & Parsons, 2008; Fettke, 2009), it was unknown how and why they are using

different types of models. Study 2 uncovered important semantic and pragmatic factors that

affect the extent and types of models IS professionals use in most of their systems analysis

148

and design tasks. These results extend the body of knowledge about conceptual modelling,

provide empirical insights, and also develop the theoretical foundations to explain why IS

professionals use multiple models.

Fourth, the study identified contextual factors other than the ontological factors as the

main determinants of using model combinations. Contextual factors not only have primary

effects on the combined use of conceptual models, there are also important interaction effects

between context and ontological factors. The results of the study confirm previous findings

that suggest the important role of contextual factors, such as organisational conventions or

tool supports (Recker et al., 2010), individual difference in understanding conceptual models

(Khatri & Vessey, 2016; Khatri et al., 2006), task objectives (Samuel et al., 2015), or different

stakeholder requirements (Topi & Ramesh, 2002). In addition, the results of Study 2 propose

that sematic and pragmatic factors interact in the use of model combinations. In particular, the

results of Study 2 indicate that contextual factors may dominate ontological factors. These

findings highlight the importance of considering both semantic factors and pragmatic factors

in further research about conceptual modelling (Bera et al., 2014; Burton-Jones et al., 2009).

Fifth, Study 3 tested the theory of faithful use of conceptual model combinations. This

study identified the role of completeness and overlap on users’ choice in selecting types of

models. These findings indicate that ontological completeness and overlap play a significant

role in model selection. However, users may choose not to select a model combination with

high ontological overlap, even if the combination provides them with a high complete

representation. These findings corroborate previous studies that have suggested that users

prefer to use representations that are easier to understand and compromised on completeness

of the representations (Samuel et al., 2015; zur Muehlen et al., 2007). Previous studies have

also indicated that the use of representations depends on the purpose of tasks and individual

differences (Figl & Recker, 2014). These findings indicate that the primacy of ontological

completeness or overlap may vary across different task settings and other pragmatic factors.

There are also future opportunities to evaluate how decisions about the selection of models

may connect to the performance of users across different tasks.

Sixth, while model users preferred a combination of models with low ontological

overlap, the results of the experiment revealed that ontological overlap improved the

performance of model users when they were dealing with a task that required a surface level


understanding. In contrast, when users were involved with a task that required a deep-level

understanding, ontological overlap decreased the performance of the users. These results

corroborate the equivocal arguments about the usefulness of ontological clarity on users’

performance. While most studies have argued that ontological clarity has a positive effect on

users’ performance (e.g., Milton et al., 2012; Parsons, 2011; Shanks et al., 2010), some

findings have challenged this argument (Allen & March, 2012; Bera et al., 2014; Bowen et

al., 2006), in addition, some studies have also identified that using different forms of

representation with redundant information improves users performance in domain

understanding (Gemino & Parker, 2009). Previous studies have demonstrated the interaction

effect between domain knowledge and ontological clarity (Bera et al., 2014; Burton-Jones &

Weber, 1999), the findings of the current research extend the research on ontological clarity

and identify the different roles of ontological clarity in different task settings. There would be

further opportunities to extend this stream of research, for instance, future studies could

examine a three-way interaction of domain knowledge, task settings, and ontological clarity.

7.2.2 Implications for Practice

Several practical implications flow from this research. First, the results indicate that the

role of ontological factors on the use of multiple models is not always straightforward. While

IS professionals prefer models with a high level of completeness and less overlapping

constructs, the correspondence between models may lead IS professionals to use partially

overlapping models. It would appear that, depending on the context, the correspondence

between models may sometimes bring forth desired qualities for users. This suggests a

practical direction for model users to find appropriate models to be able to integrate

information presented in different models and to also provide guidance for system designers

to avoid inconsistency in different representations of the domain.

Second, this research revealed that IS professionals prefer to use multiple models,

especially when dealing with large and complex domains. When the domain is large and

complex, IS professionals found it difficult to understand the entire domain through a single

model. The findings of the study also suggest that those models should be used in a

combination that meets different stakeholder requirements and fits the purpose of the task.

This provides practical guidance for IS professionals for identifying appropriate models to

use.

150

Third, the findings of this research provide practical guidance for decision makers to

invest in appropriate modelling tools and to define effective organisational conventions. The

findings indicate that organisational rules may lead IS professionals to use overlapping

models, which may lead to unnecessary complexity in the related tasks. Moreover, the results

indicate that IS professionals use the types of models in which they are supported by available

tools.

Chapter 8:Conclusions 151

Chapter 8: Conclusions

8.1 CONTRIBUTIONS

The central contribution of this thesis is that it provides results from an extensive

investigation into the combined use of conceptual models, and attempts to validate and extend

the theory of faithful use of conceptual model combinations. In addition to this central

contribution, this research brings forth a number of further contributions to IS research and

practice.

8.1.1 For research

First, the current research provides an overview of the current state of knowledge about

conceptual modelling. This research may assist researchers to understand which components

of conceptual modelling, for example, modelling grammars, methods, scripts, and context,

and what aspects of these components, as suggested by Wand and Weber (2002), have been

examined by current research efforts. The findings shed light on the aspects of conceptual

modelling that require further research. In addition, while the research framework proposed

by Wand and Weber (2002) certainly provides significant research directions, Study 1

proposed a new research framework to stimulate new pathways to research on conceptual

modelling to address the emerging practices and ongoing innovations in the IS field. This

research hopefully creates significant new opportunities for the conceptual modelling

community.

Second, this research extends the existing literature on conceptual modelling. It is the

first reported research, to the best of my knowledge, that provides a comprehensive and

empirical analysis into the use of multiple models. This research has explored the

circumstances that lead IS professionals to use multiple models and identified the factors that

affect the extent and type of models IS professionals select to use. The research also

developed a procedure and measurements to evaluate the faithful use of model combinations.

The results provide both theoretical logic and empirical evidence to explain how and why IS

professionals use model combinations.

152

Third, the research empirically demonstrates that ontological guidelines provide a

foundation and rationale to explain the combined use of conceptual models. The results

corroborate the principles of existing ontological theories (Green, 1996; Recker, 2014; Weber,

1997), and also reveal that in addition to ontological completeness and ontological overlap,

the existence of semantically related elements (i.e., correspondence) in pairs of models

influences the selection of the model combinations.

Fourth, the research reveals a strong prevalence of context, which has largely been

ignored in much of the theoretical and empirical work (Bera et al., 2014; Burton-Jones et al.,

2009; Recker, 2007). The research discloses the main effects of contextual factors on the

combined use of conceptual models, and it also indicates the important interaction effect

between ontological capabilities and contextual factors. The findings suggest that contextual

factors may override ontological principles. The research thereby sets a direction that can

assist fellow researchers to investigate the use of models in system analysis and design tasks

considering both model-related factors and contextual factors.

Fifth, this research provides the first empirical evaluation of the theory of faithful use of

conceptual model combinations. The results indicate that both ontological completeness and

ontological overlap appear to be major determinants of users’ model selection. However,

users made a trade-off between completeness and overlap. They selected models in

combinations that provided them with a less complete representation, but also less overlap. In

other words, the selection of models was governed by principles of clarity over completeness.

Sixth, the research indicates that different combinations of models are useful for

different types of tasks. The results indicate that a combination of models with high

ontological overlap can be helpful for tasks that require surface understanding, but can

decrease the performance of users who are dealing with a task that requires deep-level

understanding. These results confirmed the value of combining ontological theories with other

theories, such as cognitive theory of multimedia learning, in analysing the use of conceptual

models.

8.1.2 For practice

Given the importance of conceptual modelling in system analysis and design tasks, this

study provides practical contributions by: 1) clarifying the relevant and relatively important


factors regarding the combined use of conceptual models, and 2) providing an evaluated

theory that explains the faithful use of model combinations. These contributions are of

interest, not only for conceptual model users and relevant stakeholders, but also for

conceptual model designers and decision makers in selecting an appropriate approach and

improving overall performance of conceptual modelling processes.

While often different models are developed and used in most of the system analysis and

design tasks (Mussbacher et al., 2014), it is expected that the IS literature provides practical

guidance to assist practitioners to use appropriate types of models to improve the outcome of

the conceptual modelling processes. However, the current review of conceptual modelling

research showed that the IS literature provided little knowledge with regards to how to use

different type of models. The results of this thesis can therefore contribute to practice by

identifying the circumstances that lead IS professionals to use model combinations and which

attributes of models define the best possible combinations.

The results of this research indicate that IS professionals use models that are supported

by available tools and defined by organisational conventions. However, not all types of

models are helpful for all tasks. Different combinations of models with different levels of

overlap may lead to different levels of understanding. These results can assist model users to

select appropriate combinations of models depending on task purposes. The results also shed

light on decision making to (re-)design organisational conventions in a way that not only

provides system designers and analysts with appropriate types of models for a given task, the

results can also assist decision makers to make investment decisions about appropriate tools

to achieve better results from conceptual modelling practices. From a broader view, this

research assists model designers to develop multiple models to suit different task purposes.

With the expectation that any developed model will be used to generate an understanding of

the domain that is presented by models and based on the different role of ontological overlap

in different tasks, model designers may design representations with different levels of overlap

based on the purpose of an upcoming task that involves conceptual model use.

8.2 LIMITATIONS

Although this research makes contributions, like all research, it has limitations. The

choice of research methods brings forth the first set of limitations.

154

First, a literature review is limited to the approach used to identify the relevant studies

and the purpose of the review. Study 1 followed a systematic approach and provided a

descriptive review of the extant studies on conceptual modelling. The systematic literature

review in Study 1 was limited to the papers published in specific journals. The IS literature is

broad, and research on conceptual modelling appears in vast numbers of academic journals.

An alternative approach could be to conduct reviews using different methods (e.g., theoretical

review or meta-analysis) and consider a larger set of journals. However, in the current

research, this issue was mitigated by selecting journals considered by the academic

community to be appropriately representative of mainstream research in the field (i.e., AIS

basket of eight journals). Moreover, different databases were used to locate relevant studies,

and multiple coders analysed the articles to ensure the reliability of descriptive analysis and

evaluation of the papers.

Second, as detailed in Chapter 5, semi-structured interviews are linked to a number of

limitations, such as time limitations and the small sample of participants. An alternative

approach to investigate the use of model combinations, for example, could incorporate case

study methods. However, to increase external validity, semi-structured interviews were

conducted to collect data from professionals from different companies across various

countries. Moreover, to ensure the reliability and validity of the study, the guidelines

proposed by the leading literature were followed, and schemes and protocols were developed

to conduct high-quality interviews.

Third, there were limitations concerning the generalisability and validity of the

laboratory experiments (as summarised in Chapter 6). Experiments are artificial and may not

reflect reality. However, the primary objective of the experiment in this research was to

increase the internal validity and control potentially extraneous effects. To increase the

internal validity of the experiment, measurement items used in previous studies were adapted,

and experimental setting procedures designed in previous related studies were followed (e.g.,

Bera et al., 2014; Burton-Jones & Meso, 2008; Figl & Recker, 2014).

Fourth, limitations result from the scope of the study. This research primarily

investigated the use of UML models. UML was selected because it provides a variety of

diagrams and is popular for object-oriented system development (Batra, 2009). To increase

the generalisability of the findings, practitioners who used other modelling grammars (e.g.,


BPMN) were also interviewed; however, only UML models were used in the experiment to

control potential extraneous effects (e.g., different level of familiarity).

Fifth, the scope of this thesis is the individual level – that is, the individual user of

model combinations. Therefore, the findings are limited in terms of generalising to the group-

level or organisational-level model use. Fellow researchers can therefore consider multi-level

research when investigating the combined use of conceptual models.

8.3 FUTURE RESEARCH

In addition to the remaining research opportunities identified in Study 1 and the new

research opportunities proposed by the research framework proposed in Study 1, several

research opportunities exist to extend the current research and address the limitations of this

work.

First, Study 2 developed theoretical models and proposed theoretical logic to explain the

use of model combinations. Therefore, future research opportunities exist to use empirical

methods to operationalise and validate those theoretical conjectures proposed in Study 2.

Second, multiple conceptual models were not considered for a particular purpose or for

a specific system analysis and design approach. Further opportunities exist for future

researchers to use similar approaches to those proposed in this research to investigate the use

of model combinations for specific purposes (e.g., multidimensional database design) or in

different contexts (e.g., different system development approaches). For example, future

researchers could investigate how the use of model combinations can support agile modelling

(Ambler, 2002).

Third, the results of interviews and previous studies have indicated that IS professionals

normally integrate UML diagrams with other types of models, for example, BPMN or ERD

(Petre, 2013). Future studies could therefore use a setting similar to that proposed in this

research to replicate the study by using different types of models.

Fourth, the use of conceptual models is part of a task, and previous studies have

indicated that different task settings may influence how practitioners use conceptual models

(e.g., Figl & Recker, 2014). Future studies could therefore investigate the use of model

combinations for specific task settings, for example, query development (Bowen et al., 2009).

156

Fifth, previous studies have indicated the role of individual differences in the use of

conceptual models (e.g., Khatri & Vessey, 2016). Fellow researchers could therefore extend

the current experiment by using different domains (e.g., familiar and unfamiliar domains), or

different subjects as participants (e.g., novices versus experts). Future studies could also

extend the current experiment by using combinations of models with different levels of

completeness and overlap, for example, low, moderate, and high levels of completeness and

overlap.

Sixth, future studies could extend this study by investigating cognitive processes in

using model combinations. The results of the experiment show that ontological overlap

improves the models’ comprehension and apparently assists users to link the models.

However, high ontological overlap decreased the performance of users in the problem-solving

test. Future studies could therefore investigate how an increase in the level of ontological

overlap affects users’ information-seeking behaviour and how this behaviour varies based on

different task settings.

Finally, further studies could extend the scope of this thesis and investigate the

combined use of conceptual models at the group or the organisational level, or compare the

combined use of models in different levels, such as group-level versus the individual-level.

8.4 CONCLUSION

This research provides contributions for both research and practice. It has attempted to

provide both an empirical understanding of the combined use of conceptual models in practice

and the development of important extensions and contingencies to existing theory in

conceptual modelling research. This research identified substantive research opportunities that

can hopefully draw attention to further research on conceptual modelling and expand the

current state of knowledge with regards to conceptual modelling.

Bibliography 157

Bibliography

Ågerfalk, P. J., & Eriksson, O. (2004). Action-oriented conceptual modelling. European

Journal of Information Systems, 13(1), 80-92.

Aguirre-Urreta, M. I., & Marakas, G. M. (2008). Comparing conceptual modeling techniques:

a critical review of the EER vs. OO empirical literature. ACM SIGMIS Database,

39(2), 9-32.

Akhigbe, O., & Lessard, L. (2016). Situating requirements engineering methods within design

science research. In Breakthroughs and Emerging Insights from Ongoing Design

Science Projects: Research-in-progress papers and poster presentations from the 11th

International Conference on Design Science Research in Information Systems and

Technology (DESRIST) (pp. 35-42). Canada: DESRIST 2016.

Allen, G. N., & March, S. T. (2003). Modeling temporal dynamics for business systems.

Journal of Database Management, 14(3), 21-36.

Allen, G. N., & March, S. T. (2006). The effects of state-based and event-based data

representation on user performance in query formulation tasks. MIS Quarterly, 30(2),

269-290.

Allen, G. N., & March, S. T. (2012). A Research Note on Representing Part-Whole Relations

in Conceptual Modeling. MIS Quarterly, 36(3), 945-964.

Ambler, S. (2002). Agile modeling: effective practices for extreme programming and the

unified process. New York: John Wiley

An, Y., Hu, X., & Song, I.-Y. (2010). Maintaining mappings between conceptual models and

relational schemas. Journal of Database Management, 21(3), 36-68.

Anglim, B., Milton, S. K., Rajapakse, J., & Weber, R. (2009, 8-10 June). Current trends and

future directions in the practice of high-level data modeling: An empirical study.

Paper presented at the European Conference of Information Systems, Verona, Italy.

Arisholm, E., & Sjøberg, D. I. (2004). Evaluating the effect of a delegated versus centralized

control style on the maintainability of object-oriented software. IEEE Transactions on

Software Engineering, 30(8), 521-534.

Athenikos, S. J., & Song, I.-Y. (2013). CAM: A Conceptual Modeling Framework based on

the Analysis of Entity Classes and Association Types. Journal of Database

Management, 24(4), 51-80.

Atkins, D., Best, D., Briss, P. A., Eccles, M., Falck-Ytter, Y., Flottorp, S., . . . Henry, D.

(2004). Grading quality of evidence and strength of recommendations. BMJ (Clinical

Research ed.), 328(7454), 1490-1490.

Ausubel, D. P. (1968). Educational psychology: A cognitive view. New York: Holt, Rinehart

and Winston.

Avison, D., & Wood-Harper, A. T. (1986). Multiview—an exploration in information systems

development. Australian Computer Journal, 18(4), 174-179.

Bandara, W., Furtmuller, E., Gorbacheva, E., Miskon, S., & Beekhuyzen, J. (2015).

Achieving rigour in literature reviews: insights from qualitative data analysis and tool-

support. Communications of the Association for Information Systems, 37(8), 154-204.

Bandara, W., Gable, G. G., & Rosemann, M. (2005). Factors and measures of business

process modelling: model building through a multiple case study. European Journal of

Information Systems, 14(4), 347-360.

158 Bibliography

Batra, D. (2008). Conceptual data modeling patterns: Representation and validation. Journal

of Database Management, 16(2), 84-106.

Batra, D. (2009). Unified modeling language (UML) topics: cognitive issues in UML

research. Journal of Database Management, 20(1), I-X.

Batra, D. (2012). An Event-Oriented Data Modeling Technique Based on the Cognitive

Semantics Theory. Journal of Database Management, 23(4), 52-74.

Batra, D., & Davis, J. G. (1992). Conceptual Data Modelling in Database Design: Similarities

and Differences between Expert and Novice Designers. International Journal of Man-

Machine Studies, 37(1), 83-101.

Benbasat, I., Goldstein, D. K., & Mead, M. (1987). The case research strategy in studies of

information systems. MIS Quarterly, 11(3), 369-386.

Bera, P. (2012). Analyzing the Cognitive Difficulties for Developing and Using UML Class

Diagrams for Domain Understanding. Journal of Database Management, 23(3), 1-29.

Bera, P., Burton-Jones, A., & Wand, Y. (2011). Guidelines for designing visual ontologies to

support knowledge identification. MIS Quarterly, 35(4), 883-908.

Bera, P., Burton-Jones, A., & Wand, Y. (2014). How Semantics and Pragmatics Interact in

Understanding Conceptual Models. Information Systems Research, 25(2), 401-419.

Bera, P., & Evermann, J. (2014). Guidelines for using UML association classes and their

effect on domain understanding in requirements engineering. Requirements

Engineering, 19(1), 63-80.

Bera, P., Krasnoperova, A., & Wand, Y. (2010). Using ontology languages for conceptual

modeling. Journal of Database Management, 21(1), 1-28.

Bhattacherjee, A. (2012). Social science research: principles, methods, and practices. North

Charleston: CreateSpace Independent Publishing Platform.

Bodart, F., Patel, A., Sim, M., & Weber, R. (2001). Should optional properties be used in

conceptual modelling? A theory and three empirical tests. Information Systems

Research, 12(4), 384-405.

Boell, S. K., & Cecez-Kecmanovic, D. (2015). On being ‘systematic’ in literature reviews in

IS. Journal of Information Technology, 30(2), 161-173. doi:10.1057/jit.2014.26

Booch, G. (1986). Object-oriented development. IEEE transactions on Software Engineering,

12(2), 211-221.

Booch, G., Rumbaugh, J., & Jacobson, I. (2005). The Unified Modeling Language User Guide

(2nd ed.). Upper Saddle River, NJ: Addison Wesley Longman Inc.

Booch, G., Rumbaugh, J., & Jacobson, I. (2010). The Unified Modeling Language User

Guide, 1999. MA: Addison-Wesley.

Bowen, P. L., O'Farrell, R. A., & Rohde, F. H. (2006). Analysis of competing data structures:

Does ontological clarity produce better end user query performance. Journal of the

Association for Information Systems, 7(8), 514-544.

Bowen, P. L., O'Farrell, R. A., & Rohde, F. H. (2009). An empirical investigation of end-user

query development: the effects of improved model expressiveness vs. complexity.

Information Systems Research, 20(4), 565-584.

Bromage, B. K., & Mayer, R. E. (1986). Quantitative and qualitative effects of repetition on

learning from technical text. Journal of Educational Psychology, 78(4), 271-278.

Browne, G. J., & Parsons, J. (2012). More Enduring Questions in Cognitive IS Research.

Journal of the Association for Information Systems, 13(12), 1000-1011.

Bibliography 159

Bubenko, J. A. (1979). On the Role of'Understanding Models' in conceptual schema design.

In A. l. Furtado & H. L. Morgan (Eds.), 5th International Conference on Very Large

Data Bases (pp. 129-139). Rio de Janeiro, Brazil: IEEE.

Bunge, M. (1977). Treatise on basic philosophy: Ontology I: the furniture of the world (Vol.

3). Dordrecht, The Netherlands: Kluwer Academic Publishers.

Burton-Jones, A., & Grange, C. (2013). From use to effective use: A representation theory

perspective. Information Systems Research, 24(3), 632-658.

Burton-Jones, A., & Meso, P. N. (2006). Conceptualizing systems for understanding: an

empirical test of decomposition principles in object-oriented analysis. Information

Systems Research, 17(1), 38-60.

Burton-Jones, A., & Meso, P. N. (2008). The effects of decomposition quality and multiple

forms of information on novices' understanding of a domain from a conceptual model.

Journal of the Association for Information Systems, 9(12), 748-802.

Burton-Jones, A., Wand, Y., & Weber, R. (2009). Guidelines for empirical evaluations of

conceptual modeling grammars. Journal of the Association for Information Systems,

10(6), 495-532.

Burton-Jones, A., & Weber, R. (1999). Understanding relationships with attributes in entity-

relationship diagrams. Paper presented at the Proceedings of the 20th International

Conference on Information Systems, Atlanta.

Burton-Jones, A., & Weber, R. (2014). Building conceptual modeling on the foundation of

ontology. In H. Topi & A. Tucker (Eds.), Computing handbook: information systems

and information technology (3rd ed., pp. 15.11-15.24). Boca Raton, FL, United States:

CRC Press.

Byström, K., & Järvelin, K. (1995). Task complexity affects information seeking and use.

Information Processing & Management, 31(2), 191-213.

Calder, B. J., Phillips, L. W., & Tybout, A. M. (1981). Designing research for application.

Journal of Consumer Research, 8(2), 197-207.

Campbell, D. J. (1988). Task complexity: A review and analysis. Academy of Management

Review, 13(1), 40-52.

Chan, H. C., Tan, C.-H., & Teo, H.-H. (2014). Data Modeling: An Ontological Perspective of

Pointers. Journal of Database Management, 25(4), 17-37.

Chan, H. C., Wei, K. K., & Siau, K. L. (1993). User-database interface: The effect of

abstraction levels on query performance. MIS Quarterly, 17(4), 441-464.

Chang, H. C. (2010). A new perspective on Twitter hashtag use: Diffusion of innovation

theory. Proceedings of the American Society for Information Science and Technology,

47(1), 1-4.

Checkland, P. B., & Scholes, J. (2000). Soft systems methodology in action: Including a 30-

year retrospective. BASINGSTOKE: Macmillan Press.

Chen, H., Chiang, R. H., & Storey, V. C. (2012). Business intelligence and analytics: From

big data to big impact. MIS Quarterly, 36(4), 1165-1188.

Chen, P. P.-S. (1976). The entity-relationship model—toward a unified view of data. ACM

Transactions on Database Systems (TODS), 1(1), 9-36.

Chen, P. P.-S. (2006). Suggested research directions for a new frontier–active conceptual

modeling. In D. W. Embley, A. Olivé, & S. Ram (Eds.), Proceedings of International

Conference on Conceptual Modeling (pp. 1-4). Heidelberg: Springer.

160 Bibliography

Chen, W., & Hirschheim, R. (2004). A paradigmatic and methodological examination of

information systems research from 1991 to 2001. Information Systems Journal, 14(3),

197-235.

Chisholm, R. M. (1996). A realistic theory of categories: An essay on ontology. Cambridge,

UK: Cambridge University Press.

Chua, C. E. H., Chiang, R. H., & Lim, E.-P. (2002). On conceptual micro-object modeling.


Chua, C. E. H., Storey, V. C., & Chiang, R. H. (2012). Knowledge representation: a

conceptual modeling approach. Journal of Database Management, 23(1), 1-30.

Cierniak, G., Scheiter, K., & Gerjets, P. (2009). Explaining the split-attention effect: Is the

reduction of extraneous cognitive load accompanied by an increase in germane

cognitive load? Computers in Human Behavior, 25(2), 315-324.

Clarke, R., Burton-Jones, A., & Weber, R. (2016). On the Ontological Quality and Logical

Quality of Conceptual-Modeling Grammars: The Need for a Dual Perspective.


Clegg, B., & Shaw, D. (2008). Using process‐oriented holonic (PrOH) modelling to increase

understanding of information systems. Information Systems Journal, 18(5), 447-477.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale,

NJ: Lawrence Earlbaum Associates.

Compeau, D., Higgins, C. A., & Huff, S. (1999). Social cognitive theory and individual

reactions to computing technology: A longitudinal study. MIS Quarterly, 23(2), 145-

158.

Compeau, D., Marcolin, B., Kelley, H., & Higgins, C. A. (2012). Research commentary—

Generalizability of information systems research using student subjects—A reflection

on our practices and recommendations for future research. Information Systems

Research, 23(4), 1093-1109.

Currim, F., & Ram, S. (2012). Modeling spatial and temporal set-based constraints during

conceptual database design. Information Systems Research, 23(1), 109-128.

D'aubeterre, F., Singh, R., & Iyer, L. (2008). Secure activity resource coordination: empirical

evidence of enhanced security awareness in designing secure business processes.

European Journal of Information Systems, 17(5), 528-542.

da Silva, A. R. (2015). Model-driven engineering: A survey supported by the unified

conceptual model. Computer Languages, Systems & Structures, 43, 139-155.

Dahanayake, A., Sol, H., & Stojanovic, Z. (2003). Methodology evaluation framework for

component-based system development. Journal of Database Management, 14(1), 1-

26.

Darke, P., & Shanks, G. (1997). User Viewpoint Modelling: Understanding and Representing

User Viewpoints During Requirements Definition. Information Systems Journal 7(3),

213-219.

Davern, M., Shaft, T., & Te'eni, D. (2012a). Cognition matters: Enduring questions in

cognitive IS research. Journal of the Association for Information Systems, 13(4), 273-

314.

Davern, M., Shaft, T., & Te'eni, D. (2012b). More Enduring Questions in Cognitive IS

Research: A Reply. Journal of the Association for Information Systems, 13(12), 1012-

1016.

Bibliography 161

Davies, I., Green, P., Rosemann, M., Indulska, M., & Gallo, S. (2006). How do practitioners

use conceptual modeling in practice? Data & Knowledge Engineering, 58(3), 358-

380.

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of

information technology. MIS Quarterly, 13(3), 319-340.

de Albuquerque, J. P., & Christ, M. (2015). The tension between business process modelling

and flexibility: Revealing multiple dimensions with a sociomaterial approach. The

Journal of Strategic Information Systems, 24(3), 189-202.

de Sousa Saraiva, J., & da Silva, A. R. (2010). Evaluation of mde tools from a metamodeling

perspective. Journal of Database Management, 19(4), 21-46.

Delfmann, P., Herwig, S., Lis, Ł., & Stein, A. (2015). Supporting distributed conceptual

modelling through naming conventions-a tool-based linguistic approach. Enterprise

Modelling and Information Systems Architectures, 4(2), 3-19.

Denning, P. J. (2003). Great principles of computing. Communications of the ACM, 46(11),

15-20.

Dietz, G., & Juhrisch, M. (2012). Negotiating language barriers–a methodology for cross-

organisational conceptual modelling. European Journal of Information Systems, 21(3),

229-254.

Dobing, B., & Parsons, J. (2008). Dimensions of UML diagram use: a survey of practitioners.


Domingos, P. (1999). The role of Occam's razor in knowledge discovery. Data Mining and

Knowledge Discovery, 3(4), 409-425.

Dominguez, E., Rubio, A. L., & Zapata, M. A. (2002). Dynamic semantics of UML state

machines: a metamodeling perspective. Journal of Database Management, 13(4), 20-

38.

Dori, D., & Goodman, M. (1996). From object-process analysis to object-process design.

Annals of Software Engineering, 2(1), 25-50.

Dreiling, A., Rosemann, M., van der Aalst, W. M. P., Heuser, L., & Schulz, K. (2006).

Model-based software configuration: patterns and languages. European Journal of


Dubé, L., & Paré, G. (2003). Rigor in information systems positivist case research: current

practices, trends, and recommendations. MIS Quarterly, 24(4), 597-636.

Dunn, C. L., Gerard, G. J., & Grabski, S. V. (2011). Diagrammatic Attention Management

and the Effect of Conceptual Model Structure on Cardinality Validation. Journal of

the Association for Information Systems, 12(8), 585-605.

Dussart, A., Aubert, B. A., & Patry, M. (2004). An evaluation of inter-organizational

workflow modelling formalisms. Journal of Database Management, 15(2), 74-104.

Dwivedi, Y. K., Wastell, D., Laumer, S., Henriksen, H. Z., Myers, M. D., Bunker, D., . . .

Srivastava, S. C. (2015). Research on information systems failures and successes:

Status update and future directions. Information Systems Frontiers, 17(1), 143-157.

Eisenhardt, K. M. (1989). Building theories from case study research. Academy of

Management Review, 14(4), 532-550.

Elliott, A. C., & Woodward, W. A. (2007). Statistical analysis quick reference guidebook:

With SPSS examples. Thousand Oaks: Sage.

Erickson, J., Lyytinen, K., & Siau, K. L. (2005). Agile modeling, agile software development,

and extreme programming: the state of research. Journal of Database Management,

16(4), 88-100.

162 Bibliography

Evermann, J. (2005). Towards a cognitive foundation for knowledge representation.

Information Systems Journal, 15(2), 147-178.

Evermann, J. (2008). Theories of meaning in schema matching: A review. Journal of

Database Management, 19(3), 55-82.

Evermann, J., & Wand, Y. (2001). Towards ontologically based semantics for UML

constructs. In Proceedings of International Conference on Conceptual Modeling (ER

2001) (pp. 354-367). Yokohama, Japan: Springer.

Evermann, J., & Wand, Y. (2005). Ontology based object-oriented domain modelling:

fundamental concepts. Requirements Engineering, 10(2), 146-160.

Evermann, J., & Wand, Y. (2006). Ontological modeling rules for UML: An empirical

assessment. The Journal of Computer Information Systems, 46(5), 14-19.

Evermann, J., & Wand, Y. (2009). Ontology based object-oriented domain modeling:

Representing behavior. Journal of Database Management, 20(1), 48-77.

Fernández-Medina, E., Trujillo, J. C., & Piattini, M. (2007). Model-driven multidimensional

modeling of secure data warehouses. European Journal of Information Systems, 16(4),

374-389.

Fettke, P. (2009). How conceptual modeling is used. Communications of the Association for


Fettke, P., & Loos, P. (2005). Ontological Analysis of Reference Models. In P. F. Green & M.

Rosemann (Eds.), Business Systems Analysis with Ontologies (pp. 56-81). Hershey,

Pennsylvania: Idea Group.

Fickinger, T., & Recker, J. (2013). Construct Redundancy In Process Modelling Grammars:

Improving The Explanatory Power Of Ontological Analysis. In Proceedings of the

21st European Conference on Information Systems (pp. 1-12). Utrecht, The

Netherlands: Association for Information Systems.

Figl, K. (2017). Comprehension of Procedural Visual Business Process Models–A Literature

Review. Business & Information Systems Engineering, 59(1), 41-67.

Figl, K., & Laue, R. (2015). Influence factors for local comprehensibility of process models.

International Journal of Human-Computer Studies, 82, 96-110.

Figl, K., Mendling, J., & Strembeck, M. (2013). The influence of notational deficiencies on

process model comprehension. Journal of the Association for Information Systems,

14(6), 312-338.

Figl, K., & Recker, J. (2014). Exploring cognitive style and task-specific preferences for

process representations. Requirements Engineering, 21(1), 63-85.

Figl, K., Recker, J., & Mendling, J. (2013). A study on the effects of routing symbol design

on process model comprehension. Decision Support Systems, 54(2), 1104-1118.

Fonseca, F., & Martin, J. (2007). Learning the differences between ontologies and conceptual

schemas through ontology-driven information systems. Journal of the Association for


Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable

variables and measurement error. Journal of Marketing Research, 18(1), 39-50.

Francis, J. J., Johnston, M., Robertson, C., Glidewell, L., Entwistle, V., Eccles, M. P., &

Grimshaw, J. M. (2010). What is an adequate sample size? Operationalising data

saturation for theory-based interview studies. Psychology and Health, 25(10), 1229-

1245.

Bibliography 163

Garrigós, I., Pardillo, J., Mazón, J.-N., Zubcoff, J., Trujillo, J. C., & Romero, R. (2012). A

Conceptual modeling personalization framework for OLAP. Journal of Database


Gehlert, A., & Esswein, W. (2005). Ontological Analysis Reconsidered - A Formal Approach.

In M.-A. Sicilia, S. Sánchez-Alonso, E. García-Barriocanal, & J. J. Cuadrado-Gallego

(Eds.), Ontology, Conceptualizations and Epistemology for Software and Systems

Engineering, Proceedings of the ONTOSE 2005 International Workshop Madrid,

Spain: University of Alcalá.

Geiger, J. (2010). Why Do We Model? Information Management 20(6), 32-33.

Gemino, A., & Parker, D. (2009). Use case diagrams in support of use case modeling:

Deriving understanding from the picture. Journal of Database Management, 20(1), 1-

24.

Gemino, A., & Wand, Y. (2003). Evaluating modeling techniques based on models of

learning. Communications of the ACM, 46(10), 79-84.

Gemino, A., & Wand, Y. (2004). A framework for empirical evaluation of conceptual

modeling techniques. Requirements Engineering, 9(4), 248-260.

Gemino, A., & Wand, Y. (2005). Complexity and clarity in conceptual modeling: comparison

of mandatory and optional properties. Data & Knowledge Engineering, 55(3), 301-

326.

Genero, M., Fernández-Saez, A. M., Nelson, H. J., Poels, G., & Piattini, M. (2011). Research

review: a systematic literature review on the quality of UML models. Journal of


Glass, R. L. (2013). The Queensland Health Payroll Debacle. Information Systems


Goodchild, M. F. (2012). Twenty years of progress: GIScience in 2010. Journal of Spatial

Information Science, 2010(1), 3-20.

Green, P. (1996). An ontological analysis of information systems analysis and design (ISAD)

grammars in upper case tools. (PhD), The University of Queensland, Brisbane,

Australia.

Green, P., & Rosemann, M. (1999). An Ontological Analysis of Integrated Process

Modelling. In M. Jarke & A. Oberweis (Eds.), Advanced Information Systems

Engineering: 11th International Conference, CAiSE"99 Heidelberg, Germany, June

14—18, 1999 Proceedings (pp. 225-240). Berlin, Heidelberg: Springer Berlin

Heidelberg.

Green, P., & Rosemann, M. (2004). Applying ontologies to business and systems modelling

techniques and perspectives: Lessons learned. Journal of Database Management,

15(2), 105-117.

Green, P., Rosemann, M., Indulska, M., & Manning, C. (2007). Candidate interoperability

standards: An ontological overlap analysis. Data & Knowledge Engineering, 62(2),

274-291.

Green, P., Rosemann, M., Indulska, M., & Recker, J. (2011). Complementary use of modeling

grammars. Scandinavian Journal of Information Systems, 23(1), 59-86.

Grossman, M., Aronson, J. E., & McCarthy, R. V. (2005). Does UML make the grade?

Insights from the software development community. Information and Software

Technology, 47(6), 383-397.

Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An

experiment with data saturation and variability. Field Methods, 18(1), 59-82.

164 Bibliography

Guizzardi, G. (2005). Ontological foundations for structural conceptual models (Vol. 015).

Enschede, The Netherlands: Telematica Instituut.

Hadar, I., & Soffer, P. (2006). Variations in conceptual modeling: classification and

ontological analysis. Journal of the Association for Information Systems, 7(8), 568-

592.

Halpin, T. A. (2002). Metaschemas for ER, ORM and UML data models: A comparison.


Henderson-Sellers, B. (2002). The use of subtypes and stereotypes in the UML model.


Hirschheim, R., Klein, H. K., & Lyytinen, K. (1994). Information Systems Development and

Data Modeling: Conceptual and Philosophical Foundations. . Cambridge,

Massachusetts: Cambridge University Press.

Hoffer, J., Ramesh, V., & Topi, H. (2012). Modern database management (11th ed.). Boston,

MA: Prentice Hall Press.

Hoyningen-Huene, P. (2013). Systematicity: The nature of science. Oxford, UK: Oxford

University Press.

Hutchinson, J., Whittle, J., Rouncefield, M., & Kristoffersen, S. (2011). Empirical Assessment

of MDE in Industry. Paper presented at the 33rd International Conference on Software

Engineering (ICSE), Honolulu, HI, USA.

Indulska, M., Recker, J., Rosemann, M., & Green, P. (2009). Business process modeling:

Current issues and future challenges. Paper presented at the Advanced Information

Systems Engineering, Heidelberg, Berlin.

Irwin, G., & Turk, D. (2005). An ontological analysis of use case modeling grammar. Journal

of the Association for Information Systems, 6(1), 1-36.

Jabbari Sabegh, M. A., Lukyanenko, R., Recker, J. C., Samuel, B., & Castellanos, A. (2017).

Conceptual modeling research in information systems: What we now know and what

we still do not know. Paper presented at the Symposium on Research in Systems

Analysis and Design (AIS SIGSAND 2017).

Jabbari Sabegh, M. A., & Recker, J. (2017). Combined Use of Conceptual Models in Practice:

An Exploratory Study. Journal of Database Management, 28(2), 56-88.

Jabbari Sabegh, M. A., Recker, J., & Green, P. (2016). Designing Experiments to Test the

Theory of Combined Ontological Coverage. Paper presented at the International

Conference on Information Systems, Dublin, Ireland

Joosten, S., & Purao, S. (2002). A rigorous approach for mapping workflows to object-

oriented is models. Journal of Database Management, 13(4), 1-19.

Kaur, K., & Rani, R. (2013, 6-9 Oct. 2013). Modeling and querying data in NoSQL

databases. Paper presented at the 2013 IEEE International Conference on Big Data.

Kent, W. (1978). Data and Reality: Basic assumptions in data processing reconsidered.

North-Holland, Amsterdam: North-Holland Pub. Co.

Khatri, V., & Vessey, I. (2016). Understanding the Role of IS and Application Domain

Knowledge on Conceptual Schema Problem Solving: A Verbal Protocol Study.

Journal of the Association for Information Systems, 17(12), 759 – 803

Khatri, V., Vessey, I., Ramesh, V., Clay, P., & Park, S.-J. (2006). Understanding conceptual

schemas: Exploring the role of application and IS domain knowledge. Information


Bibliography 165

Kim, H. M., Fox, M. S., & Sengupta, A. (2007). How to build enterprise data models to

achieve compliance to standards or regulatory requirements (and share data). Journal

of the Association for Information Systems, 8(2), 105-128.

Kim, J., Hahn, J., & Hahn, H. (2000). How do we understand a system with (so) many

diagrams? Cognitive integration processes in diagrammatic reasoning. Information


Kirschner, P. A. (2002). Cognitive load theory: Implications of cognitive load theory on the

design of learning. Learning and Instruction, 12(1), 1-10.

Kitchenham, B., Brereton, O. P., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009).

Systematic literature reviews in software engineering–a systematic literature review.

Information and Software Technology, 51(1), 7-15.

Koschmider, A., Song, M., & Reijers, H. A. (2010). Social software for business process

modeling. Journal of Information Technology, 25(3), 308-322.

Krishnan, M. S., Mukhopadhyay, T., & Kriebel, C. H. (2004). A decision model for software

maintenance. Information Systems Research, 15(4), 396-412.

Krogstie, J. (2001). Using a Semiotic Framework to Evaluate UML for the Development of

Models of High Quality. In Unified modeling language: Systems analysis, design and

development issues (pp. 89-106): IGI Global.

Krogstie, J., Sindre, G., & Jørgensen, H. (2006). Process models representing knowledge for

action: a revised quality framework. European Journal of Information Systems, 15(1),

91-102.

Kung, C. H., & Sølvberg, A. (1986). Activity Modeling and Behavior Modeling of

Information Systems. In T. W. Olle, H. G. Sol, & A. A. Verrijn-Stuart (Eds.),

Information Systems Design Methodologies: Improving the Practice (pp. 145-171).

Amsterdam: North-Holland.

La Rosa, M., Reijers, H. A., van der Aalst, W. M. P., Dijkman, R. M., Mendling, J., Dumas,

M., & GarcíA-BañUelos, L. (2011). APROMORE: An advanced process model

repository. Expert Systems with Applications, 38(6), 7029-7040.

Lange, C. F., Chaudron, M. R. V., & Muskens, J. (2006). In practice: UML software

architecture and design description. IEEE Software, 23(2), 40-46.

Larsen, T. J., Niederman, F., Limayem, M., & Chan, J. (2009). The role of modelling in

achieving information systems success: UML to the rescue? Information Systems

Journal, 19(1), 83-117.

Laurier, W., & Poels, G. (2012). Research note: ontology-based structuring of conceptual data

modeling patterns. Journal of Database Management, 23(3), 50-64.

Lee, A. S. (1991). Integrating positivist and interpretive approaches to organizational

research. Organization Science, 2(4), 342-365.

Lee, A. S., & Hubona, G. S. (2009). A scientific basis for rigor in information systems

research. MIS Quarterly, 33(2), 237-262.

Leonardi, P. M. (2011). When flexible routines meet flexible technologies: Affordance,

constraint, and the imbrication of human and material agencies. MIS Quarterly, 30(1),

147-167.

Liddle, S. W. (2011). Model-Driven Software Development. In D. W. Embley & B. Thalheim

(Eds.), Handbook of Conceptual Modeling: Theory, Practice, and Research

Challenges (pp. 17-54). Berlin, Heidelberg: Springer Berlin Heidelberg.

Liddle, S. W., & Embley, D. W. (2007). A Common Core for Active Conceptual Modeling

for Learning from Surprises. In P. P. Chen & L. Y. Wong (Eds.), Active Conceptual

166 Bibliography

Modeling of Learning: Next Generation Learning-Base System Development (pp. 47-

56). Berlin, Heidelberg: Springer Berlin Heidelberg.

Lincoln, Y. S., Lynham, S. A., & Guba, E. G. (2011). Paradigmatic controversies,

contradictions, and emerging confluences, revisited. In The Sage handbook of

qualitative research (4th ed., pp. 97-128). Thousand Oaks: Sage Publication.

Lindland, O. I., Sindre, G., & Solvberg, A. (1994). Understanding quality in conceptual

modeling. IEEE Software, 11(2), 42-49.

Liu, F., & Myers, M. D. (2011). An analysis of the AIS basket of top journals. Journal of

Systems and Information Technology, 13(1), 5-24.

Liu, L., Pu, C., & Ruiz, D. D. (2004). A systematic approach to flexible specification,

composition, and restructuring of workflow activities. Journal of Database


Loucopoulos, P., & Kadir, W. M. N. (2008). BROOD: business rules-driven object oriented

design. Journal of Database Management, 19(1), 41-73.

Lukyanenko, R., & Parsons, J. (2013). Is Traditional Conceptual Modeling Becoming

Obsolete? In W. Ng, V. C. Storey, & J. C. Trujillo (Eds.), Proceedings of 32th

International Conference Conceptual Modeling, ER 2013, Hong-Kong, China,

November 11-13, 2013. (pp. 61-73). Berlin, Heidelberg: Springer Berlin Heidelberg.

Lukyanenko, R., & Parsons, J. (2014). Using Field Experimentation to Understand

Information Quality in User-generated Content. Paper presented at the

CodeCon@MIT.

Lukyanenko, R., Parsons, J., & Samuel, B. M. (2015). Do We Need an Instance-Based

Conceptual Modeling Grammar? Paper presented at the Symposium on Research in

Systems Analysis and Design (AIS SIGSAND 2015).

Lukyanenko, R., Parsons, J., & Wiersma, Y. F. (2014). The IQ of the Crowd: Understanding

and Improving Information Quality in Structured User-Generated Content.


Lukyanenko, R., Wiersma, Y., Huber, B., Parsons, J., Wachinger, G., & Meldt, R. (2017).

Representing Crowd Knowledge: Guidelines for Conceptual Modeling of User-

generated Content. Journal of the Association for Information Systems, 18(4), 297-

339.

Lyytinen, K. (1987). Two views of information modeling. Information & Management, 12(1),

9-19.

Ma, Z. (2005). A conceptual design methodology for fuzzy relational databases. Journal of


Maes, A., & Poels, G. (2007). Evaluating quality of conceptual modelling scripts based on

user perceptions. Data & Knowledge Engineering, 63(3), 701-724.

Marcus, N., Cooper, M., & Sweller, J. (1996). Understanding instructions. Journal of

Educational Psychology, 88(1), 49-63.

Masri, K., Parker, D., & Gemino, A. (2008). Using iconic graphics in entity-relationship

diagrams: the impact on understanding. Journal of Database Management, 19(3), 22-

41.

Maxwell, J. A. (1992). Understanding and validity in qualitative research. Harvard

Educational Review, 62(3), 279-301.

Mayer, R. E. (1983). Can you repeat that? Qualitative effects of repetition and advance

organizers on learning from science prose. Journal of Educational Psychology, 75(1),

40-49.

Bibliography 167

Mayer, R. E. (1989). Models for understanding. Review of Educational Research, 59(1), 43-

64.

Mayer, R. E. (2005). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The

Cambridge Handbook of Multimedia Learning (pp. 31-48). New York: Cambridge

University Press.

Mayer, R. E. (2009). Multimedia learning (2nd ed.). United States of America: Cambridge

university press.

Mendling, J., Reijers, H. A., & Cardoso, J. (2007). What Makes Process Models

Understandable? In G. Alonso, P. Dadam, & M. Rosemann (Eds.), Business Process

Management: 5th International Conference, BPM 2007, Brisbane, Australia,

September 24-28, 2007. Proceedings (pp. 48-63). Berlin, Heidelberg: Springer Berlin

Heidelberg.

Mendling, J., Reijers, H. A., & Recker, J. (2010). Activity labeling in process modeling:

Empirical insights and recommendations. Information Systems, 35(4), 467-482.

Mendling, J., Reijers, H. A., & van der Aalst, W. M. P. (2010). Seven process modeling

guidelines (7PMG). Information and Software Technology, 52(2), 127-136.

Mendling, J., & Strembeck, M. (2008). Influence Factors of Understanding Business Process

Models. In W. Abramowicz & D. Fensel (Eds.), Business Information Systems: 11th

International Conference, BIS 2008, Innsbruck, Austria, May 5-7, 2008. Proceedings

(pp. 142-153). Berlin, Heidelberg: Springer Berlin Heidelberg.

Merschbrock, C., & Munkvold, B. E. (2012). A Research Review on Building Information

Modeling in Construction―An Area Ripe for IS Research. Communications of the


Mertens, D. M. (2014). Research and evaluation in education and psychology: Integrating

diversity with quantitative, qualitative, and mixed methods (4th ed.). Los Angeles:

Sage Publications.

Milton, S. K., & Kazmierczak, E. (2004). An ontology of data modelling languages: a study

using a common-sense realistic ontology. Journal of Database Management, 15(2),

19-38.

Milton, S. K., Rajapakse, J., & Weber, R. (2012). Ontological clarity, cognitive engagement,

and conceptual model quality evaluation: An experimental investigation. Journal of

the Association for Information Systems, 13(9), 657-693.

Mingers, J. (2001). Combining IS research methods: towards a pluralist methodology.


Mitra, S., Bagchi, A., & Bandyopadhyay, A. K. (2007). Design of a data model for social

network applications. Journal of Database Management, 18(4), 51-79.

Montero, S., Díaz, P., & Aedo, I. (2007). From requirements to implementations: a model-

driven approach for web development. European Journal of Information Systems,

16(4), 407-419.

Moody, D. L. (2000). Building links between IS research and professional practice:

improving the relevance and impact of IS research. Paper presented at the Proceedings

of the twenty first International Conference on Information systems.

Moody, D. L. (2005). Theoretical and practical issues in evaluating the quality of conceptual

models: current state and future directions. Data & Knowledge Engineering, 55(3),

243-276.

168 Bibliography

Moody, D. L. (2009). The “physics” of notations: toward a scientific basis for constructing

visual notations in software engineering. IEEE Transactions on Software Engineering,

35(6), 756-779.

Mussbacher, G., Amyot, D., Breu, R., Bruel, J.-M., Cheng, B. H. C., Collet, P., . . . Whittle, J.

(2014). The Relevance of Model-Driven Engineering Thirty Years from Now. In J.

Dingel, W. Schulte, I. Ramos, S. Abrahão, & E. Insfran (Eds.), Model-Driven

Engineering Languages and Systems: 17th International Conference, MODELS 2014,

Valencia, Spain, September 28 – October 3, 2014. Proceedings (pp. 183-200). Cham:

Springer International Publishing.

Myers, M. D., & Newman, M. (2007). The qualitative interview in IS research: Examining

the craft. Information and Organization, 17(1), 2-26.

Nesbit, J. C., & Adesope, O. O. (2006). Learning with concept and knowledge maps: A meta-

analysis. Review of Educational Research, 76(3), 413-448.

Neuman, W. L. (2006). Social research methods: Qualitative and quantitative approaches

(6th ed.). Boston: Pearson/Allyn and Bacon.

Oei, J. L. H., Van Hemmen, L. J. G. T., Falkenberg, E. D., & Brinkkemper, S. (1992). The

meta model hierarchy: a framework for information systems concepts and techniques.

Retrieved from The Netherlands:

Offir, B., Lev, Y., & Bezalel, R. (2008). Surface and deep learning processes in distance

education: Synchronous versus asynchronous systems. Computers & Education,

51(3), 1172-1183.

Olle, T. W., Sol, H. G., & Verrijn-Stuart, A. A. (1982). Information System Design

Methodologies: A Comparative Review: Proceedings of the IFIP WG 8.1 Working

Conference on Comparative Review of Information Systems Design Methodologies

North-Holland, Amsterdam, The Netherlands.

OMG. (2011a). Business Process Model and Notation (BPMN) Version 2.0. Retrieved from

http://www.omg.org/spec/BPMN/2.0

OMG. (2011b). Unified Modeling Language (UML), Version 2.4.1. Retrieved from

http://www.omg.org/spec/UML/2.4.1/

Opdahl, A. L., & Henderson-Sellers, B. (2002). Ontological evaluation of the UML using the

Bunge–Wand–Weber model. Software and Systems Modeling, 1(1), 43-67.

Opdahl, A. L., & Henderson-Sellers, B. (2004). A template for defining enterprise modelling

constructs. Journal of Database Management, 15(2), 39-73.

Orlikowski, W. J. (1993). CASE tools as organizational change: Investigating incremental

and radical changes in systems development. MIS Quarterly, 17(3), 309-340.

Paas, F. G., Tuovinen, J. E., Tabbers, H., & Van Gerven, P. W. (2003). Cognitive load

measurement as a means to advance cognitive load theory. Educational Psychologist,

38(1), 63-71.

Pallant, J. (2007). SPSS survival manual: a step by step guide to data analysis using SPSS for

Windows (version 15) (3rd ed.). England: McGraw-Hill Education.

Palvia, P., Jacks, T., & Brown, W. (2015). Critical issues in EHR implementation: provider

and vendor perspectives. Communications of the Association for Information Systems,

36(36), 706-726.

Palvia, P., Leary, D., Mao, E., Midha, V., Pinjani, P., & Salam, A. (2004). Research

methodologies in MIS: an update. Communications of the Association for Information

Systems, 14(24), 526-542.

http://www.omg.org/spec/UML/2.4.1/

Bibliography 169

Pardillo, J., Mazón, J.-N., & Trujillo, J. C. (2011). An MDA Approach and QVT

Transformations for the Integrated Development of Goal-Oriented Data Warehouses

and Data Marts. Journal of Database Management, 22(1), 43-68.

Paré, G., Trudel, M.-C., Jaana, M., & Kitsiou, S. (2015). Synthesizing information systems

knowledge: A typology of literature reviews. Information & Management, 52(2), 183-

199.

Parsons, J. (2002). Effects of local versus global schema diagrams on verification and

communication in conceptual data modeling. Journal of Management Information

Systems, 19(3), 155-183.

Parsons, J. (2011). An experimental study of the effects of representing property precedence

on the comprehension of conceptual schemas. Journal of the Association for


Parsons, J., & Cole, L. (2005). What do the pictures mean? Guidelines for experimental

evaluation of representation fidelity in diagrammatical conceptual modeling

techniques. Data & Knowledge Engineering, 55(3), 327-342.

Parsons, J., & Wand, Y. (2008). Using cognitive principles to guide classification in

information systems modeling. MIS Quarterly, 30(4), 839-868.

Parsons, J., & Wand, Y. (2013). Extending Classification Principles from Information

Modeling to Other Disciplines. Journal of the Association for Information Systems,

14(3), 245-273.

Parsons, J., & Wand, Y. (2014). A foundation for open information environments. Paper

presented at the European Conference on Information Systems, Tel Aviv, Israel.

Petre, M. (2013). UML in practice. Paper presented at the Proceedings of the 2013

International Conference on Software Engineering, San Francisco, California.

Petri, C. A. (1962). Fundamentals of a Theory of Asynchronous Information Flow. In C. M.

Popplewell (Ed.), IFIP Congress 62: Information Processing (pp. 386-390). Munich,

Germany: North-Holland.

Pettit, R. G. (2008). Increasing Confidence in Concurrent Software through Architectural

Analysis. In F. Kordon & T. Vardanega (Eds.), Reliable Software Technologies – Ada-

Europe 2008: 13th Ada-Europe International Conference on Reliable Software

Technologies, Venice, Italy, June 16-20, 2008. (pp. 199-210). Berlin, Heidelberg:

Springer.

Pettit, R. G., & Street, J. A. (2005). Lessons Learned Applying UML in the Design of Mission

Critical Software. In N. Jardim Nunes, B. Selic, A. Rodrigues da Silva, & A. Toval

Alvarez (Eds.), UML Modeling Languages and Applications: < <UML> > 2004

Satellite Activities, Lisbon, Portugal, October 11-15, 2004, Revised Selected Papers

(pp. 129-137). Berlin, Heidelberg: Springer.

Poels, G. (2011). Understanding business domain models: The effect of recognizing resource-

event-agent conceptual modeling structures. Journal of Database Management, 22(1),

69-101.

Poels, G., Maes, A., Gailly, F., & Paemeleire, R. (2011). The pragmatic quality of Resources‐Events‐Agents diagrams: an experimental evaluation. Information Systems Journal,

21(1), 63-89.

Poppe, E., Brown, R., Recker, J., Johnson, D., & Vanderfeesten, I. (2017). Design and

Evaluation of Virtual Environments Mechanisms to Support Remote Collaboration on

Complex Process Diagrams. Information Systems, 66, 59-81.

170 Bibliography

Power, D. (2011). Data modeling: The lingua franca of a successful MDM effort. Information


Purao, S., Storey, V. C., & Han, T. (2003). Improving analysis pattern reuse in conceptual

design: Augmenting automated processes with supervised learning. Information


Rai, A. (2017). Diversity of design science contributions. MIS Quarterly, 41(11), iii–xviii.

Ramesh, B., Cao, L., & Baskerville, R. (2010). Agile requirements engineering practices and

challenges: an empirical study. Information Systems Journal, 20(5), 449-480.

Ramesh, V., & Browne, G. J. (1999). Expressing casual relationships in conceptual database

schemas. Journal of Systems and Software, 4, 225-232.

Recker, J. (2007). A socio-pragmatic constructionist framework for understanding quality in

process modelling. Australasian Journal of Information Systems, 14(2), 43-63.

Recker, J. (2010a). Continued use of process modeling grammars: the impact of individual

difference factors. European Journal of Information Systems, 19(1), 76-92.

Recker, J. (2010b). Explaining usage of process modeling grammars: Comparing three

theoretical models in the study of two grammars. Information & Management, 47(5),

316-324.

Recker, J. (2012). “Modeling with tools is easier, believe me”—The effects of tool

functionality on modeling grammar usage beliefs. Information Systems, 37(3), 213-

226.

Recker, J. (2013). Empirical investigation of the usefulness of Gateway constructs in process

models. European Journal of Information Systems, 22(6), 673-689.

Recker, J. (2014). Theorizing the Use of Multiple Conceptual Models in Combination Paper

presented at the 25th Australasian Conference on Information Systems 8th-10th Dec

2014, , Auckland New Zealand.

Recker, J. (2015). Research on conceptual modelling: less known knowns and more unknown

unknowns, please. Paper presented at the 11th Asia-Pacific Conference on Conceptual

Modelling, Sydney, Australia.

Recker, J., & Dreiling, A. (2011). The effects of content presentation format and user

characteristics on novice developers’ understanding of process models.

Communications of the Association for Information Systems, 28(6), 65-84.

Recker, J., Indulska, M., Rosemann, M., & Green, P. (2010). The ontological deficiencies of

process modeling in practice. European Journal of Information Systems, 19(5), 501-

525.

Recker, J., Mendling, J., & Hahn, C. (2013). How collaborative technology supports cognitive

processes in collaborative process modeling: A capabilities-gains-outcome model.


Recker, J., Reijers, H. A., & van de Wouw, S. G. (2014). Process model comprehension: the

effects of cognitive abilities, learning style, and strategy. Communications of the


Recker, J., & Rosemann, M. (2010a). A measurement instrument for process modeling

research: development, test and procedural model. Scandinavian Journal of


Recker, J., & Rosemann, M. (2010b). The measurement of perceived ontological deficiencies

of conceptual modeling grammars. Data & Knowledge Engineering, 69(5), 516-532.

Recker, J., Rosemann, M., Green, P., & Indulska, M. (2011). Do ontological deficiencies in

modeling grammars matter? MIS Quarterly, 35(1), 57-79.

Bibliography 171

Recker, J., Rosemann, M., Indulska, M., & Green, P. (2009). Business process modeling-a

comparative analysis. Journal of the Association for Information Systems, 10(4), 333-

363.

Reijers, H. A., Freytag, T., Mendling, J., & Eckleder, A. (2011). Syntax highlighting in

business process models. Decision Support Systems, 51(3), 339-349.

Reinhartz-Berger, I., & Sturm, A. (2008). Enhancing UML models: a domain analysis

approach. Journal of Database Management, 19(1), 74-94.

Rittgen, P. (2006). A language-mapping approach to action-oriented development of

information systems. European Journal of Information Systems, 15(1), 70-81.

Rosemann, M. (2006). Potential pitfalls of process modeling: part A. Business Process

Management Journal, 12(2), 249-254.

Rosemann, M., & Green, P. (2002). Developing a meta model for the Bunge–Wand–Weber

ontological constructs. Information Systems, 27(2), 75-91.

Rosemann, M., Green, P., & Indulska, M. (2004). A reference methodology for conducting

ontological analyses. In H. Lu, W. Chu, P. Atzeni, S. Zhou, & T. W. Ling (Eds.),

Conceptual Modeling–ER 2004 (pp. 110-121). Shanghai, China: Springer.

Rosemann, M., Recker, J., Green, P., & Indulska, M. (2009). Using ontology for the

representational analysis of process modelling techniques. International Journal of

Business Process Integration and Management, 4(4), 251-265.

Rowe, F. (2014). What literature review is not: diversity, boundaries and recommendations.

European Journal of Information Systems, 23(3), 241-255.

Rumbaugh, J., Jacobson, I., & Booch, G. (2004). The Unified Modeling Language Reference

Manual Massachusetts: Addison-Wesley.

Saghafi, A., & Wand, Y. (2014). Do Ontological Guidelines Improve Understandability of

Conceptual Models? A Meta-analysis of Empirical Work. Paper presented at the 47th

Hawaii International Conference on System Sciences (HICSS), 2014 Hawaii.

Samuel, B. M., Watkins III, L. A., Ehle, A., & Khatri, V. (2015). Customizing the

Representation Capabilities of Process Models: Understanding the Effects of

Perceived Modeling Impediments. IEEE Transactions on Software Engineering,

41(1), 19-39.

Scheer, A.-W., & Nüttgens, M. (2000). ARIS Architecture and Reference Models for

Business Process Management. In W. M. P. van der Aalst, J. Desel, & A. Oberweis

(Eds.), Business Process Management: Models, Techniques, and Empirical Studies

(pp. 376-389). Berlin, Heidelberg: Springer

Sein, M. K., Henfridsson, O., Purao, S., Rossi, M., & Lindgren, R. (2011). Action Design

Research. MIS Quarterly, 35(1), 37-56.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-

experimental designs for generalized causal inference. Boston: Houghton Mifflin.

Shanks, G. (1997). Conceptual Data Modelling: an empirical study of expert and novice data

modellers. Australasian Journal of Information Systems, 4(2), 63-73.

Shanks, G. (2002). Guidelines for conducting positivist case study research in information

systems. Australasian Journal of Information Systems, 10(1), 76-85.

Shanks, G., Moody, D., Nuredini, J., Tobin, D., & Weber, R. (2010). Representing classes of

things and properties in general in conceptual modelling: An empirical evaluation.


172 Bibliography

Shanks, G., Tansley, E., Nuredini, J., Tobin, D., & Weber, R. (2008). Representing part-

whole relations in conceptual modeling: an empirical evaluation. MIS Quarterly,

32(3), 553-573.

Shanks, G., Tansley, E., & Weber, R. (2004). Representing composites in conceptual

modeling. Communications of the ACM, 47(7), 77-80.

Shanks, G., & Weber, R. (2012). THE HOLE IN THE WHOLE: AResponse TO ALLEN

AND MARCH. MIS Quarterly, 36(3), 965-980.

Shiu, H., & Fong, J. (2008). Reverse engineering from an XML document into an extended

DTD graph. Journal of Database Management, 19(4), 62-80.

Siau, K. L. (2004). Informational and computational equivalence in comparing information

modeling methods. Journal of Database Management, 15(1), 73-86.

Siau, K. L. (2010). An Analysis of Unified Modeling Language (UML) Graphical Constructs

Based on BWW Ontology. Journal of Database Management, 21(1), i-viii.

Siau, K. L., Erickson, J., & Lee, L. Y. (2005). Theoretical vs. practical complexity: The case

of UML. Journal of Database Management, 16(3), 40-57.

Siau, K. L., & Lee, L. Y. (2004). Are use case and class diagrams complementary in

requirements analysis? An experimental study on use case and class diagrams in

UML. Requirements Engineering, 9(4), 229-237.

Siau, K. L., & Rossi, M. (2011). Evaluation techniques for systems analysis and design

modelling methods–a review and comparative analysis. Information Systems Journal,

21(3), 249-268.

Siau, K. L., & Tan, X. (2005). Improving the quality of conceptual modeling using cognitive

mapping techniques. Data & Knowledge Engineering, 55(3), 343-365.

Silverman, D. (2006). Interpreting qualitative data: Methods for analyzing talk, text and

interaction (3rd ed.). Thousand Oaks, London: Sage Publications.

Soffer, P. (2005). Refinement equivalence in model-based reuse: Overcoming differences in

abstraction level. Journal of Database Management, 16(3), 21-39.

Soffer, P., & Hadar, I. (2007). Applying ontology-based rules to conceptual modeling: a

reflection on modeling decision making. European Journal of Information Systems,

16(5), 599-611.

Soffer, P., & Kaner, M. (2011). Complementing business process verification by validity

analysis: A theoretical and empirical evaluation. Journal of Database Management,

22(3), 1-23.

Soffer, P., Kaner, M., & Wand, Y. (2010). Assigning Ontological Meaning to Workflow Nets.


Soffer, P., Wand, Y., & Kaner, M. (2015). Conceptualizing Routing Decisions in Business

Processes: Theoretical Analysis and Empirical Testing. Journal of the Association for


Sowa, J. F. (2000). Knowledge representation: logical, philosophical, and computational

foundations. London: Brooks/Cole, Pacific Grove.

Storey, V. C., Goldstein, R. C., & Ding, J. (2002). Common sense reasoning in automated

database design: An empirical test. Journal of Database Management, 13(1), 3-14.

Storey, V. C., & Song, I.-Y. (2017). Big data technologies and management: What conceptual

modeling can do. Data & Knowledge Engineering, 108, 50-67.

Storey, V. C., Trujillo, J. C., & Liddle, S. W. (2015). Research on conceptual modeling:

Themes, topics, and introduction to the special issue. Data & Knowledge Engineering,

98, 1-7.

Bibliography 173

Straub, D., Boudreau, M.-C., & Gefen, D. (2004). Validation guidelines for IS positivist

research. Communications of the Association for Information Systems, 13(1), 380-427.

Sun, S. X., Zhao, J. L., Nunamaker, J. F., & Sheng, O. R. L. (2006). Formulating the data-

flow perspective for business process management. Information Systems Research,

17(4), 374-391.

Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design.

Learning and Instruction, 4(4), 295-312.

Sweller, J., & Chandler, P. (1994). Why some material is difficult to learn. Cognition and

Instruction, 12(3), 185-233.

Tabchnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Boston:

Pearson Education.

Thompson, J. D. (1967). Organizations in action: Social science bases of administrative

theory. New York: McGraw-Hill.

Topi, H., & Ramesh, V. (2002). Human factors research on data modeling: a review of prior

research, an extended framework and future research directions. Journal of Database


Trujillo, J. C., Luján-Mora, S., & Song, I.-Y. (2004). Applying UML and XML for designing

and interchanging information for data warehouses and OLAP applications. Journal of


Urquhart, C. (2013). Grounded theory for qualitative research: A practical guide. Thousand

Oaks, California: Sage Publication.

van der Aalst, W. M. P. (2011). Process Mining Discovery, Conformance and Enhancement

of Business Processes. Heidelberg, Germany: Springer.

van der Aalst, W. M. P., & ter Hofstede, A. H. M. (2005). YAWL: yet another workflow

language. Information Systems, 30(4), 245-275.

VanderMeer, D., & Dutta, K. (2009). Applying learner-centered design principles to UML

sequence diagrams. Journal of Database Management, 20(1), 25-47.

Venkatesh, V., Brown, S. A., & Bala, H. (2013). Bridging the qualitative-quantitative divide:

Guidelines for conducting mixed methods research in information systems. MIS

Quarterly, 37(1), 21-54.

Venkatesh, V., Brown, S. A., & Sullivan, Y. W. (2016). Guidelines for conducting mixed-

methods research: An extension and illustration. Journal of the Association for


Veres, C., & Mansson, G. (2005). Cognition and Modeling: Foundations for Research and

Practice. Journal of Information Technology Theory and Application, 7(1), 93-104.

Vergara, N. M., Linero, J. M. T., & Moreno, A. V. (2007). Model-driven component

adaptation in the context of Web Engineering. European Journal of Information

Systems, 16(4), 448-459.

Vessey, I., Ramesh, V., & Glass, R. L. (2002). Research in information systems: An empirical

study of diversity in the discipline and its journals. Journal of Management


Vidyasankar, K., & Vossen, G. (2013). Multi-level modeling of web service compositions

with transactional properties. Journal of Database Management, 22(2), 1-31.

Wagelaar, D., & Van Der Straeten, R. (2007). Platform ontologies for the model-driven

architecture. European Journal of Information Systems, 16(4), 362-373.

Walsham, G. (2006). Doing interpretive research. European Journal of Information Systems,

15(3), 320-330.

174 Bibliography

Wand, Y., & Weber, R. (1989). An ontological evaluation of systems analysis and design

methods. In E. D. Falkenberg & P. Lindgren (Eds.), Information Systems Concepts:

An In-depth Analysis: proceedings of the IFIP TC 8/WG 8.1 Working Conference on

Information System Concepts: an in-depth analysis Namur, Belgium, 18-20 October,

1989 (pp. 79-107). Amsterdam, The Netherlands: Elsevier.

Wand, Y., & Weber, R. (1990a). Mario Bunge's Ontology as a formal foundation for

information systems concepts. In P. Weingartner & G. J. W. Dorn (Eds.), Studies on

Mario Bunge's Treatise (pp. 123-149). Atlanta: Rodopi.

Wand, Y., & Weber, R. (1990b). An ontological model of an information system. IEEE

Transactions on Software Engineering, 16(11), 1282-1292.

Wand, Y., & Weber, R. (1993). On the ontological expressiveness of information systems

analysis and design grammars. Information Systems Journal, 3(4), 217-237.

Wand, Y., & Weber, R. (1995). On the deep structure of information systems. Information

Systems Journal, 5(3), 203-223.

Wand, Y., & Weber, R. (2002). Research commentary: information systems and conceptual

modeling—a research agenda. Information Systems Research, 13(4), 363-376.

Wand, Y., & Weber, R. (2004). Reflection: Ontology in information systems. Journal of

Database Management, 15(2), I-VI.

Wand, Y., Woo, C., & Wand, O. (2008). Role and Request Based Conceptual Modeling – A

Methodology and a CASE Tool. In Q. Li, S. Spaccapietra, E. Yu, & A. Olivé (Eds.),

Conceptual Modeling - ER 2008: 27th International Conference on Conceptual

Modeling, Barcelona, Spain, October 20-24, 2008. Proceedings (pp. 540-541). Berlin,

Heidelberg: Springer Berlin Heidelberg.

Weber, R. (1996). Are attributes entities? A study of database designers' memory structures.


Weber, R. (1997). Ontological foundations of information systems. Melbourne, Australia:

Coopers & Lybrand and the Accounting Association of Australia and New Zealand

Melbourne.

Weber, R. (2003). Conceptual modelling and ontology: Possibilities and pitfalls. Journal of


Weber, R., & Zhang, Y. (1996). An analytical evaluation of NIAM's grammar for conceptual

schema diagrams. Information Systems Journal, 6(2), 147-170.

Webster, J., & Watson, R. T. (2002). Analyzing the past to prepare for the future: Writing a

literature review. MIS Quarterly, 26(2), xiii-xxiii.

Whiteley, D. (2013). An Introduction to Information Systems London, UK: Palgrave

Macmillan.

Whittle, J., Hutchinson, J., & Rouncefield, M. (2014). The state of practice in model-driven

engineering. IEEE Software, 31(3), 79-85.

Wyssusek, B. (2006). On ontological foundations of conceptual modelling. Scandinavian

Journal of Information Systems, 18(1), 63-80.

Yin, R. K. (2009). Case study research: Design and methods (4th ed.). Thousand Okas,

California: Sage publications.

Zhang, H., Kishore, R., & Ramesh, R. (2007). Semantics of the MibML conceptual modeling

grammar: an ontological analysis using the Bunge-Wang-Weber framework. Journal

of Database Management, 18(1), 1-19.

Bibliography 175

Zhang, H., Liu, L., & Li, T. (2011). Designing IT systems according to environmental

settings: A strategic analysis framework. The Journal of Strategic Information

Systems, 20(1), 80-95.

Zhang, J. (1997). The Nature of External Representations in Problem Solving. Cognitive

Science, 21, 179-217.

zur Muehlen, M., & Indulska, M. (2010). Modeling languages for business processes and

business rules: A representational analysis. Information Systems, 35(4), 379-390.

zur Muehlen, M., & Recker, J. (2008). How much language is enough? Theoretical and

practical use of the business process modeling notation. In International Conference

on Advanced Information Systems Engineering (pp. 465-479). Berlin Heidelberg:

Springer.

zur Muehlen, M., Recker, J., & Indulska, M. (2007). Sometimes less is more: are process

modeling languages overly complex? Paper presented at the Eleventh International

IEEE EDOC Conference Workshop Annapolis, MD, USA.

176 Appendices

Appendices

Appendix A

Literature Review Process

Table A. 1. Search Results

Sea

rch

Ter

m:

Con

ceptu

al M

odel

ling

Sea

rch

Ter

m: “C

once

ptu

al M

odel

*”

Sea

rch

Ter

m:

Con

ceptu

al m

odel

ling g

ram

mar

Sea

rch

Ter

m: “o

nto

logy”

Sel

ecte

d P

aper

s

Dat

abas

e

Journals # # # # #

MIS Quarterly 451 100 18 84 6 AISeL

Information Systems Research 149 71 9 14 18 Informs

Journal of the association for information systems 316 71 23 72 18 AISeL

European journal of Information Systems 321 111 20 56 19 Springer

Journal of Management Information Systems 278 39 242 17 6 Jstor

Information Systems Journal 240 63 7 42 8 Wiley Online

Library

Journal of Information Technology 181 45 4 33 2 Springer

Journal of Strategic Information Systems 148 36 1 11 3 Informs

Journal of Database Management 122 74 16 61 53 Science

Direct

Appendices 177

Table A. 2 Summary of Coding Scheme

What is the objective of the paper?

Focus of Study

Which research opportunity/ies the study intended to address?

Grammar Which research opportunity

related to conceptual modelling

grammar addressed by the

study?

Script Which research opportunity


script addressed by the study?

Method

Which research opportunity


method addressed by the study?

Context

Which research opportunity


context addressed by the study?

Type of Grammar What type of conceptual modelling grammar studied in the paper?

Example: BPMN, UML, ER

Type of Study

Empirical Study Which research method or approach has been

used?

Case study

Action Research

Survey

Experiment

Interview

Panel

Delphi Study

Design Science

Non-empirical study Which research method or approach has

been used?

Literature Review

Commentary

Theoretical

Empirical Evidence

if the paper is an empirical study

What are the quality and quantity of the empirical evidence

(considering the type of sample)?

What are the measurement items and materials?

(e.g. material: questionnaire, interview protocol, case description,

interface, developed diagrams, etc.)

(e.g., measurement items: accuracy, time, and perceived ease of

use)

and availability of measurement items and materials

178 Appendices

Table A. 3. Papers Addressed each opportunities proposed by Wand and Weber (2002)

Res

earc

h F

ram

ework

Ele

men

t

Research Opportunities Reference

Co

nce

ptu

al-M

od

elli

ng

Gra

mm

ars

G1

Evaluating ontologies based on

empirical testing of their

predictions

G2

Evaluating grammars for

ontological expressiveness

(Irwin & Turk, 2005); (Recker et al., 2009);

(Recker et al., 2010); (Green & Rosemann, 2004);

(Milton & Kazmierczak, 2004); (Hong Zhang et al.,

2007); (Dussart, Aubert, & Patry, 2004);

G3

Assigning ontological meaning to

constructs of design grammars

and generating ontologically

motivated modelling rules

(Allen & March, 2006); (Rittgen, 2006); (Bera et

al., 2010); (Evermann & Wand, 2009); (Laurier &

Poels, 2012); (Opdahl & Henderson-Sellers, 2004);

(Soffer & Kaner, 2011); (Soffer et al., 2010);

G4

Resolving outstanding ontological problems that impact conceptual

modelling—e.g., nature of the

part-of relationship

(Shanks & Weber, 2012); (Shanks et al., 2008)

G5

Empirically testing predicted

strengths and weaknesses in new

and existing grammars based on

their ontological expressiveness

G6

Determining which combinations

of grammars best support users who undertake conceptual-

modelling work

G7

Empirically testing the predicted

implications of construct deficit

and overload in grammars

(Allen & March, 2012); (Recker et al., 2011);

(Shanks et al., 2008); (Bera et al., 2014); (Bowen et

al., 2009); (Bowen et al., 2006); (Milton et al.,

2012); (Parsons, 2011); (Recker et al., 2010);

(Shanks et al., 2010)

G.O

ther

(Clarke et al., 2016); (Burton-Jones et al., 2009);

(Figl, Mendling, et al., 2013); (Recker, 2010a);

(Dobing & Parsons, 2008); (VanderMeer & Dutta,

2009); (L. Liu, Pu, & Ruiz, 2004); (Dussart et al.,

2004)

Conce

ptu

al-

Model

ling

Met

hods

M1

Evaluating how well different

methods allow users to elicit and model critical domain knowledge

(Burton-Jones & Meso, 2006);(Purao et al., 2003);

(Dunn et al., 2011); (Soffer et al., 2015); (Recker, 2013); (Poels et al., 2011); (Siau & Rossi, 2011);

(Poels, 2011); (de Sousa Saraiva & da Silva, 2010)

Appendices 179

M2

Developing procedures to assist

users of a grammar in identifying

and classifying phenomena

according to the grammar's

constructs

(Parsons & Wand, 2008); (Currim & Ram,

2012);(Hadar & Soffer, 2006);(Parsons & Wand,

2013); (Ågerfalk & Eriksson, 2004); (Dietz &

Juhrisch, 2012); (Soffer & Hadar, 2007); (Clegg &

Shaw, 2008); (Evermann, 2005); (Athenikos &

Song, 2013); (Batra, 2012); (Bera et al., 2010);

(Chua, Chiang, & Lim, 2002); (Dominguez, Rubio,

& Zapata, 2002); (Loucopoulos & Kadir,

2008);(Ma, 2005); (Opdahl & Henderson-Sellers,

2004); (Reinhartz-Berger & Sturm, 2008); (Batra,

2012)

M3

Determining the beliefs and

values that underlie different methods and evaluating the

consequences of these beliefs and

values for practice

(Hadar & Soffer, 2006)

M.O

ther

(Clarke et al., 2016);(Lukyanenko & Parsons,

2014); (Rittgen, 2006);(Allen & March, 2003); (An

et al., 2010); (Batra, 2008); (Pardillo et al., 2011);

(Storey et al., 2002); (H. M. Kim, Fox, & Sengupta,

2007); (Joosten & Purao, 2002)

Co

nce

ptu

al-m

odel

lin

g S

crip

ts

S1

Evaluating competing scripts

generated via the same grammar

to describe some phenomenon

(Allen & March, 2012); (Allen & March, 2006);

(Parsons & Wand, 2008); (Shanks et al., 2008);

(Bera et al., 2014); (Bowen et al., 2009); (Burton-

Jones & Meso, 2006); (Bowen et al., 2006);

(Burton-Jones & Meso, 2008); (Dunn et al., 2011); (Milton et al., 2012); (Parsons, 2011); (Recker,

2013); (Parsons, 2002); (Poels et al., 2011); (Bera,

2012); (Masri et al., 2008); (Poels et al., 2011);

(Reinhartz-Berger & Sturm, 2008); (Shanks et al.,

2010); (Soffer & Kaner, 2011);

S2

Evaluating competing scripts

generated via different grammars

to describe the same phenomenon

(Khatri et al., 2006); (Figl, Mendling, et al., 2013);

(Chan, Tan, & Teo, 2014)

S3

Evaluating different combinations

of scripts to determine which

combination best supports the

task at hand

S4

Developing theory to predict and

understand how humans use

scripts to accomplish various tasks

(Burton-Jones & Meso, 2008); (Parsons & Wand,

2013); (Krogstie et al., 2006); (Bera, 2012);(Bera et

al., 2010); (Siau, 2004)

S.O

ther

(Sun, Zhao, Nunamaker, & Sheng, 2006); (Burton-

Jones & Meso, 2008); (Bandara et al.,

2005);(Gemino & Parker, 2009); (Evermann,

2008);(Genero, Fernández-Saez, Nelson, Poels, &

Piattini, 2011); (Halpin, 2002)

Conce

pt

ual

-

Model

li

ng

Conte

xt

Indiv

idu

al

dif

fere

n

ces

Tas

k

Soci

al

agen

da

CI1

Development of knowledge-based

tools to support conceptual

modelling

(Purao et al., 2003);(Koschmider et al., 2010);

(Loucopoulos & Kadir, 2008); (Pardillo et al.,

2011); (Storey et al., 2002);

180 Appendices

CI2

Predicting which cognitive and

personality variables bear on a

user's ability to undertake

conceptual- modelling work

(Allen & March, 2006); (Bera et al., 2014); (Khatri

et al., 2006); (Davern et al., 2012b); (Browne &

Parsons, 2012); (Figl, Mendling, et al., 2013);

(Recker, 2010a); (Recker et al., 2010); (Soffer &

Hadar, 2007); (Evermann, 2005); (Green &

Rosemann, 2004); (Masri et al., 2008); (Shanks et

al., 2010); (VanderMeer & Dutta, 2009);

CI3

Predicting and testing empirically

which social skills affect the

outcomes of conceptual

modelling tasks

(Topi & Ramesh, 2002)

CI.

Oth

er (Bandara et al., 2005)

CT

Evaluating the strengths and

weaknesses of conceptual

modelling grammars, methods,

and scripts in the context of

different tasks

(Khatri et al., 2006); (Recker, 2010a); (Recker et

al., 2010); (Green & Rosemann, 2004);

CT

.Oth

er (Bandara et al., 2005); (Recker et al., 2010); (Soffer

& Hadar, 2007); (Topi & Ramesh, 2002);

(VanderMeer & Dutta, 2009)

CS

1 Understanding which values and

beliefs underlie conceptual-

modelling work in practice

(Hadar & Soffer, 2006); (Recker et al., 2010);

(Larsen et al., 2009);

CS

2

Determining the costs and

benefits of adopting different

values and beliefs when

undertaking conceptual-modelling

work

CS

3

Articulating detailed conceptual-modelling procedures that are

congruent with different beliefs

and values

CS

4

Understanding how existing

conceptual modelling grammars

and methods facilitate conceptual-

modelling work under different

values and beliefs

CS

.Oth

er (Koschmider et al., 2010); (He Zhang et al., 2011)

Appendices 181

Oth

ers

(Fonseca & Martin, 2007); (Chua et al., 2012);

(Garrigós et al., 2012); (Trujillo et al., 2004); (de

Albuquerque & Christ, 2015); (Fernández-Medina

et al., 2007); (D'aubeterre et al., 2008); (Dreiling et

al., 2006); (Montero, Díaz, & Aedo, 2007);

(Vergara et al., 2007); (Wagelaar & Van Der

Straeten, 2007); (H. M. Kim et al., 2007);

(Henderson-Sellers, 2002); (Soffer, 2005); ;

(Vidyasankar & Vossen, 2013);(Dahanayake, Sol,

& Stojanovic, 2003); (Dussart et al., 2004); (Shiu &

Fong, 2008); (Wand & Weber, 2004); (Mitra,

Bagchi, & Bandyopadhyay, 2007); (Weber, 2003)

182 Appendices

Appendix B

Appendix: Semi-structured Interview Protocol

Commencing Interviews

<State appreciation for time taken to participate> <Introduce myself>

<Explain the purposes of interview>

<Check if Consent to audio record, before recording>

<Begin>

Explanations of terms:

1. Conceptual models: all kinds of graphical representation of the real world created using modeling

methods such as ER diagrams, Class diagrams, business process models, activity diagrams or others.

2. Domain: A set of things in the real-world that a model represents, for example, the process of reserving

a flight ticket, or the accounts department in an organization.

3. System: An organized, purposeful structure that consists of a set of procedures created to carry out

specific functions and activities, for example online booking of flight ticket. 4. Interpretation: reading models by humans to understand its content, e.g., how different components of

the domain/system work or influence one another such as processes, data, associations, relationships.

5. Combined use: using the models that you have access to them at the same time or together, e.g.

simultaneously or in parallel.

Part A: Demographic information (background and experience)

A1: what is your current role in “organization”? Can you describe your responsibilities and tasks in general?

___________________________________________________________________________

A2: Over your working life how many years have you been working in business?

___________________________________________________________________________ A3: Over your working life how many years have you been working in IT-related business?

___________________________________________________________________________

A4: Over your working life how many years have you been working in system analysis and design?

___________________________________________________________________________

A5: Over your working life how many year you have been involved in conceptual modelling?

___________________________________________________________________________

A6: How long have you used conceptual models?

If you never used (Exit interview)

___________________________________________________________________________

A7: The training you have received in conceptual modelling includes:

(Specify level/ type / duration/ other details) (When undertook) 1. Formal training courses

2. Internal training courses

3. University courses

4. On the Job / mentor

5. Learnt by myself (how?)

6. None.

How about specific model you are using? (UML)

__________________________________________________________________________________________

____________________________________________________________

A8: In general, your primary purpose of using conceptual models is:

1. To understand system processes 2. To understand requirements of the system/project

3. Detect possible mistakes/problems in system development processes

4. System planning/design

Appendices 183

5. Communication

6. Other, please specify

__________________________________________________________________________________________

______________________________________________________

A9: Generally, the organizational level of models you use is:

1. High level (general level model, less detailed) (GIVE EXAMPLES)

2. Medium level (medium detailed models)

3. Low level (very detailed models)

A10: In particular, do you use conceptual models for the following purposes? (Multiple answers possible) Please score the purpose of use of conceptual models

Purpose

Score

Never Rarely Sometimes Often Always

1 Database design and management

2 Business process documentation

3 Improvement of internal business processes

4 Software development

5 Improvement of collaborative business processes

6 Workflow management

7 Design of enterprise architecture

8 Change management

9 Knowledge management

10 End user training

11 Software configuration

12 Software selection

13 Certification/quality management

14 Human resource management

15 Activity-based costing

16 Auditing

17 Simulation

18 Schedule and manage development process

Other, please specify and score:

----------------------------------------------------------------------------------------------------------------

A11: How many members are in the team you are working with?

___________________________________________________________________________

A12: How do the members cooperate in the project that you are involved? (How models involve in this

cooperation?)

Everyone does specific part of the project

Each part of the project is done separately and finally combined together

The entire project is done collectively

A13: Describe the project in which you are using conceptual models

__________________________________________________________________________________________

__________________________________________________________________________________________

___________________________

Part B: The use of combinations of conceptual models

B1: How are you involved in conceptual modelling?

1. You just design them (Exit Interview)

2. You just use them 3. Both

184 Appendices

B2: What is your role in the project you are involved in?

__________________________________________________________________________________

B3: Is there any requirement in your organization to use models; for example, the organization requires you to

use specific types of models?

Yes, please specify

No, why not?

________________________________________________________________________

B4: Do you need multiple types of models?

Yes, how many?

No Go to Part D ________________________________________________________________________

B4: Are different types of conceptual models available for you to use in the project that you are involved in?

Yes, which ones?

No Go to Part D

B5: Do you use different types of models?

Yes, which ones?

No Go to Part D

B6: Why do you use more than one model? 1. There is no one model that presents all concepts need for complete representation of the domain

2. That combination of model is the standard set to be used within my organization

3. Other, please specify.

__________________________________________________________________________________

B7: What types of models do you use? (e.g. class diagrams)

__________________________________________________________________________________

B8: What leads you to use these types of models?

__________________________________________________________________________________

B9: What do these models represent?

__________________________________________________________________________________

B10: As a user, do the models completely specify the requirements and convey all relevant information about the domain?

__________________________________________________________________________________

B11: How do you deal with models that are difficult to understand?

__________________________________________________________________________________

B12: How do you deal with models that are incomplete?

__________________________________________________________________________________

B13: How do you deal with models that are not accurate?

__________________________________________________________________________________

B14: How do you get more information if you need it?

1. I look at additional models (process to next question)

2. I look at documents 3. Other (explain)

__________________________________________________________________________________

B15: How do you choose additional models? (What circumstances lead you to choose and what factors in the

models are important for decision)

__________________________________________________________________________________

Appendices 185

Part C: Familiarity with Conceptual Models

Which conceptual models are you familiar with? What is your level of familiarity (1 to 5)? (Multiple answers

possible), How frequently do you use models (1 to 7)? Number of models you have seen (#)?

Familiarity Frequency of Use

Number of Models

You have Seen

Models

Very Low

Intermediate

Mediate

Advance

Professional

Never

Rarely

Occasionally

Sometimes

Frequently

Usually

Every time

Numbers

e.g.: 5, 20, 100 ….

Class diagrams

Use case diagrams

Sequence diagrams

Activity Diagrams

State chart diagrams

Object diagrams

Component diagrams

Deployment diagrams

Package diagrams

Composite structure diagram

Profile diagram

Communication diagram

Interaction overview diagram

Timing diagram

ER Diagrams

Business Process Models (BPMN)

Event-driven Process Chain (EPC)

Data Flow Diagrams

Petri Nets

Workflow

Gantt charts

Flowchart

Application diagrams

Object-Role Models

Role activity diagrams

Role interaction diagrams

Integrated definition modelling

(IDEF)

Other, Please Specify:

C2. Is there anything else you want to tell me about how you use conceptual models for your work?

________________________________________________________________________

C3. What do you think are benefits from using conceptual models?

________________________________________________________________________

C4. What are the biggest challenges in using conceptual models?

________________________________________________________________________

Part D: Using a Single Conceptual Model

186 Appendices

D1: which model (type) are you using?

________________________________________________________________________

D2: Why do you use a single conceptual model?

I use only one model because:

A model completely specifies the requirements and conveys all relevant information we need.

That is standard procedure set by my organization.

Other, please specify

__________________________________________________________________________________

D2: What leads you to choose this specific model? Why you did not use another type of model?

__________________________________________________________________________________

D3: What does the single model represent? __________________________________________________________________________________

D4: If you need more information, do you want to use additional models?

No (Explain how you overcome with that conflict?)

Yes (Explain why do you choose additional model?)

__________________________________________________________________________________

D5: How do you choose an additional model? What factors are important for you to choose additional model?

__________________________________________________________________________________

D6: Is there anything else you want to tell me about how you use conceptual models for your work?

________________________________________________________________________

D7: What do you think are benefits from using conceptual models?

________________________________________________________________________ D8: What are the biggest challenges in using conceptual models?

________________________________________________________________________

<Ask interviewee to suggest other respondents if possible>

<Provide opportunity for participant to comment on anything further>

<Thank the participant for their time and valuable responses>

<Note if participant would like a copy of results and obtain contact details (Email or post)>

End of Interview

Appendices 187

Appendix C

Table C. 1 Ontological Analysis of Graphical Constructs of Selected UML-Diagrams

UML-

Constructs

/elements

Shapes BWW-

constructs Rational

Cla

ss D

iagra

m

Class

User

Student ID*UserNameUserInfo

Class

Class type shows types of

things which share similar

properties in general

Member

ClassName

-className-classID

Property

Specifies attributes of a class “Attributes are the names that

we use to represent properties

of things” (e.g., ID, Name)

(Recker et al., 2009)

Relationship

Type

Bill

billNumberStudentID*billDateFineAmount

Coupling and

binding mutual

properties

Relationship types describe

the binding mutual properties

that couple two classes of

things.

Operations

Bill

-billNumber

-StudentID*

-billDate

-fineAmount

+createBill()

+updateBill()

Transformation

An operation is a behavioural

feature that may be owned by an interface. UML-operations

corresponds to BWW-

transformation (Opdahl &

Henderson-Sellers, 2002)

Cardinality 1..1

1..8

State law

Cardinality constraints

represent a state law that

constrains the values of a

binding mutual property to

certain conditions.

Inheritance

Association:

represents

generalization

Hereditary

property

Representing the “kind of”

relationship between classes

(things) which indicate that

the properties of the source

class inherit from the target class.

Aggregation

association:

represents the

part-of

relationship

Composite

thing

Special kind of association

which represents a structural

relationship between a whole

and its parts. It associates a

thing to a composite thing.

(Shanks et al., 2008)

188 Appendices

Composition

association:

represents part-

of relationship

Composite

thing

A composition is a form of

aggregation, with strong

ownership and coincident

lifetime as part of the whole.

Composition association

represents that the source

class is a component of the

target class (composite thing)

(Shanks et al., 2008).

Use

-Cas

e

Use Case

Transformation

Use cases describe sets of

actions as mappings that will

change the state of the

system.

Actor

Class

Actors are roles, which

describe specific types of

things (such as humans). See

Irwin and Turk (2005, p. 13).

Association

Binding

mutual

property

Associations draw linkages

between actors and use cases

such as which role are

authoritative for carrying out

an action.

Generalization

Excess

Generalization between use

cases does not carry an ontological meaning because

they violate the “kind of”

relationship that can exist

between things (but not

between processes changes of

states. See (Irwin & Turk,

2005, p. 13)

Act

ivit

y

Activity Validate User

Transformation

Describes a change in a state

of a thing.


Control flow

Lawful

transformation

Describes the sequence of

actions and the process. A

control flow shows the flow

of control from one action to the next. (Opdahl &

Henderson-Sellers, 2002). A

transition is a relationship

between two activities

indicating the flow of the

activities.

(Booch et al., 2005)

Fork node, join

node Excess

Synchronization bar

represents splitting and

linking parallel flows of

control

Initial/final node

Event

Represent the end event and

start event in process models. (Recker et al., 2009).

The initial state and end state

represent the start and end of

Appendices 189

the control flow (Booch et

al., 2005).

Merge node

Excess

Merge node is a control node

that brings together multiple

incoming alternate flows to

accept single outgoing flow.

Decision node

Excess

The control flows coming

away from a decision node

will have guard conditions

which will allow control to

flow if the guard condition is

met. It does not carry an

ontological meaning.

Guard condition Fine=0 Fine>0

Stability

condition

Specifies the states that are

allowable under the

transformation law


Swimlane


Library User

Phas

e

System and Class

Partitions the activity states

on an activity diagram into

groups, each group

representing the business organization responsible for

those activities.

(Opdahl & Henderson-

Sellers, 2002, p. 51; Recker

et al., 2009)

Object b: Bill

[paid]

State

Objects in this particular

diagram show that how

activities may modify

properties of the Bill (for

example, Pay Bill will change the state of the Bill

object to paid). (Booch et al.,

2005)

Object Flow

State law

Represent the flow an object

and specifies how the state of

an object should change.

Sta

te

State Item Found

State

Represents the value of the

property function of a thing

(Opdahl & Henderson-

Sellers, 2002, p. 48; Recker

et al., 2009)

Initial/final state

Stable State

Initial State is the default

starting state for the state

machine. The final state is a

special kind of state signifying that the enclosing

region is completed.

190 Appendices

Choice

Excess

Choice pseudostate realizes a

dynamic conditional branch.

It shows that the transitions

should follow the guard

conditions. It does not carry

an ontological meaning.

Guard

Condition

[Fine=>0]/Issue Bill

[Fine=0]/Update

Status

Stability

condition

Guard conditions describe

properties that restrict the

functions of a mutual

property between things to a

lawful subset.

Transition Calculate Fine

Transformation

A transition shows the flow

of control from one state to

the next. A transition is a relationship between two

states indicating that an

object in the first state will

perform certain actions and

enter the second state when a

specified event occurs and

specified conditions are

satisfied. (Booch et al., 2005)

Staff

maintain bike register

maintain rental rates

Customer

fail to return

rent out

rent return

register new bike

record bike sale

Figure C. 1 Use Case Diagram for High Peak Bicycles

Appendices 191

Bike Class

-classCode

Bicycle

-bikeRegistrationNumber

-classCode*

-bikeMake

-bikeDateOfSale

-bikeModel

-bikeNumber

-bikeDateOfPurchase

-bikePurchasePrice

-bikeSize

-bikeSellingPrice

-bikeWrite-Off

Rental Rate

-ClassCode*

-rateStartDate

-rateDay

-rateHalf-Day

-rateLateDay

Rent Line

-rentNumber*

-bikeRegistrationNumber*

-rentPre-PaidAmount

-rentActualPayment

Customer

-customerNumber

-customerName

-customerTelephoneNumber

Rental Agreement

-rentNumber

-customerNumber*

-rentDate

-rentTimeOut

-rentExpectedTimeBack

-rentExpectedPeriod

-rentActualTimeBack

-rentActualPeriod

-rentDepositType

-rentTotalPre-Paid

-rentActualTotalPaid

-classDescription

1..1

1..*

1..1

0..*

1..1 0..*

1..4

1..1

1..1

1..*

+haveClass()

+createClass+createRate()

+readRate()+haveCustomer()

+createCustomer()

+findBike()

+changeAvailability()

+createBike()

+amendBike()

+createRentLine()

+createRent()

+returnRent()

Figure C. 2 Class Diagram for High Peak Bicycles

Idle

[Request== return]

Return Request Submitted

Rent Request Submitted

Calculate rental

Actual Rental calculated

Bicycle Status Updated

Disp

lay re

ntal

rate

Bicycle Found

Rate Accepted

Rental Agrreement Issued

Update bicycle availability

Bicycle Status Updated

Checkavailability

Assign rental agreement

[Acc

epte

d]

[Not accepted]

Request Declined

[Available]

[Req

uest

==re

nt]

[Not

avail

able

]

[Another request]

[No other request]

Pay rental

Rental Paid

Update bicycle availability

Rent Rate Displayed

Bicycle Not Found

[Another request]

[No other request]

Figure C. 3 State Machine Diagram for High Peak Bicycles

192 Appendices

High Peak Bicycles

Customer Rentline

Submit a Request Check the Request

[Ren

t B

ike]

Check Rental Type

[Ret

urn

Bik

e]

Check Rental Agreement

Calculate Actual Rental

Period

Charge the Customer

Update Rental Agreement

Return Deposit

Change Bike Status

Read Bicycle Class

Read Rent Rate

Read Bike

Inform Customer

Display the rent rate

[Available][Not available]

Check the customer

Create Customer

Create Rental Line

Change Bike/s Status

Print Rental Agreement

[New Customer]

[Existing Customer]

Choose Other Bike/s

[Choose Other Bike]

[No

othe

r ch

oice

]

[Rent RateNot Accepted]

[Rent Rate Accepted]

Sign Rental Agreement

Check Bike Availability

Pay Deposit

Make Decision

Check Rent Rate

Read Customer

Figure C. 4 Activity Diagram for High Peak Bicycles

Appendices 193

Table C. 2 Ontological Analysis of Models for High Peak Bicycles

Ontological construct Class Use

Case Activity

State

Machines

Thing Class * * * Composition Thing Property in general * Hereditary Property Binding Mutual property * * State * State Law * Stable State * History Event * External Event Transformation * * * * Lawful Transformation * Stability Condition * * Coupling *

Acts on

System * Sum out of 17 6 3 6 4

Table C. 3 Overlap Analysis of Combinations of Two Models


Diagram

Class

Diagram

Activity

Diagram

State Machine

Diagram


Class Diagram 6 2 1



Table C. 4 Overlap analysis of combinations of 3 and 4 models

Use Case

Diagram

Use Case

Diagram

Use Case


Use Case

Diagram

Class Diagram Class Diagram Activity

Diagram

Activity


Activity

Diagram

State Machine

Diagram

State Machine

Diagram

State Machine

Diagram

Activity

Diagram

State Machine

Diagram

MOC 10 9 9 12 12

MOO 3 3 3 3 4

194 Appendices

Figure C. 5 Use Case Diagram for Library Management System

Appendices 195

User

-Student ID*

-userName

-userInfo

Undergraduate

student

-Student ID

-Name

-DoB

-Major

-Faculty

-Address

Postgraduate

student

-Student ID

-Name

-DoB

-Major

-Faculty

-Address

-Thesis Topic

-Thesis supervisor

Transaction

-transactionID

Reservation

-dateReserve

-dateAvailable

-studentID*

-bookID*

Copy

-bookID*

-copyNumber

-copyAvailability

1..1

1..8

Book

-ISBN

-bookTitle

-bookAuthor

-Year

-Edition

-Topic

-Category

-Keywords

-Description

0..*

1..1

-bookID

-bookReservation

Loan

-copyNumber*

-dateOut

-dateBack

0..*

1..1

1..1

1..*

Author

-biography

-birthdate

-name

Research

Journals

-journalTitle

-publisher

-volume

-issue

0..1

0..*-author

1..*

1..*

1..*

1..*

Collection

-Name of collection

-relevant student major(s)

-number of books in

collection

Bill

-billNumber

-StudentID*

-billDate

-fineAmount0..4

1..1

+createBill()

+updateBill()

+createUser()

+checkUser()

+cancelUser()

+payFine()

+returnBook()

+createReservation()

+getReservation()

+checkAvailableDate()

+createLoan()

+getLoan()

+completeDateBack()

+createJournal()

+createAuthor()

+createCopy()

+addBook()

+searchBook()

+displayBookDetails

+updateBookDetails()

+createCollection()

+createTransaction()

+getTransaction()

0..1

0..1

Figure C. 6 Class Diagram for Library Management System

196 Appendices


Library UserP

ha

se

Request an ItemCheck the Request

Calculate Fine

Check Availability

Validate User

Re

turn

Bo

ok

Request Book

Re

qu

est

Jo

urn

al A

rtic

leN

ot

Av

aila

ble

Inform User

Available or Hired

Ava

ila

ble

Request Loan H

ire

d Request Reservation

Validate User

Validate User

Make Reservation

Issue Book

Change Book Status

No

t V

alid

or

Exc

ee

d lo

an

or

bil

l

No

t V

alid

or

Exc

ee

d lo

an

or

bil

l

Submit Request

Sto

p t

he

ord

er

Continue to Order

Authorise Journal Access

Po

st-g

rad

e S

tud

en

t

Not ValidShelve Book

Fin

e =

0

Fin

e >

0

Shelve Book Issue Bill

Change User Status

Check User Status

Check User Status

Valid & No exceed loan and bill

Valid & No exceed loan and bill

b: Bill

[unpaid]

Save as Debt

Pay the Bill

Change Book Status

b: Bill

[paid]

Change User Status

Change Book Status

Figure C. 7 Activity Diagram for Library Management System

Appendices 197

Figure C. 8 State Machine Diagram for Library Management System

Table C. 5 Combined Ontological Completeness and Ontological Overlap for Library Management System

Ontological construct Class Use

Case Activity

State

Machines

Thing

Class * * *

Composition Thing *

Property in general *

Hereditary Property *

Binding Mutual property * *

State * *

State Law * *

Stable State *

History

Event *

External Event

Transformation * * * *

Lawful Transformation *

Stability Condition * *

Coupling *

Acts on

System *

Sum out of 17 8 3 8 4

198 Appendices

Table C. 6. Overlap Analysis of Combinations of Two Models


Diagram

Class

Diagram

Activity

Diagram

State Machine

Diagram


Class Diagram 8 3 1



Table C. 7. Analysis of MOC and MOO

Combination Completeness Overlap

High Com, Low Overlap Class + State 11 1

Low Com, High Overlap Class + Use Case 8 3

Low Comp, Low Overlap State + Use case 6 1

High Com, High Overlap Class + Activity 13 3

Table C. 8 Measurement Instruments for the Designed Experiment to test the Theory of Combined Ontological

Coverage

Demographic Questions

What is your gender?

Male

Female

Are you an under- or post-graduate student?

Under-graduate

Post-graduate

Have you ever learned UML (Unified Modeling Language)?

Yes

No

Are you enrolled in a unit that teaches conceptual modeling in semester 2/2016, or have you completed such a

unit in an earlier semester?

Yes

No

Modeling Experience (Adapted from (Burton-Jones & Meso, 2008; Recker, 2010a))

Roughly, how many UML diagrams have you:

……..Created to date? None

……..Read to date? None

Familiarity with UML (Adapted from Recker (2010a))

Please rate your agreement with the following statements about your familiarity with UML on a scale from 1 (strongly disagree) to 7 (strongly agree):

(1) Overall, I am very familiar with UML.

(2) I feel very confident in understanding diagrams created with UML.

Appendices 199

(3) I feel very competent in using UML diagrams.

(4) Compared to a professional system developer or analyst, I would rate my level of experience in interpreting UML diagrams as: (a scale from 1 (Very low) to 7 (Very high))

Domain Knowledge (Bera et al., 2014; Burton-Jones & Meso, 2008)

Please rate your level of knowledge and experience on a scale from 1 (Very low) to 7 (Very high):

(1) Compared to someone who works in a bicycle rental shop, I would rate my level of knowledge of

activities in a bicycle rental shop (such as updating rent rates, preparing rental agreements,

categorizing bicycle classes) as:

(2) Compared to someone who works in a library, I would rate my level of knowledge of activities in

a library (such as shelving books, updating book status, reserving books) as:

Model Familiarity Test [the correct answers are written in bold] (Figl & Recker, 2014)

What type of model is this?

o UML Class diagram

o UML Component Diagram

o UML Object Diagram

o UML State Machine Diagram

o UML Activity Diagram

o I don’t know



o UML Class diagram

o UML Sequence Diagram


o UML Use Case Diagram

o I don’t know



o UML Class diagram




o I don’t know


o UML Class diagram



o UML Communication Diagram


o I don’t know

200 Appendices




o UML Timing Diagram

o UML Communication Diagram

o UML Interaction Overview Diagram

o I don’t know


o UML Component Diagram o UML Profile Diagram

o UML Package Diagram

o UML Development Diagram

o UML Object Diagram

o I don’t know

Measurement Instruments for the first phase of the experiment (model selection)

Problem Solving Questions - Upcoming task for model selection (self-developed)

1. The managers of the Bicycle rental shop are interested in improving the overall revenue. Based on the

information in the diagrams, identify at least two ways how they can do so.

2. Customer feedback indicates that client satisfaction significantly decreased over the last six months.

From the information in the diagrams, identify at least two reasons why this could be the case.

Use of Information –learning effect (Adapted from Burton-Jones and Meso (2008))

On a 1-5 scale (“Not at all” to “A great extent”), to answer the previous two problem-solving questions…:

- To what extent did you rely on the information in diagram (A)?

- To what extent did you rely on the information in diagram (B)?

- To what extent did you rely on the information in diagram (C)?

- To what extent did you rely on the information in diagram (D)?

- To what extent did you rely on your general knowledge?

- To what extent did you make assumptions?

Deselection of models (multiple answers are possible) (Self developed)

If you had an opportunity to remove a diagram/s which, in hindsight, you feel was not useful in answering the

questions, which diagram/s would you have removed?

- Diagram (A)

- Diagram (B)

- Diagram (C)

- Diagram (D)

- None of them

Measurement Instruments for the second phase of the experiment

Appendices 201

Comprehension Questions (Self developed)

Please answer the following questions about the Library based on the information in the diagrams you received:

(the correct answers highlighted in bold)

- Are there any limitations for a student to extend a loan? (Yes/No/Not Sure/There is not enough

information to know)

- Can a post-graduate student borrow more than 6 research journals at the same time? (Yes/No/Not

Sure/There is not enough information to know)

- Can a user of a library be an employee of the university? (Yes/No/Not Sure/There is not enough


- Do students need to pay a fee to access research journals? (Yes/No/Not Sure/There is not enough


- Can a student request books that are loaned by someone else? (Yes/No/Not Sure/There is not enough


- Are there a limited number of copies of each book in the library? (Yes/No/Not Sure/There is not

enough information to know)

- Can an undergraduate student borrow a book and a research journal at the same time? (Yes/No/Not

Sure/There is not enough information to know)

- Do students need to pay fines for a delay in returning books? (Yes/No/Not Sure/There is not enough


Problem Solving Questions (Self developed)

Please answer the following questions, based on your understanding, from the diagrams.

- Explain the main functionality of the library management system. Write down as many supported

activities as you can.

- Under what circumstances would a student fail to borrow an item s/he needs? Write down as many

possible reasons as you can think of.

- How can the library management system be improved to ensure that students in each major can have access to relevant books and journals they need? Write down as many possible solutions that should be

considered.

Cloze Test (Self developed) (No. of correct answers)

Please fill in the blanks with the appropriate words based on the understanding you gained about the Library

Management System.

The library management system is designed to track books and journal articles in the library of a university. It

also provides functions to …(1)… and control orders and loans, and calculate fines for …(2)… . Users of

library management system are either …(3)… or …(4)… .

The library has books in a wide variety of areas and topics. Normally, there are multiple …(5)… of a particular

book in the library. The library also provides access to research journals.

…(6)… do not have access to research journals . But, books in the library are categorised in specific …(7)… to

be sure that the library provides relevant books for students in different majors.

Users can search items using title, authors name, topic and any keywords related to their majors. In fact, …(8)…

can search items in the library, but only …(9)… can borrow books or have access to journal articles.

The results of a search for a book can be threefold: … (10) …, …(11)…, or …(12)…. If a book is on loan by someone else, a user can still …(13)… the book. There are no limitations on how many …(14)… a …(15)…

student can have access at the same time.

202 Appendices

Answers to the Cloze test

1) Monitor/manage, 2) late returns/none returns, 3) undergraduate-students, 4) postgraduate-students, 5)

copies, 6) undergraduate-students, 7) collections, 8) everybody, 9) valid users, 10) available, 11)

unavailable, 12) On loan, 13) reserve/request, 14) research journals, 15) postgraduate

Perceived Usefulness of model combinations (Adopted from (Recker & Rosemann, 2010a, 2010b) on a 1-7

scale (“Strongly disagree” to “Strongly agree”)

Please answer the following questions based on your level of agreement using a 1-7 scale from “Strongly

disagree” to “Strongly agree”

- Overall I found the combination of diagrams useful for the given tasks.

- I found the combination of diagrams useful for achieving the purpose of the given tasks.

- The combination of diagrams helped me in meeting the objectives of the given tasks.

Cognitive Load (Adapted from Cierniak et al. (2009)) on a 1-7 scale (“Not at all” to Extremely”).

Please answer the following questions based on the rates from 1- (Not at all) to 7-(Extremely).

- How difficult were the tasks for you?

- How difficult was the content of the provided diagrams for you?

- How difficult was it for you to perform the tasks with the provided diagrams?

- How much did you concentrate during the tasks?

Ease of Understanding (Burton-Jones & Meso, 2008)

Please answer the following questions based on the rates from 1- (Not at all) to 7-(Extremely).

- To what extent did you find the diagram (A) [type of diagram will vary for each group] to be complex?

- To what extent did you find the diagram (A) to be difficult to understand?

- To what extent did you find the diagram (B) [type of diagram will vary for each group] to be complex?

- To what extent did you find the diagram (B) to be difficult to understand?

Appendices 203

Table C. 9 Interrater Agreement

Value

Asymp. Std.

Errora Approx. Tb

Approx.

Sig.

Measure of Agreement Kappa

Problem-Solving .811 .037 27.208 .000

Measure of Agreement Kappa

Cloze Test .907 .027 33.037 .000

N of Valid Cases 131 a. Not assuming the null hypothesis.

b. Using the asymptotic standard error assuming the null hypothesis.

Table C. 10. Factor Loadings

Component Matrixa

Component

1

Familiarity Self Rate 1 .836




Usefulness 1 .799

Usefulness 2 .898

Usefulness 3 .884

Cognitive Load 1 .852




Table C. 11 Reliability Test

Reliability Statistics

Cronbach's Alpha

Cronbach's Alpha

Based on

Standardized Items N of Items

Modeling Familiarity .775 .778 4

Perceived Usefulness .822 .825 3

Cognitive Load .806 .815 3

Documents

Mohammad Ali Jabbari Sabegh - QUT · Chapter 4: Conceptual Modelling Research in Information Systems: ... AIS Association for Information Systems BWW Bunge, Wand, and Weber BPMN Business