Embed Size (px)
Citation preview
Middlesex University Research Repository
An open access repository of
Middlesex University research
http://eprints.mdx.ac.uk
Barat, Souvik and Clark, Tony and Barn, Balbir and Kulkarni, Vinay(2017) A model-based approach to systematic reviews of research
literature. In: ISEC 2017: Innovations in Software EngineeringConference, 05-07 Feb 2017, Jaipur, India.
http://dx.doi.org/10.1145/3021460.3021462
Final accepted version (with author's formatting)
Available from Middlesex University’s Research Repository athttp://eprints.mdx.ac.uk/21071/
Copyright:
Middlesex University Research Repository makes the University’s research available electronically.
Copyright and moral rights to this thesis/research project are retained by the author and/or other copyright owners. The work is supplied on the understanding that any use for commercial gain is strictly forbidden. A copy may be downloaded for personal, non-commercial, research or study without prior permission and without charge. Any use of the thesis/research project for private study or research must be properly acknowledged withreference to the work’s full bibliographic details.
This thesis/research project may not be reproduced in any format or medium, or extensive quotations taken from it, or its content changed in any way, without first obtaining permissionin writing from the copyright holder(s).
If you believe that any material held in the repository infringes copyright law, please contact the Repository Team at Middlesex University via the following email address:
The item will be removed from the repository while any claim is being investigated.
A Model-Based Approach to Systematic Reviews ofResearch Literature
Souvik BaratTata Consultancy Services
Research, [email protected]
Tony ClarkSheffield Hallam University,
United Kingdom
Balbir BarnMiddlesex University, United
Kingdom
Vinay KulkarniTata Consultancy Services
Research, [email protected]
ABSTRACT
A systematic approach to develop a literature review is at-tractive because it aims to achieve a repeatable, unbiasedand evidence-based outcome. However the existing formof systematic reviews such as Systematic Literature Review(SLR) and Systematic Mapping Study (SMS) are knownto be an effort, time, and intellectual intensive endeavour.To address these issues, this paper proposes a model-basedapproach to Systematic Review (SR) production. The ap-proach uses a domain-specific language expressed as a meta-model to represent research literature, a meta-model to spec-ify SR constructs in a uniform manner, and an associateddevelopment process all of which can benefit from computer-based support. The meta-models and process are validatedusing real-life case study. We claim that the use of meta-modeling and model synthesis lead to a reduction in time,effort and the current dependence on human expertise.
Keywords
Literature Review; Systematic Literature Review; System-atic Mapping Study; Meta Modeling; Model Based Litera-ture Review
1. INTRODUCTIONA thorough literature review on a topic establishes a firm
foundation for advancing knowledge [25]. It identifies exist-ing research and the areas where research is needed. System-atic Review (SR) methodologies in the form of SystematicLiterature Review (SLR) [18] and Systematic Mapping Study(SMS) [22] methodology are two popular choices for manydisciplines such as medicine, genetics, psychology and socialscience. The rigorous planning, methodical execution of theplan, unbiased outcomes, and repeatable process make SRattractive to the researchers [5, 16, 23].
ACM ISBN 978-1-4503-2138-9.
DOI: 10.1145/1235
However, the general experience of using SR methodolo-gies on Software Engineering (SE) related topics is not equallyencouraging. Software research practitioners have raisedseveral concerns including: methodological challenges; us-age issues and a steep learning curve. The commonly citedissues are: (i) SR guidelines are time consuming and error-prone [16], (ii) lack of guideline for conducting individualreview process steps [17], for example, the guidance on howto eliminate bias[14] from a literature corpus or how to jus-tify the quality of a review outcome, etc. (iii) and lack ofguidance to adopt a specific SR technique, i.e., when to pre-fer SMS over SLR or vice versa [10]. Moreover, softwareresearchers have expressed their difficulties to manage andcorrelate large number of review artefacts those are pro-duced in the various phases of SR.
We have experienced many of the cited concerns ourselveswhile using SMS and SLR for our primary research1. As acourse of action, we conducted a tertiary review on SR lit-erature (discussed in section 2), explored solutions proposedin the literature and tried to use them as suggested to over-come the limitations encountered. In our experiment, wecombined SMS and SLR as recommended in [13], introducedan iterative approach in conventional SR as suggested in [20]and used snowballing technique[26] in SMS/SLR. Like [21]we experienced significant challenges in combining the rele-vant concepts of SR within either SMS or SLR ue to theirinflexible nature.
This paper proposes an approach that refines conventionalSR to combine frequently used concepts of literature reviewsand address some of the commonly faced challenges withoutcompromising the rigour, precision and quality outcome ofSR. In particular, we propose a domain-specific language(expressed as a meta-model) for representing research lit-erature in a precise term, conceptualise a meta model todescribe core concepts of systematic review, and introducea model-based realisation of systematic review method. Ourproposed approach visualises SR production as coordinatedmodel creation (instantiation), model navigation, and modelsynthesis effort that can largely be carried out by provenmodeling and model-processing technologies. The approachis presented in section 3. The use of a model-based approachfor conducting SR on SE related topic is new in SR prac-
1http://www.tcs.com/research/Pages/Model-Driven-Organization.aspx
Table 1: Validation properties of SR methodologyProperty Topic Description
BiasnessandThreatstovalidity
Study iden-tification
Finding publications using searchstring, snowballing [26], etc,.
Literaturecoverage
Coverage of identified literatureand conformance of quasi-goldstandard [28].
Qualityevaluation
Assessing the quality of protocol,selected publications and study.
Data ex-traction
Data extraction technique, datarepresentation for further process-ing, and classification technique.
Usability
RepeatableThe same synthesis can be reachedeven when performed by a differentpractitioner.
Structuredrepresenta-tion
Structured representation of pub-lication template and study tem-plates
Representa-tion andVisualisa-tion
Representation and visualizationof review artefacts such as publica-tion, intermediate and final reviewoutcomes.
TraceabilityTraceability and analyzability ofreview artefacts.
tice but it appears promising for a variety of reasons. First,meta-modeling techniques are an effective means of unify-ing multiple concepts. Second, a resulting model can be im-parted with well-defined semantics thus enabling automatedprocessing such as validation, verification, and transforma-tion. Third, it helps in establishing traceability relationshipsand ensuring consistency of various artefacts. Finally, meta-modeling, model-validation and model-synthesis techniqueshave been proven across a variety of application domains.
With the proposed approach, we claim to overcome twocommonly faced issues: i) error-proneness (due to lack oftimely validation), and ii) effort-intensiveness (due to lim-ited scope for automation and inability to establish trace-ability between review artifacts).
We illustrate the proposed approach and evaluate ourclaims using a case study that explores existing enterprisemodels (EM) and evaluates suitability of identified EM tech-niques for our primary research1. The case is presented insection 4. How the proposed method helps in reducing theerror-proneness and effort-intensiveness for proving goodnessof a systematic review is illustrated in section 5. We alsoevaluate the goodness of the review process and review out-come of SR using two quality properties namely biasnessand threats to validity, and a set of usability properties asdepicted in Table 1. Finally, we believe the proposed ap-proach can serve as a foundation for a robust model-basedliterature review tool. We discuss our plan on future SR tooldevelopment by extending the SLRTool [4] in section 6.
2. BACKGROUNDWe conducted a tertiary study using SMS methodology
to understand the trend of systematic review in SE, reviewtechniques adopted in SE and the experience of the literaturereview practitioners.
We observed an increasing trend of SR publications infive digital libraries namely Scopus, ScienceDirect, IEEEXplore, ACM Digital Library and Web of Science as shownin Fig. 1. We also found homogeneity in adopting SMSand SLR methodology. Primarily the review practitionersfrom SE followed a three-phase review method that includes
Figure 1: Overview of SR on SR literature
planning, execution and synthesis and they largely differedin terms of: how a research questions is formed, how thepublication corpus is explored, what is the reviewing style,and what is the principal objective of review outcome. Forexample, the SR that adopted SMS methodology focused onbroad research question, reviewed large number of publica-tions, adopted a style which is not as thorough as SLR, andaimed for publication classification leading to a high-levelunderstanding. In contrast, the SR with SLR methodologyfocused on precise research questions leading to precise out-comes by conducting a thorough review of relatively smallnumber of publications.
The literature presenting SLR and SMS case studies em-phasized several benefits such as improved precision, fair-ness, trustworthiness and auditability of review method andreview outcomes. The process for conducting SR is alsohighlighted as rigorous and repeatable. However, SR is con-sistently reported as time, effort and intellectual intensiveactivity in SE. Morover, an important trend we observedin our tertiary study that the research contribution is notuniformly distributed across the research communities. Forinstance, the 227 publications out of 837 contributions inScopus digital library (i.e., 27%) are from 10 affiliations/in-stitutions wherein the Universidade Federal de Pernambucohas 49 publications and Keele University has 38 publica-tions. Similarly 170 publications (20% of total publication)in Scopus digital library are from Brazil. Spain and Swe-den are the next in the table with 83 and 76 publicationsrespectively. So there is a clear indication that the use ofSR is largely limited to a set of research groups from fewinstitutions and counties. Zhang et.al have reported simi-lar observation based on their tertiary review conducted in2011 [27]. They used semi-structured interview techniqueto understand the cause for such low adoption in largerresearch community. They found that around 50% noviceliterature review practitioners are unaware of the detailedsteps of SR. Knowledge intensiveness, tediousness and error-proneness are the key factors cited for low adoption of SRin SE. Several tertiary reviews [24, 1, 7, 8, 10] are congruentwith these observations.
As a course correction, several methodological refinementsto SR are suggested [16, 23]. But these advancements arefound to be limited to methodological improvements [23].Thus the experts seem to benefit significantly by these ad-vancements but larger research communities (who are notexpert but keen on using SR in their primary research) donot find them beneficial in adopting SR in their research.
A group of expert SR practitioners have advocated toolsfor conducting SR. Tools, such as SLuRp [6], StArt [11],SLR-Tool [9] and SLRTool[4] to automate the process stepsdescribed in Table 2 at varying levels of sophistications. But
Figure 2: Research Literature Meta Model
Table 2: Process steps of SRPhase Associated Process Steps
Planning
1. Defining motivation for conducting SR,2. Defining research questions, 3. Definingsearch strategy to identify relevant literature,4. Defining data capture strategy, and 5. Re-view of the planning
Execution1. Select Publications, 2. Assess publicationquality, 3. Conduct complete review and ex-tract data, 4. Document review outcomes.
Synthesisand report-ing
1. Synthesis of collected data, 2. Documen-tation and visualization of study reports, 3.Publish Results
the current state of SR tools is not matured enough to helpSE researchers to adopt SR in primary research in a seam-less manner. Nor is there evidence of any substantial use.Marshall et al. concluded their comparative study on SRtools [21] with a similar observation.
We conducted multiple literature review using SMS andSLR for our primary reviews [19, 3]. We started with tradi-tional literature review (TLR) and moved to SR [2]. Thoughthe quality of review outcomes and evidence produced by SR[2] was significantly high as compared to TLR, it came ata price too. Our experience gathered through reviews usingconventional SR and the evidences (gathered through ter-tiary study) about the lacunae of current form of SR ledus to work on SR approach. Our proposed approach thatovercomes some of the concerns raised by SE researchers ispresented next.
3. APPROACHWe propose a model based approach for conducting sys-
tematic review of research literature. The approach is com-posed of three research contributions: i) a domain-specificlanguage expressed using meta-model (termed as RLMModel)to represent research literature, ii) a conceptual meta-model(SRMModel) to represent the core concepts of SR, and iii)model based realisation of a SR process (SR developmentprocess). The RLMModel represents the concepts of pub-lications and digital libraries in a uniform and machine in-terpretable form. The SRMModel combines frequently usedconcepts of SR and serves a basis for specifying different
exp ::= metacnstr| phrase in content| phrase holds in content
metacnstr ::= char| (metacnstr)| metacnstr and metacnstr| metacnstr or metacnstr| not(metacnstr)
char ::= data op valueop ::= ’=’ | ’<’ | ’>’ | ’!=’ | or | and
data ::= publicationDate
| subjectArea
| sourceType
| citationCount
| language
| authorCountry
| authorInstitution
content ::= text
| section*section ::= t i t le
| keyword
| abstract
| intro
| papersection
phrase ::= phrase and phrase| phrase or phrase| not(phrase)
value ::= number | string
Figure 3: Syntax of expression language
artefacts of an SR in relatable, traceable, navigable andanalysable form. Proposed SR process ensures the method-ological guidelines of supported concepts.
We consider two widely accepted SR methodologies i.e.,SMS proposed by Petersen et al. [22] and SLR recommendedby Kitchenham [15] as our methodological foundation. Wealso adopt the protocol concept defined by EBSE ResearchGroup [14] for describing the format of the research litera-ture and review artefacts, snowballing technique describedin [26] for improving search space, the methodological im-provements proposed by experienced literature review prac-titioners such as [16, 23] for incorporating latest method-ological developments, and finally incremental and iterativeapproach suggested by [20] for enabling SR to novice prac-titioners.
The core concepts of RLMModel, SRModel and SR devel-opment process are described below.
3.1 Research Literature Meta Model
Figure 4: Systematic Review Meta Model
We define a research literature meta-model (RLMModel)to describe research literature and literature corpus in a uni-form and machine interpretable form. The core concepts ofRLMModel are depicted in RLMModel in Fig. 2.
As shown in the figure, the research literature is repre-sented as Publication. A Publication has two key identitiesnamely title and refId. Attribute title captures the Titleof the Publication and refID refers to PublicationIdentifica-tion that represents a unique identifier such as bibliographyreference. Typically, a Publication is contributed by set ofAuthor from specific institution and country and it has threemeta-elements namely MetaData, TextContent, and Refer-ences. MetaData describes the characteristics of the Pub-lication such as Publication Date, Subject Area, PublicationSource, Citation Count, Language, etc. We use the prop-erties suggested by EBSE Research Group [14] to form theattributes of MetaData. TextContent describes the contentof the Publication. TextContent is typically described us-ing set of Phrase where a Phase is a sequence of words orstring. The key elements of TextContent are Keywords, andAbstract, Introduction and other Sections. The Referencesare list of PublicationIdentifications of other Publicationsthose are cited in TextContent.
We use the term Corpus to represent collection of Pub-lications. Digital libraries that achive publications such asACM Digital Library, IEEE Xplore, Scopus are specialisedCorpus, termed as Digital Library. A Corpus (and thus Dig-ital Library) often holds some Criteria. It is expected thatall Publications belongs to a Corpus must conform to itsassociated Criteria. We introduce an OCL like expressionlanguage to specify required expression to describe Criteria.Element Expression represents the textual expression. Thekey constructs of proposed expression language is illustratedin Fig. 3. The constructs support three kinds of expressions- i) metacnstr that describes the evaluation criteria of Meta-Data elements (e.g., publicationDate > 2010), ii) phrase ex-istence (i.e., phase in content) that evaluates the existenceof Phrase in TextContent (e.g., phrase ”Organisation” existsin abstract section), and iii) meaning existence (i.e., phaseholds in content) that judges if set of Phrases holds true in
a TextContent (e.g., paper ”describes a case study”).
3.2 SR Meta ModelThe elements of SR Meta Model (SRMModel) are ex-
pressed using the concepts described in RLMModel. Asshown in Fig. 4, a literature review starts with a broad prob-lem statement, which we term as ResearchProblem. A re-search problem triggers one or multiple ResearchQuestion.A ResearchProblem can have multiple sub-problems; sim-ilarly a ResearchQuestion can be elaborated with multi-ple sub-questions. Association subProblem and subQuestionspecify those relationships.
A ResearchQuestion requires at-least one SearchProtocolfor conducting reviews. A SearchProtocol can be describedsufficiently using eight basic concepts that include ReviewStyle (considered as an attribute of SearchProtocol namedstyle), DigitalLibrary, InclusionCriteria, ExclusionCriteria,QualityCriteria, SnowballingRule, Classification, and StudyTem-plate. Attribute style captures the review style, such as SMSand SLR style, for conducting reviews. DigitalLibraries de-scribes the (initial) sources of the Publications. Search crite-ria such as InclusionCriteria, ExclusionCriteria, QualityCri-teria help in identifying relevant Publications from selectedDigitalLibraries. These criteria also form the Criteria ofthe new Corpus. Model element SnowballingRule capturessnowballing rules as recommended in [26].
We introduce two meta-elements Classification and StudyTem-plate to represent a Study of a Publication. Classificationcategorizes the Publications and Corpus. StudyTemplate iscollection of attributes of Publication element described inRLMModeland and the attributes that capture the findingsof a study. Findings can be described using Descriptionelement wherein a Description represent a specific findingusing one of the two form:- simple text using the descrip-tion attribute of Description or structured information thatcan represented using a customized Meta Model. A Studyelement is an instance of StudyTemplate and a Report isthe synthesis of conducted studies which are captured usingStudy elements.
SRMModel introduces a set of associations to representthe progress of the review process. Association inclusion-
Figure 5: SR Process
Finds and exclusionFinds represent the list of Publicationsselected using InclusionCriteria and ExclusionCriteria re-spectively. The association qualifies represents the list ofPublications that conforms to QualityCriteria and associa-tion adds indicates the Publictions added using SnowballingRule.Association selects represents final list of Publications thatneed to be considered for conducting detailed study.
In this formulation, a SR is systematic way of exploringa set of DigitalLibraries for a set ResearchQuestions (thatare originated for specific ResearchProblem) using multiplecriteria, collecting evidences for answering ResearchQues-tions using standard format specified using StudyTemplateand producing consolidated Report by analysing sufficientlylarge number of Studies. An SR starts with a large Corpusof Publications collated from multiple DigitalLibraries, pro-duces a reasonable Corpus that satisfies InclusionCriteria,ExclusionCriteria, QualityCriteria and SnowballingRule, andends with sufficient evidences that are extracted from consol-idated Corpus to answer ResearchQuestions. The detailedmethod for conducting SR using the concepts described us-ing SRMModel is illustrated next.
3.3 SR ProcessWe define a process for conducting SR by refining three
phases SR process that includes planning, execution andsynthesis as depicted in Table 2. The refinements are interms of how these process steps need to be performed us-ing the concepts described using SRMModel. As shown inFig. 5, the planning phase of the review process starts withSpecify Problem Statement. This step instantiates meta el-ement ResearchProblem. The second process step, Defineresearch Question, defines the ResearchQuestions and theirsub-questions. The subsequent step specifies eight basic el-
ements of SearchProtocol namely review style, set of Digi-talLibrary, InclusionCriteria, ExclusionCriteria, QualityCri-teria, SnowballingRule, Classification and StudyTemplate.The InclusionCriteria, ExclusionCriteria and QualityCrite-ria should be specified using the expression language pre-sented in Fig. 3. The final process step of planning phaseis Validate Review Protocol. This step is considered as thecheckpoint of planning phase that validates all the manda-tory guidelines.
We classify the validation rules into two types - structuralconformance and quality conformance. The structural con-formance ensures the structural correctness of the instancemodel. For example, model of ResearchQuestion must havea SearchProtocol, SearchProtocol must have review style,at-least one DigitalLibrary and InclusionCriteria. We usemodel cardinality and OCL based pre- and post-conditionto specify and ensure these structural of validation.
The quality conformance is a qualitative assessment of themodel instances. For example, research questions are rele-vant and well-formed, the InclusionCriteria and Exclusion-Criteria are aligned with ResearchQuestions, DigitalLibrarylist is exhaustive for a topic, and so on. These validationsrequire precise understanding of ResearchProblem and othermodel elements thus it is non-automatable task.
The auditing related information can be captured by refin-ing SRMModel. For example, who has done the planning ofa review process can be captured by adding Reviewer infor-mation. Many such extensions are possible and consideredin our approach, which are not discussed in this paper.
The execution phase has two logical steps - select publica-tion and conduct study. Select publication logical step cre-ates publication Corpus from selected DigitalLibraries usinga series of iterative process steps. Involved process steps
Figure 6: Planning phase specific SR Model - An instance of SRMModel
are: Select Digital Library, Find Publication using Inclu-sion Criteria, Eliminate (irrelevant) Publications using Ex-clusion Criteria and Consolidate Selected Publications. Se-lect Digital Library process step selects one DigitalLibraryfrom set of DigitalLibrary that are associated with Search-Protocol and translates InclusionCriteria and ExclusionCri-teria into DigitalLibrary specific search strings (that can beused for searching specific publications from the digital li-braries). The process steps Find Publication using InclusionCriteria and Eliminate Publications using Exclusion Crite-ria apply InclusionCriteria and ExclusionCriteria on Dig-italLibrary and establish inclusionFind and exclusionFindlinks respectively. The process of for searching Publicationsfrom multiple DigitalLibrary is shown as an iterative loopin Fig. 5, but it can be done in parallel as well. ConsolidateSelected Publications process step validates and consolidatesthe list of publications that need to be considered for evalu-ating QualityCriteria.
We propose to read title, Abstract, and Introduction sec-tions of a Publication to assess QualityCriteria. Select Pub-lications using Quality Criteria process step evaluates Qual-ityCriteria and establishes qualify links.
The next step of select publication logical step is ValidateCorpus. This step removes duplicate entries, validates selec-tion process, confirms the list of Publications that need tobe considered for detailed study and establishes selects links.Essentially this process step construct a new Corpus for sub-sequent review process. If required, one can perform forwardsnowballing [26] to add more publications to Corpus. Thisstep also establishes refers links between Publications (ifone Publication refers other Publication). We recommendthis time- and effort intensive activity as this relationshipsays a lot about the biasness of newly constructed Corpus.Significant percentage of Publications in a Corpus from agroup of Authors or large number of Publications connectedthrough refers links is an indication of biased Corpus. Theverification of quasi-gold standard [28] to evaluate the com-
prehensiveness of a Corpus can be performed in this step.The next logical step of execution phase is conducting
study on selected Publications or constructed Corpus. Theinvolved activities of this step are to read title, Abstract andIntroduction sections of a Publication to decide a Classifica-tion, conduct detailed study of the Publication using reviewstyle specified in SearchProtocol and record conducted studyby instantiating StudyTemplate into Study. The conduct-ing study can be done in parallel by segmenting publicationCorpus into multiple clusters.
The final phase of the review process is the summariza-tion, synthesis and reporting. The quantitative statisticalanalyses, qualitative sense making, and theory building ac-tivity are often considered for synthesizing research findingsof Study instances and producing meaningful Report.
As shown in Fig. 5, a review report can conclude a re-view or it can raise other set of ResearchQuestions. Thisloop back mechanism enables iterative and incremental re-view process as oppose to linear review method followed inconventional SR.
We illustrate our approach using a case study on reviewingthe state-of-the-art of enterprise modeling related researchcontributions. Next section presents this illustration.
4. ILLUSTRATING CASE STUDYAs part of our primary research initiative, we are work-
ing towards a technological infrastructure to support organ-isational decision-making [19, 3]. We have conducted mul-tiple literature reviews to explore the state-of-the-art andstate-of-the practice of organisational decision-making, themodelling and analysis capabilities of Enterprise Modeling[2], exploration of Actor Model of Computation [12] in thecontext of organisation decision making, and so on. A sys-tematic review (SR) to explore Enterprise modeling (EM)related literature for evaluating the suitability of EM tech-niques in the context of organisational decision-making usingconventional Systematic Mapping Study (SMS) methodol-
ogy is presented in [2]. This section illustrates the same SRusing proposed approach.
A brief problem statement that motivate a literature re-view is summarised as follows:
One of the key challenges modern organisations face ishow to make effective decisions within a dynamic environ-ment. Precise understanding of various aspects of the or-ganisation such as goals, organisation structure, operationalprocesses, historic data and the stakeholders of the organi-sation is necessary to arrive at effective decisions. Currentindustry practice of decision making relies heavily on hu-man experts using tools such as spreadsheets, word proces-sors, and diagram editors. The state-of-the-art of enterprisemodeling (EM) and Enterprise Architecture (EA) related re-search contributions claim precise representation and sophis-ticated analysis capabilities for enterprises. Thus leveragingexisting EM approaches in a meaningful manner could bea way to improve the state-of-the-practice of organisationaldecision-making.
We consider this statement as ResearchProblem (RP1) ofa SR that we illustrate in this paper. The research problemRP1 triggers several research questions, such as: What arethe publications on EM techniques that focus on enterprisemodeling? What are the existing EM modeling techniques?What are their characteristics? What kinds of analysis aresupported by EM approaches? Are they capable of support-ing organisational decision-making? so on. he Three phasesof SR to explore a research question using our proposed ap-proach is illustrated below:
4.1 Planning PhaseThe planning phase formalizes the research questions and
research protocol for conducting literature review. The pro-cess step Specify Problem Statement instantiate Research-Problem RP1 as shown in Fig. 6. Next step Define ResearchQuestins instantiate a broad ResearchQuestion RQ1- Whatare the papers on Enterprise Modelling (EM) and EnterpriseArchitecture (EA) that focus on enterprise modelling? anda sub-ResearchQuestions- RQ1.1 - What are the EM tech-niques cited by identified papers? Process step Define SearchProtocol defines the complete specification of ResearchPro-tocol using the SRMModel. As shown in the figure, the pro-cess step selects Systematic Mapping Study as the reviewStyle, chooses five DigitalLibraries namely Scopus, ACMDigital Library, IEEE Xplore, ScienceDirect and Web of Sci-ence as initial Corpus to identify EM related Publicationsand defines InclusionCriteria, ExclusionCriteria and Qual-ityCriteria to form a new Corpus for conducting detailedstudy. All criteria are specified using the expression lan-guage specified in Fig. 3. In this review, the InclusionCri-terion IC1 =[(subjectArea = Computer Science AND docu-mentType = (Conference Paper OR Journal) AND language= English) AND EXISTENCE OF (”Enterprise Architec-ture”OR ”Enterprise Model”OR ”Enterprise Modelling”OR”Enterprise Modeling”) IN TextContent)] is very broad as itis designed to find all Enterprise Modeling (EM) and En-terprise Architecture (EA) related Publication from Com-puter Science Subject Area and published in Conference orJournal in English Language. The ExclusionCriterion EX1= [EXISTENCE OF (”workflow” OR ”BPR” OR ”gover-nance” OR ”government” OR ”security” OR ”mining” OR”re-engineering” OR ”Six Sigma” OR ”SOA” OR ”mashups”OR ”Web Service” OR ”Cloud” OR ”data warehouse” OR
Table 3: Example of Search String
Inclusion Criteria Exclusion CriteriaTITLE-ABS-KEY (Enterprise Ar-chitecture OREnterprise ModelOR EnterpriseModelling OR En-terprise Modeling) AND ( LIMIT-TO(DOCTYPE,"cp") OR LIMIT-TO(DOCTYPE,"ar") OR LIMIT-TO(DOCTYPE,"ch") OR LIMIT-TO(DOCTYPE,"bk") ) AND ( LIMIT-TO(SUBJAREA,"COMP") ) AND ( LIMIT-TO(LANGUAGE,"English" ) )AND ( LIMIT-TO(SRCTYPE,"p") OR LIMIT-TO(SRCTYPE,"j" ))
( TITLE-ABS-KEY ( EnterpriseArchitecture OR EnterpriseModel OR Enterprise ModellingOR Enterprise Modeling ) ANDNOT ALL ( "security" OR "gov-ernance" OR "government" OR"mining" OR "re-engineering" OR"BPR" OR "Six Sigma" OR "SOA"OR "mashups" OR "Web Service"OR "Cloud" OR "data warehouse"OR "ERP" OR "SAP" OR "DigitalMedia" OR "MIS" OR "workflow"OR "RFID" OR "sensor network"OR network management OR "LAN"OR "database" OR "network in-frastructure" OR "NAS")) AND( LIMIT-TO(DOCTYPE,"cp" ) ORLIMIT-TO(DOCTYPE,"ar" ) ORLIMIT-TO(DOCTYPE,"ch" ) ORLIMIT-TO(DOCTYPE,"bk" ) ) AND (LIMIT-TO(SUBJAREA,"COMP" ) ) AND( LIMIT-TO(LANGUAGE,"English" )) AND ( LIMIT-TO(SRCTYPE,"p" )OR LIMIT-TO(SRCTYPE,"j" ) )
”ERP” OR ”SAP” OR ”Digital Media” OR ”MIS” OR OR”RFID” OR ”sensor network” OR ”network management”OR ”LAN” OR ”database” OR ”network infrastructure” OR”NAS”) IN TextContent)] is designed to eliminate EM Pub-lications that are irrelevant for this study. We consider Pub-lications that solely focus on workflow, process mining, secu-rity, and infrastructure related topics as not much relevantto organisational decision making. Two constraints are de-fined as part of QualityCriteria, they are: (i) paper shouldbe aligned with the ResearchProblem RP1 and (ii) papershould be cited by at least one refereed paper (excludingself-citation) if it is published before 2014. The former qual-ity criterion checks the relevance and the latter validatesminimum acknowledgment from research community.
Process step Define Search Protocol process step also de-fines the StudyTemplate for conducting reviews. In thiscase, we select a set of attributes suggested by by EBSEResearch Group [14] that includes Authors and their in-stitute and country, and other MetaData such as publi-cationDate, subjectArea, language, citationCount, source-Type and sourceName. In addition, we consider two at-tributes namely EM Technique Referred and Summary ofthe Publication to capture the list of cited EM techniquesand high-level description of the publication respectively.
The final step of planning phase, i.e., Validate Review Pro-tocol step, validates structural and quality conformances.The structural conformances are validated while instantiat-ing SRMModel and the quality conformances are evaluatedthrough manual review.
This phase executes the review plan described in Fig. 6through two logical steps as described in section 3.2. Thefirst logical step select publications explores Scopus, Sci-enceDirect, Web of Science, ACM Digital Library and IEEEXplore DigitalLibrary and finds relevant Publications byquerying transformed search strings of associated Inclusion-Criteria and ExclusionCriteria wherein the process step Se-lect Digital Library transform InclusionCriteria and Exclu-sionCrteria into search strings. As an example, we showthe transformed search strings for Scopus DigitalLibrary in
Figure 7: Progression in execution phase
Figure 8: A representative view of Scopus
Table 3. The transformed search strings of other DigitalLi-brary can be found in appendices section2
4.2 Execution PhaseThe process step Select Publication using Inclusion Cri-
teria find the Publications that satisfy the InclusionCrite-ria expression. A sample representation of Publications se-lected from Scopus DigitalLibrary is depicted in Fig. 8. Theprocess step Eliminate Publication using Exclusion Criteriaeliminates Publications from list of selected Publications.For example, the Publication with id 3 will be eliminatedfrom list of Publication depicted in Fig. 8 as its title con-tains the Phrase ’mashups’ (a Phrase in Exclusion Criteriaexpression).
The count of identified Publications for each process stepis depicted in Fig. 7. The InclusionCriterion collectively se-lect 7622 Publications (with multiple duplicate entries) andExclusionCriteria short-list 1855 Publications. Supportedadvanced search capabilities of DigitalLibrary are exploitedto find Publications.
The QualityCriteria are evaluated by studying title, Meta-Data, Abstract and Introduction sections of the Publica-
2available at https://www.researchgate.net/publication/305481180 Appendices of Paper Enterprise Modeling as anAid to Complex Dynamic Decision Making A SystematicMapping Study
Figure 9: A representation of Corpus
tions selected after evaluating ExclusionCriteria. From ear-lier example, the Publication with id 4 will be eliminated byQualityCritera as the Publication published before 2014 anddoesn’t have any citationCount. In this study, the Quality-Criteria selects 173 Publications. Finally, the Validate Pub-lication step validates selected publication and constructs aCorpus for detailed study. A sample Corpus with selectedPublications from initial Publication list (depicted in Fig. 8)is shown in Fig. 9 for illustration purpose.
The second logical step conduct study is performed on 173Publications (this number is a high but we continued withit as our motive was to cover the breadth of the topic). Thestudy is conducted using SMS review style and the reviewfindings are captured in the form of Study by instantiatingStudyTemplate defined in Protocol Definition phase.
4.3 Synthesis PhaseThe synthesis phase analyzes all Study model captured
in execution phase and answers ResearchQuestions formu-lated in the SearchProtocol in the form of a Report. The fi-nal outcomes of the review synthesis answering two researchquestions are briefly discussed below:
Answers to RQ 1 - What are the papers on EnterpriseModeling (EM) and Enterprise Architecture (EA) that focuson organisation modelling?
As shown in Fig. 7, 173 Publications satisfy the criteriadefined in SearchProtcol SP1. The complete list of Publi-cations can be found in appendices section and the detailedreport on review findings for ResearchQuestions RQ1 andRQ1.1 can be found in [2]. The consolidation of EM tech-niques attribute of 173 publication studies collectively re-port 29 EM techniques as an answer to sub-question - Whatare the EM techniques cited by those publications? Table 4describes identified EM techniques. The useful referencesassociated with identified EM techniques are also listed inappendices section.
We conducted trend analysis on final selection of 173 pub-lications by considering the attributes values of of Studymodels such as country, institute, publicationYear, etc. Inbrief, these publications are contributed from 35 countries
Table 4: Identified EM Approaches
Zachman Framework, ArchiMate, Department of DefenseArchitecture Framework (DoDAF), The British Ministryof Defence Architecture Framework (MoDAF), The OpenGroup Architecture Framework (TOGAF), ARIS, ExtendedEnterprise Modeling Language (EEML), Enterprise Knowl-edge Development (EKD),MoKI, Knowledge Acquisitionin automated specification (KAOS), i*, Business Moti-vation Model (BMM),Business Process Model and Nota-tion (BPMN), Integrated enterprise modeling (IEM), Uni-fied Modeling Language (UML),Perdue Enterprise ReferenceFramework (PERA), GRAI Integrated Methodology (GIM),Computer Integrated Manufacturing Open Systems Archi-tecture Framework (CIMOSA),Generalized Enterprise Ref-erence Architecture and Methodology (GERAM), Designand Engineering Methodology for Organizations (DEMO),Multi-Perspective Enterprise Modelling (MEMO), Integra-tion DEFinition (IDEF), European Interoperability Frame-work (EIF), Semantics of Business Vocabulary and Rules(SBVR), System Dynamics, Unifed Enterprise Modeling Lan-guage (UEML), Systemic Enterprise Architecture Methodol-ogy (SEAM), Event-driven process chain (EPC), and Refer-ence Model of Open Distributed Processing (RM-ODP)
involving 161 institutions in time span of 1987 to 2016. Thecomplete Report of this review, which is conducted usingconventional SMS methodology, can be found in[2].
This experiment was conducted for illustrating and val-idating proposed approach. We used the RLMModel andSRMModel meta-models and SR method described in sec-tion 3 to conduct this SR. In this section, we illustrated pro-posed approach by repeating the SR which was conductedusing conventional SMS methodology. The benefits of pro-posed approach is discussed in section 6. Prior to illustratethe benefit, we briefly discuss the implementation optionsand out implementation strategy that we have chosen forthis case study.
5. IMPLEMENTATIONThe proposed approach realises the systematic review pro-
cess as a series of model creation, model instantiation, modeltransformation and model evaluation activities.
In particular, the Protocol Definition phase is modeling ofResearchQuestions and SearchProtocol as shown in Fig. 6.The explore publication logical step of Execution phase com-prises three activities: i) transform InclusionCriteria andExclusionCriteria into DigitalLibrary specific search stringsas illustrated in Table. 3, ii) apply transformed search stringsinto DigitalLibraries to find relevant Publications, and iii)construct a new Corpus with set of Publications that satifythe condition (InclusionCriteria AND NOT(ExclusionCriteria)AND QualityCriteria). The logical step conduct study is forreading Publications from Corpus and instantiating StudyTem-plate. The synthesis phase navigates and analyses Studymodels and produce a Report. The possible implementationoptions of these activities and our implementation strategiesare highlithed in the Table 5.
As depicted in the table, we are using xModeler5 for cre-ating models (by instantiating LRMModel and SRMModel)in Protocol Definition phase. However, one can use anymeta-modeling tool to realise proposed approach. In ourrealisation, we used xModeler to represent the Corpus of173 Publications. A sample Corpus with two Publications
5http://www.eis.mdx.ac.uk/staffpages/tonyclark/Softw-are/XModeler.html
Table 5: Implementation Options
ActivityImplementation Op-tions
Chosen Op-tion
Planning Model Re-
search Questionand ResearchProtocol
1. Use of modelingtool. 2. Spreadsheet
xModeler
Execution
Transformsearch criteriainto searchstrings
Automated trans-lation using M2T3
technique. 2 Manualtransformation
Manualtransforma-tion
Evaluate meta-data evel andphrase existenceexpressions
1. Manual, 2. Useof digital library spe-cific search capability,3. Use generic websearch capability suchas Python Scrapy4
Digitallibrary spe-cific searchcapability
Execute mean-ing existenceexpression
1.Manual interpreta-tion, 2. Sophisticatedmachine learning algo-rithm, deep learning
Manual In-terpretaion
Create and up-date Corpus
1. Use of model-ing tools, 2. use ofSpreadsheet
xModeler
Read publica-tion
Manual Manual
InstantiateStudy Template
1. Use of Spreadsheet,document, 2. Use ofmodeling tools
use ofxModeler
Synth
esis
Evaluatiion ofStudies andprepare Report
1. Manual interpreta-tion. 2. Modeling toolassited interpretation
xModelerassisted in-terpretationand man-ual reportgeneration
is shown in Fig. 9 for illustration purpose. We used semi-automated technique to apply search strings on digital li-braries and manual effort to evaluate quality criteria. Thepopulation of Corpus from selected publications, instantia-tion of StudyTemplate by reading Publications and syntesisof Study models to produce Report activities are manual atpresent. However, population of Corpus and synthesis ac-tivity can be automated to a large extent by using scriptinglanguage supported by xModeler.
6. ANALYSES AND DISCUSSIONWe support iterative and incremental approach, use SMS
and SLR methodologies across iterations, and leverage mod-eling and model processing technique in a systematic form.Further we visualise a literature review process as step-wiseinstantiation of SRMModel. The planning activity is instan-tiation of SRMModel and defining new meta-model (e.g.,meta-model type study template), execution is instantia-tion of Publications of RLMModel and population of StudyTemplate, and synthesis is model-synthesis(of instances ofSRMModel and Study Templates).We conducted 2 literature reviews using conventional SLR
and SMS methodologies and using proposed approach. Ourobservations is that the iterative and incremental approachimprove the precision and the use of SMS and SLR in con-cert help in managing complex research questions. The useof snowballing along with search-string based approach en-sure better search coverage (and thus manage threats to va-lidity better). The model-based realisation improve specificsto use SR effectively. The classification of validations and
conceptualisation structural conformance improve the scopefor automation (hence the reduction of effort and time) andreduce error-proneness. Moreover, seamless traceability be-tween various artefacts reduce the effort for correlation andsense-making; higher abstraction of the artifacts improvethe visualization (and thus interpretation); analysable rep-resentation of review artefacts enable rigorous analysis andcomplex synthesis. The use of meta-modeling as study tem-plate also improve the automated synthesis.
We evaluated proposed approach based on its ability toensure the quality and usability properties described in table1. Our analysis is illustrated below:
Biasness: Biasness on study identification, literature cor-pus, quality evaluation and data extraction (as shown intable 1) significant impact the quality of review outcome.Evaluating biasness in conventional SR mostly manual andthus effort- and time- intensive activity. Proposed approachenforces constraints on publication database and validatesselection counts (structural conformance) from each digitallibrary to manage study identification biasness. The liter-ature corpus biasness can be evaluated using refers associ-ation and finding list of publications from an author or agroup of authors. We say a publication corpus is biasedif the large percentage of publications are connected withrefers association and/or large percentage of publicationsin a corpus are from same group of authors. Data extrac-tion biasness can be managed by enforcing Study Templatedefinition.
Threats to validity: Providing convincing justificationto threats to validity for study identification, literature cov-erage, quality evaluation and data extraction improves theconfidence of a review outcome. We argue that proposedapproach provides better support to manage threats to va-lidity. The constraints can be defined at meta-model leveland ensured through model-validation techniques. For ex-ample, minimum number of selected publications for eachstage i.e., number of publications linked with inclusionFinds,exclusionFinds and select associations in RLMModel.
Repeatability: The proposed approach is a refinementof conventional SRs thus it is repeatable. The mode-basedrealisation further improves the repeatability as the processis essentially series of model instantiation, model navigation,model extraction and model validation activities.
Structured Representation: SMS and SLR method-ologies use table and text based template to represent ex-tracted information. These representation techniques are re-strictive in two senses - a) limitation of the format: Text hasno structure and thus interpretation of a text is vulnerableto human interpretation; the table is restrictive in represent-ing complicated relationships. We propose a meta-modelingtechnique for information representation, and b) synthesis ofcollected data: unstructured and semi-structured data hasless power than models. We propose to use OCL, QVT andModel-to-Text transformation techniques to validate cap-tured information and transform them into appropriate form.
Traceability: Establishing traceability between variousartefacts in conventional SR is managed by individual reviewteam. In particular, there is no specific recommendation orguideline. Poor traceability significantly impacts the reviewtime and quality. The application of model-based approachhelps in establishing the traceability within and across iter-ations of a literature review.
Visualization: The model-based approach for represent-
ing review artefacts, such as studies, enables model-to-modeland model-to-text transformation. Thus automated trans-formation of review artefacts into a form that can be usedby visualization tools is possible with proposed approach.
In addition to these standard properties, two other prop-erties are equality important for SR and proposed approachscore better as compare to conventional SR:
Usability: Conventional SR mostly uses a linear approachstarting from problem statement and research question toreview output. This is an effective approach for expert liter-ature review practitioners but it is unlike that novice practi-tioners will get everything right in first attempt. We proposean iterative style review where practitioners can start withsimple research question adopting one review style and sub-sequently they can shift to more complex research questionswith different review style.
Accountability and change management: Model basedapproach improves the accountability of the review processby storing additional information in model. The changemanagement can be introduced by supporting model ver-sion management. These are well-researched area in model-ing community.
7. CONCLUSIONIn this paper, we presented a model-based realisation of
SR in terms of meta-models and a precise process definition.We illustrated the same using a case study which was alsorepeated using TLR, conventional SMS and proposed ap-proach. We found that the review outcomes with SMS weresignificantly better than with TLR. The proposed approachled to even better outcomes in that:
• Availability of all artefacts in a model form enablesanalysis, easy navigation as well as traceability thusreducing burden on reviewers and review time. It alsohelped in establishing properties such as biasness, cor-pus quality, validity of threats etc. Use of visualisa-tion, model-transformation and validation techniquesimproved precision in synthesis and report generation.
• Combining necessary concepts of SR in a meta-modelform improves the usability. Incorporating snowballingtechnique into SR led to better search coverage. Sup-porting a hybrid approach through integration of SLRand SMS led to improved precision. Precise defini-tion of process steps, semi-automated model valida-tion, and model-synthesis led to improved quality.
• Enabling iterative and incremental approach for con-ducting SR helps in managing complexity of SR
We believe the proposed approach serves as a foundationfor a robust literature review tool. An implementation strat-egy using xModeler and advanced search capability of digi-tal libraries is briefly highlighted in this paper. We aim toextend SLRTool [4] with the proposed approach. In this re-gard, we have evaluated the possibility of using model-basedtechniques such as meta-modeling, model-validation, model-visualisation, model-to-text and model-to-model transfor-mation in prospective process steps. We are now exploringthe use of natural language processing and text mining tech-niques to further improve the automation. We also have planto introduce this approach to the researchers from industryand academia who have less experience in SR production assuggested in [20].
8. REFERENCES[1] M. A. Babar and H. Zhang. Systematic literature
reviews in software engineering: Preliminary resultsfrom interviews with researchers. In Proceedings of the2009 3rd International Symposium on EmpiricalSoftware Engineering and Measurement, pages346–355. IEEE Computer Society, 2009.
[2] S. Barat, V. Kulkarni, T. Clark, and B. Barn.Enterprise modeling as a decision making aid: Asystematic mapping study. In 9th IFIP WG 8.1Working Conference on The Practice of EnterpriseModeling (PoEM), Skovde, Sweden. 2016.
[3] S. Barat, V. Kulkarni, T. Clark, and B. Barn. Asimulation-based aid for organisationaldecision-making. In ICSOFT-EA 2016: 11thInternational Conference on Software Engineering andApplications. 2016.
[4] B. Barn, F. Raimondi, L. Athiappan, and T. Clark.Slrtool: a tool to support collaborative systematicliterature reviews. 2014.
[5] J. Biolchini, P. G. Mian, A. C. C. Natali, and G. H.Travassos. Systematic review in software engineering.System Engineering and Computer ScienceDepartment COPPE/UFRJ, Technical Report ES,679(05):45, 2005.
[6] D. Bowes, T. Hall, and S. Beecham. Slurp: a tool tohelp large complex systematic literature reviewsdeliver valid and rigorous results. In Proceedings of the2nd international workshop on Evidential assessmentof software technologies, pages 33–36. ACM, 2012.
[7] P. Brereton, B. A. Kitchenham, D. Budgen,M. Turner, and M. Khalil. Lessons from applying thesystematic literature review process within thesoftware engineering domain. Journal of systems andsoftware, 80(4):571–583, 2007.
[8] J. C. Carver, E. Hassler, E. Hernandes, and N. A.Kraft. Identifying barriers to the systematic literaturereview process. In Empirical Software Engineering andMeasurement, 2013 ACM/IEEE InternationalSymposium on, pages 203–212. IEEE, 2013.
[9] A. M. Fernandez-Saez, M. G. Bocco, and F. P.Romero. Slr-tool: A tool for performing systematicliterature reviews. In ICSOFT (2), pages 157–166,2010.
[10] E. Hassler, J. C. Carver, D. Hale, and A. Al-Zubidy.Identification of slr tool needs–results of a communityworkshop. Information and Software Technology,70:122–129, 2016.
[11] E. Hernandes, A. Zamboni, S. Fabbri, and A. D.Thommazo. Using gqm and tam to evaluate start-atool that supports systematic review. CLEI ElectronicJournal, 15(1):3–3, 2012.
[12] C. Hewitt. Actor model of computation: scalablerobust information systems. arXiv preprintarXiv:1008.1459, 2010.
[13] S. Jalali and C. Wohlin. Agile practices in globalsoftware engineering-a systematic map. In GlobalSoftware Engineering (ICGSE), 2010 5th IEEEInternational Conference on, pages 45–54. IEEE, 2010.
[14] S. Keele. Guidelines for performing systematicliterature reviews in software engineering. In Technicalreport, Ver. 2.3 EBSE Technical Report. EBSE. 2007.
[15] B. Kitchenham. Procedures for performing systematicreviews. Keele, UK, Keele University, 33(2004):1–26,2004.
[16] B. Kitchenham and P. Brereton. A systematic reviewof systematic review process research in softwareengineering. Information and Software Technology,55(12):2049–2075, 2013.
[17] B. Kitchenham, R. Pretorius, D. Budgen, O. P.Brereton, M. Turner, M. Niazi, and S. Linkman.Systematic literature reviews in softwareengineering–a tertiary study. Information andSoftware Technology, 52(8):792–805, 2010.
[18] B. A. Kitchenham, T. Dyba, and M. Jorgensen.Evidence-based software engineering. In Proceedings ofthe 26th international conference on softwareengineering, pages 273–281. IEEE Computer Society,2004.
[19] V. Kulkarni, S. Barat, T. Clark, and B. Barn. Towardovercoming accidental complexity in organisationaldecision-making. In Model Driven EngineeringLanguages and Systems (MODELS), 2015ACM/IEEE 18th International Conference on, pages368–377. IEEE, 2015.
[20] M. Lavallee, P.-N. Robillard, and R. Mirsalari.Performing systematic literature reviews with novices:An iterative approach. Education, IEEE Transactionson, 57(3):175–181, 2014.
[21] C. Marshall, P. Brereton, and B. Kitchenham. Toolsto support systematic reviews in software engineering:a cross-domain survey using semi-structuredinterviews. In Proceedings of the 19th InternationalConference on Evaluation and Assessment in SoftwareEngineering, page 26. ACM, 2015.
[22] K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson.Systematic mapping studies in software engineering.In 12th international conference on evaluation andassessment in software engineering, volume 17, pages1–10. sn, 2008.
[23] K. Petersen, S. Vakkalanka, and L. Kuzniarz.Guidelines for conducting systematic mapping studiesin software engineering: An update. Information andSoftware Technology, 64:1–18, 2015.
[24] M. Riaz, M. Sulayman, N. Salleh, and E. Mendes.Experiences conducting systematic reviews fromnovicesı£¡ perspective. In Proc. of EASE, volume 10,pages 1–10, 2010.
[25] J. Webster and R. T. Watson. Analyzing the past toprepare for the future: Writing a. MIS quarterly,26(2):13–23, 2002.
[26] C. Wohlin. Guidelines for snowballing in systematicliterature studies and a replication in softwareengineering. In Proceedings of the 18th InternationalConference on Evaluation and Assessment in SoftwareEngineering, page 38. ACM, 2014.
[27] H. Zhang and M. A. Babar. An empirical investigationof systematic reviews in software engineering. InEmpirical Software Engineering and Measurement(ESEM), 2011 International Symposium on, pages87–96. IEEE, 2011.
[28] H. Zhang, M. A. Babar, and P. Tell. Identifyingrelevant studies in software engineering. Informationand Software Technology, 53(6):625–637, 2011.
