A Rule-based Framework for Enhancing Architectural

A Rule-based Framework for Enhancing Architectural Decision Guidance

Doctoral DefenceMohsen AnvaariMay 27, 2016

1

Agenda

• Introduction• ResearchDesign• Contributions• Discussion• DirectionsforFutureWork

2

In a Nutshell

Ultra-Large-ScaleSystems

SoftwareEcosystems

CognitiveBiases

ResearchMethods

ArchitecturalDebt

ArchitecturalDecision-Making

ArchitecturalDecisionGuidance

SoftwareQuality

Agility

InformationExtractionMethods

Rule-basedFrameworkforEnhancingArchitectural

DecisionGuidance

havemorepotentialtocreate

haveimpacton

developedwithin

causedbypoor

diminishes

negativelyaffect

Improvesqualityandspeedof

enhances

develop andevaluate

compromises

accumulates

increases

increase

employs

requiremoresystematic

3

Architectural Decision Making Process

4

ArchitecturalIssues

AlternativeSolutions

DecisionDrivers

QualityAttributes BusinessFactors

DecisionOutcome(SelectedAlternative)

Rationale

have

relate

has

isatrade-offbetween

extends extends

affects(positively/negatively)

Identifying

ATAM,CBAM,SAAM,etc.

DecisionVP,SEURAT,etc.

Intuitiveandexperiencebased

CognitiveBiases

Cognitive Biases

Cognitive bias refers to a systematic pattern of deviation fromrationality in judgment.

The availability heuristic– Only 46% of architects search for alternative options ifthey already have a solution in mind (van Heesch andAvgeriou, 2011)

The affect heuristic– 68% of architects prefer well-known solutions in favor ofunknownalternatives (van Heesch and Avgeriou, 2011)

5

An Example

ArchitecturalIssue1 Whatshouldthe integrationstylebe? ESB

ArchitecturalIssue2 Whatshouldthe integrationproduct be? BizTalk

ArchitecturalIssue3 Whatshouldthedeploymentstrategyofintegration solutionbe?

Private Cloud

…

Project:IntegratingApplications inEnterpriseX

6

Which critical architectural issue is missing?• What should the ESB implementation strategy or pattern be? Global ESB or

ESB Gateway or Federated ESB?

The default alternative (global ESB) has been implemented (subconscious decision)• Scalability is compromised

Three years later• Global ESB has become the single point of failure = Architectural Debt

Architectural Debt

• Technical Debt (TD) refers to technical compromises that canprovide short-term benefit for project stakeholders but mayhurt the long-term health of a software system

• Architectural debt (AD) is a type of TD which is caused byarchitecture decisions that consciously or subconsciouslycompromise software quality attributes

7

Research Goals and Questions

Goal1. This thesis explores the current state of architecturaldecision making process: in companies that develop large anddistributed software and in the literature that propose tools forsupporting practitioners

RQ1. What is the attitude of large-scale enterprises in makingand reusing architectural decisions and how do available toolsand research prototypes support them?

8

Research Goals and Questions (cont.)

Goal2. By grounding on the advances in information extractionand natural language processing domains, it develops a rule-based framework called SADGE that enhances architecturalguidance from architecture-related documents in a rapid way.

RQ2. How can a framework be established to developarchitectural decision guidance from architecture-relateddocuments in a rapid way?

9

Research Goals and Questions (cont.)

Goal3. The thesis evaluates the efficiency and effectiveness ofthe framework by conducting a case study on documentsreceived from a telecommunication company, and by performingexperiments on both IT students and expert IT/softwarearchitects.

RQ3. How efficient and effective will such a framework be indeveloping architectural decision guidance?

10

Research Methods

11

ResearchMethod

Setting ResearchQuestions

Papers

QualitativeInterviews

Interviewing8informants(ITmanagersandarchitects)from6organizationsintheNorwegianelectricityindustry

RQ1 Paper1

Online Survey Aninter-rateragreementstudyon37ITexperts RQ1 Paper5

LiteratureReview

Analyzing5toolsandresearchprototypes RQ1 Paper2

DesignScience Developingandtestinganinformation extractionframework

RQ2 Paper3,4

CaseStudy Using3architecturaldocuments fromatelecommunication companytotestandevaluatetheframework

RQ2,3 Paper4

Experiment1 Evaluatingtheframeworkwith19ITstudents RQ3 Paper3

Experiment2 Evaluatingtheframeworkwith21expertarchitectsfrom5Norwegianorganizations

RQ3 Paper4

Research Scope

Solution Problem

Theory Practice

RQ1 P2

RQ1 P1

RQ1 P5

RQ2 P3,P4

RQ3 P3,P4

Inception

Development Evaluation

Inception

12

Research Contributions

C1. An overview of the state-of-the-art and state-of-the-practicein making and reusing architectural decisions.

C2. SADGE, a rule-based framework for enhancing architecturaldecision guidance from architectural knowledge in projectdocuments and in the domain literature.

C3. Results of empirical evaluation of developing architecturaldecision guidance by employing a rule-based framework.

13

Contribution 1 State-of-Practice

• Most of the companies are not using well-known academicapproaches such as ATAM, they are rather using their ownprocedures

• There is high potential and need for reusing the architectural issuesand decisions across various projects inside an enterprise or acrosssimilar enterprises in a domain– In 63% of cases, architects stated that “no previous similar decisions in

their projects to compare the current decision making against” was achallenge for their decision-making (Tofan et. al, 2013).

• The relationships among the actors of a software ecosystem (SECO)could significantly affect the architectural-decision making processin each of the actors

14

Contribution 1Expert Agreement on Associating Architectural

Issues with Quality Attributes

ICC:Intra-ClassCorrelationsKalpha:Krippendorff's alpha

15

Category of subjects ICC Kalpha All scenarios together 0.12 0.09

Scenarios

Scenario1 0.12 0.09 Scenario2 0.15 0.10 Scenario3 0.04 0.01 Scenario4 0.06 0.03 Scenario5 0.25 0.09

Quality Attributes

Functionality 0.14 0.09 Security 0.15 0.07

Reliability 0.13 0.09 Efficiency 0.10 0.05 Usability 0.11 0.04

Maintainability 0.08 0.05 Portability 0.16 0.11

0ICC,Kalpha 1

NoagreementFullagreement

Contribution 1State-of-Art

Available tools and research prototypes havemade significant contributions in the area ofarchitectural knowledge capturing, but stillrequire a number of extensions so that thecaptured knowledge can serve as architecturaldecision guidance

16

Contribution 2Semi-automated Design Guidance Enhancer (SADGE)

17

ArchitecturalIssuesIdentifying

Intuitiveandexperiencebased

CognitiveBiases

Company documents

Software Architecture

Domain literature

Electronic Documents (PDF, DOC, TXT, etc.)

Architectural Decisions

Step1

Step2

O3

Catalogue Construction

Catalogue of Terms (CoT)

High Priority

LowPriority

Annotation Rules

Automatic Annotator

(AA)

Guidance Generator

Annotated SentencesStep3

to step1, for annotating the next document

to step4, when all documentsare annotated


«AnnotationRule»

Conditionif

(asentencecontainsatleastoneofthetermsfromthecatalogueofhighpriorityterms)

or(containsatleasttwotermsfromthecatalogueoflowpriorityterms)

18

Action

annotateitasanarchitecturalissue


«Catalogue of Terms (Default Version)»

19


«Catalogue Construction Procedure»

20

Target Documents

New Clue Terms

1- Manually find clue terms from

10% of documents

Task Work Product Input/Output

Default Catalog of

Terms 2- Find

synonym of each term

in WordNet

Ad-hoc Catalog of Terms +

Add new terms

Contribution 3Evaluation of SADGE: Metrics

21

Relevant Sentences

Annotated Sentences

Non-relevant Sentences

Recall = Effort Reduction=

Contribution 3Evaluation of SADGE

22

Document Effort Reduction (%)

Recall (%)

Doc1 66 89 Doc2 51 72 Doc3 63 73 Mean 60 78

SD 7.93 9.53 ResultsofCaseStudy


23

SADGE

SADGE

Manual

Automa/c

Semi-Automa/c


“Several studies have shown that human decision makers are inferior to a predictionformula even when they are given the score suggested by the formula. They feelthat they can overrule the formula because they have additional information aboutthe case, but they are wrong more often than not” (Kahneman, 2011)

24

Inlinewiththeempiricalresultsinthefieldofpsychologyofdecision-making:

Approach Processing Time (min) Recall (%)

Manual 14 34 Automatic 0.03 53.5

Semi-automatic 7 32

Comparison p-value

Document1 Automatic vs. Manual 0.000225 Automatic vs. Semi-automatic 0.0000908

Document2 Automatic vs. Manual 0.003323 Automatic vs. Semi-automatic 0.0002254

Approach Processing Time (min)

Recall (%)

Manual 9 38 Automatic 0.03 86

Semi-automatic 3 55 ResultsofexperimentonITstudents Resultsofexperimentonexpertarchitects

ResultsofK-Stest

Strengths and Weaknesses of Research Outcome

Strengths• Balancing agility and architecture-awaredevelopment• Ad-hoc construction method for developing the Catalogue of

Terms• Informing the practitioners that the quality attributes they

consider irrelevant or less relevant for an architectural issue,might be very relevant in the opinion of other practitioners

Weaknesses• Eliminating the context• Treatingall architectural issueswith a general attitude

25

Limitations of Research Studies

• Generalization of results of exploratory study and casestudy

• Just one of the authors has annotated the referencedocuments in the case study

• Participants might interpret an "architectural issue"differently from researchers

• Inconsistence interpretations of "architectural issue" and"quality attribute" between researchers and participants

26

Directions for Future Study

• Supporting more natural languages

• Developing domain-specific Catalogues of Terms

• Creating data sets for training and testinginformation extraction framework

• More in-depth understanding of architects'rationale for making architectural decision andthe role of cognitivebiases in their decisions

27

PapersP0 Smart Grid SoftwareApplications as an Ultra-Large-Scale System - Challenges for Evolution.Mohsen Anvaari, Daniela S. Cruzes, Reidar Conradi.In Proc. Innovative Smart Grid Technologies (ISGT), 2012 IEEE PES.P1 Architectural Decision-Making in Enterprises - Preliminary Findings From an Exploratory Study inNorwegian Electricity Industry.Mohsen Anvaari, Reidar Conradi, Letizia Jaccheri.In Proc. The 7th European Conference on Software Architecture, ECSA 2013.P2 Towards Reusing Architectural Knowledge as Design Guides - Functional Requirements, Tool Analysisand Research Roadmap.Mohsen Anvaari, Olaf Zimmermann.In Proc. The 26th International Conference on Software Engineering and Knowledge Engineering, SEKE2014.P3 Semi-automated Design Guidance Enhancer (SADGE): A Framework for Architectural Guidance

Development.Mohsen Anvaari, Olaf Zimmermann.In Proc. The 8th European Conference on Software Architecture, ECSA 2014.P4 Rule-based Extraction of Architectural Issues fromSoftware Architecture Documents.Mohsen Anvaari, Olaf Zimmermann, Carl-Fredrik Sørensen.Submitted forpublication.P5 Associating Architectural Issues with Quality Attributes: A Survey on Expert Agreement.Mohsen Anvaari, Carl-Fredrik Sørensen, Olaf Zimmermann.Submitted forpublication.

28