Embed Size (px)
DESCRIPTION
The complexity of modern computer systems is often comparable with the one of biological systems. As much as this complexity can be effectively hidden from the end-user, it is inherently absorbed in the design of the system. Software Architecture is an effective design abstraction that allows designers to divide and conquer the complexity. A modern way of looking at the Software Architecture is to see it as a set of principal design decisions. The design of Software Architecture for large and complex systems often requires expertise exceeding what can be delivered by the individual software architect, therefore successful design relies on effective collaborative decision making within the design team of diverse domain experts. We tackle the problem of collaborative decision making in the software architecture design teams by proposing the decision argumentation viewpoint extension to the architecture description standard. Its main purpose is to support fine-granular decision argumentation modeling. Within the viewpoint we devise the architecture decision consensus lifecycle, and design issue choice state machine that enable precise characterization of the decision state. Based on the argumentation viewpoint we define an analytical framework designed to estimate the structural and temporal characteristics of decision models. The framework comprises fifteen metrics and offers a comprehensive introspection into the state and dynamics of the decision making process. Building upon this foundation we designed and implemented the Software Architecture Warehouse (SAW) - the tool to assist software architects in collaborative decision/making during the architecture design workshops. SAW features low-latency, structured architecture decision capturing and decision consensus management. Furthermore, the Software Architecture Warehouse is accompanied by the implementation of the aforementioned decision argumentation metrics framework. Finally, we evaluate the framework by applying it on the decision spaces recorded during the masters course on Software Architecture and Design. We conclude with an interpretation of differences observed between the workshops assisted with the use of the Software Architecture Warehouse and those supported by EtherPad, an alternative unstructured collaborative editor. Category
Citation preview
Collaborative Software Architecture Decisions: Structure
and Dynamics
Marcin Aleksander NowakArchitecture Design and Web Information Systems Engineering Group
Dissertation CommitteeProf. Dr. Mehdi Jazayeri Prof. Dr. Michele Lanza Prof. Dr. Patricia LagoProf. Dr. Olaf Zimmermann
Under supervision ofProf. Dr. Cesare Pautasso
Collaborative Software Architecture Decisions: Structure
and Dynamics
Marcin Aleksander NowakArchitecture Design and Web Information Systems Engineering Group
Dissertation CommitteeProf. Dr. Mehdi Jazayeri Prof. Dr. Michele Lanza Prof. Dr. Patricia LagoProf. Dr. Olaf Zimmermann
Under supervision ofProf. Dr. Cesare Pautasso
Software Complexity
4
Software Architecture: Boxes and Arrows
7
Software Architecture
8
Set of principal design decisions about the system
Richard N. Taylor, Nenad Medvidovic, and Eric M. Dashofy. Software Architecture - Foundations, Theory and Practice. Willey, 2009
9
The life of a software architect isa long and rapid succession of suboptimal design decisions
taken partly in the dark *
Philippe Kruchten
Good decisions
InformedSmart
10Panel discussion on Architecture Decisions, SATURN 2013 Minneapolis, Minnesota
Good decisions
InformedSmart
11
Situational Awareness
16
Recognize & MonitorPerception
Make SenseComprehension
Predict FutureProjection
* Endsley, M.R. Toward a theory of situation awareness in dynamic systems. Human Factors, 1995, 37(1), 32–64
Assumption #1
Situational Awareness
Good Decisions
17
[…] 86% of architectural decisions are group decisions
18* Difficulty of Architectural Decisions – A Survey with Professional Architects” Dan Tofan, Matthias Galster, Paris Avgeriou, ECSA 2013
Good decisions
InformedSmart
Consensual
20
Good decisions
InformedSmart
Consensual
21
Team Situational Awareness
The degree to which every team member
possesses the SA required for his or her responsibilities *
22* Endsley, M.R. (1995b). Toward a theory of situation awareness in dynamic systems. Human Factors 37(1), 32–64
Team Situational Awareness
The degree to which every team member
possesses the SA required for his or her responsibilities *
23* Endsley, M.R. (1995b). Toward a theory of situation awareness in dynamic systems. Human Factors 37(1), 32–64
Extension of Assumption #1
Team Situational Awareness
Good Collaborative Decisions
24
26
COLLABORATIVE SOFTWARE ARCHITECTURE DECISIONS
28
Concerns
Fragmentation of expertise and decision power
29
Concerns
Inefficient deliberation and conciliation
30
Concerns
Chaotic brainstorming
32
Research Problem #1
Collaborative Architecture Design Decision Consensus
How to support collaborative software architectural decision making?
Q:
33
Research Problem #2
Quality of the Collaborative Architecture Decisions
How to identify, and quantify properties of good, collaborative design decision making process?
Q:
34
Thesis
Support for the low latency, structured architecture decision argumentation
improves the quality of the decision making process.
35
Approach
Theoretical Practical Evaluation
RQ1
Argumentation Viewpoint SAW Formative
Evaluation
RQ2Argumentation
Metrics Analyzer EmpiricalEvaluation
36
DECISION ARGUMENTATION VIEWPOINT
4+1 Architecture Views
37
Software Architecture
Logical
Process
Development
Physical
P.B. Kruchten. The 4+1 view model of architecture. Software, IEEE, 12(6):42–50, Nov 1995
Scenarios
Decision Viewpoints
39
Architecture Decision
Relationships
Involvement
Chronology
Force
Uwe van Heesch, Paris Avgeriou, and Rich Hilliard. A documentation framework for architecture decisions. Journal of Systems and Software 2012.
Decision Viewpoints
40
Architecture Decision
Relationships
Involvement
Chronology
Force
Argumentation
Marcin Nowak and Cesare Pautasso. Team situational awareness and architectural decision making with the software architecture warehouse. In volume 7957 of Lecture Notes in Computer Science, pages 146–161. Springer, 2013.
Argumentation viewpoint
Issue
addressesAlternative
addresses
solvesArchitecture Decision
RationaleConcern
depends upon
pertains
raises
justifies
Position
Action
StakeholderDecision Force
recommends
pertains states
Elementary Decision Model
42
Design Issue
Design Alternative
Design Alternative
Design Alternative
Design Decision
Elementary Decision Model
43
Design Issue
Design Alternative
Design Alternative
Design Alternative
Design Decision
Design Decision
Design Decision
Decision Argumentation Model
44
Design Issue
Design Alternative
Design Alternative
Design Alternative
Design Decision
Design Decision
Design Decision
Position: Positive
Position: Positive
Position: Positive
Position: Negative
Position: Negative
Argumentation view example
45
AccountAccess
Security
WS-Security
Web Services Security
Mechanism
Design Issue
HTTPS
Plain text
Architecture Decision Design Alternatives
Compatibility issues
Heavyweight
Unsecure!
Simple
Practice proven
Stakeholder Positions
Decision Lifecycle
47
AccountAccess
Security
WS-Security
Web Services Security
Mechanism
Design Issue
HTTPS
Plain text
Architecture Decision Design Alternatives Stakeholder Positions
No positions Aligned Colliding
Decision Lifecycle
48
AccountAccess
Security
WS-Security
Web Services Security
Mechanism
Design Issue
HTTPS
Plain text
Architecture Decision Design Alternatives
Practice proven
Stakeholder Positions
No positions Aligned Colliding
Decision Lifecycle
49
AccountAccess
Security
WS-Security
Web Services Security
Mechanism
Design Issue
HTTPS
Plain text
Architecture Decision Design Alternatives
Practice proven
Stakeholder Positions
No positions Aligned Colliding
Decision Lifecycle
50
AccountAccess
Security
WS-Security
Web Services Security
Mechanism
Design Issue
HTTPS
Plain text
Architecture Decision Design Alternatives
Compatibility issues
Heavyweight
Practice proven
Stakeholder Positions
No positions Aligned Colliding
Decision Lifecycle
51
AccountAccess
Security
WS-Security
Web Services Security
Mechanism
Design Issue
HTTPS
Plain text
Architecture Decision Design Alternatives
Compatibility issues
Heavyweight
Practice proven
Stakeholder Positions
No positions Aligned Colliding
Decision Lifecycle
52
AccountAccess
Security
WS-Security
Web Services Security
Mechanism
Design Issue
HTTPS
Plain text
Architecture Decision Design Alternatives
Compatibility issues
Heavyweight
Unsecure!
Simple
Practice proven
Stakeholder Positions
No positions Aligned Colliding
Decision Lifecycle
53
AccountAccess
Security
WS-Security
Web Services Security
Mechanism
Design Issue
HTTPS
Plain text
Architecture Decision Design Alternatives
Compatibility issues
Heavyweight
Unsecure!
Simple
Practice proven
Stakeholder Positions
No positions Aligned Colliding
Plain text
66
No Alternatives
Inconclusive Choice
Complete choice
Incomplete Choice
Conclusive Choice
Warring Choice
No Positions
Plain text
HTTPS
WS-Security
AccountAccess
Security
Web Services Security
MechanismHTTPS
Design Issue lifecycle
Design Issue lifecycle
Plain text
HTTPS
WS-Security
AccountAccess
Security
Web Services Security
MechanismHTTPS
No Alternatives
Inconclusive Choice
Complete choice
Incomplete Choice
Conclusive Choice
Warring Choice
No Positions
67
Design Issue lifecycle
Plain text
HTTPS
WS-Security
AccountAccess
Security
Web Services Security
MechanismHTTPS
No Alternatives
Inconclusive Choice
Complete choice
Incomplete Choice
Conclusive Choice
Warring Choice
No Positions
68
Design Issue lifecycle
Plain text
HTTPS
WS-Security
AccountAccess
Security
Web Services Security
MechanismHTTPS
No Alternatives
Inconclusive Choice
Complete choice
Incomplete Choice
Conclusive Choice
Warring Choice
No Positions
69
Design Issue lifecycle
Plain text
HTTPS
WS-Security
AccountAccess
Security
Web Services Security
MechanismHTTPS
No Alternatives
Inconclusive Choice
Complete choice
Incomplete Choice
Conclusive Choice
Warring Choice
No Positions
70
Summary
71
Design Issue
Design Alternative
Design Alternative
Design Alternative
Design Decision
Design Decision
Design Decision
Position: Positive
Position: Positive
Position: Positive
Position: Negative
Position: Negative
Architecture ArgumentationViewpoint
No Alternatives
Inconclusive Choice
Incomplete Choice
Conclusive Choice
Warring Choice
No PositionsDesign IssueLifecycle
Marcin Nowak and Cesare Pautasso. Team situational awareness and architectural decision making with the software architecture warehouse. ECSA 2013
Decision ArgumentationLifecycle
No positions Aligned Colliding
72
SOFTWARE ARCHITECTURE WAREHOUSELive and collaborative architecture decision making support
Software Architecture Warehouse
74
• Live collaborative architecture decision support– Centralized– Live synchronized– Handling conflicts automatically
• Rich-client Web Application– Modern MVVM JavaScript UI– RESTful Ruby On Rails back-end
Project homepage at: http://saw.inf.unisi.ch open source code available on GitHub: https://github.com/ian7/saw
Software Architecture Warehouse
75
• Architecture decision management – Capture – Reuse– Analysis
• Decision process support– Assistance in deliberation – Progress monitoring
Project homepage at: http://saw.inf.unisi.ch open source code available on GitHub: https://github.com/ian7/saw
Software Architecture Warehouse
76
• High degree of liveness• Explicit but flexible decision meta-model• Design and decision space monitoring
Project homepage at: http://saw.inf.unisi.ch open source code available on GitHub: https://github.com/ian7/saw
Software Architecture Warehouse
78
a demo is worth more than million wordshttp://demo.saw.sonyx.net
79
ARCHITECTURE DECISION ARGUMENTATIONGoals Questions Metrics
Goal
80
Assess the level of consensus of a decision model involving multiple decision makers
Questions
81
1. How aligned are the decisions?2. How volatile is the consensus over the decisions?3. How democratic are the decisions?
????
Metric characteristics
82
• Name: text• Domain: {Project, Issue, Decision}• Scale: {Ratio, Ordinal}• Range: {%, N, T}
Empirical Evaluation
84
• Within the Software Architecture and Design master course at the University of Lugano
• 8 design workshop runs• Two groups of 9 students each• 90 minutes long workshops
85
Thesis
Support for the low latency, structured architecture decision argumentation
improves the quality of the decision making process.
Architecture Decision Support
86
LivenessHigh Low/None
Dec
isio
n M
eta-
Mod
el T
ype
Impl
icit
Expl
icit
EtherPad
SoftwareArchitectureWarehouse
Compendium
ADkWik
ODREA
Comparative Evaluation
87
• EtherPad – open-source, live, collaborative, rich-text editor used as a reference
• Software Architecture Warehouse – live, collaborative, structured decision support tool
Analyzer
88
• Offline analysis of design workshop dynamics• Extensible framework in terms of
– event sources– event types– metrics
Analyzer
89
HTTP Log
UI Event Log
SAW Database
EP Timeline
HTTP Log
Event Log
Project Items
Issue Items
Decision Items
Acquisition 1st stage
Raw data Item-centric event log
Decision Space Model
reconstruction
Projects matrix
Issues matrix
Decision matrix
M1
M2
M3
…
Micro-metric item model
Metric evaluation
2nd stage
Event reconstruction
debug logs
MetricMatrix.csvfiles
Metricvalue
CSV files
90
HOW ALIGNED ARE THE DECISIONS?
How aligned are the decisions?
91
• Position count (M7)
• Decision count with particular consensus state (M8)
• Consensus state of particular decision (M9)
• Choice state of particular issue (M10)
• Relative number of contributors (M3)
Positions in decision spaces
92
Metric 7: Position type count Parameter: position type, Scale: Ratio, Range: [0,N]
Positions in decision spaces
93
Metric 7: Position type count Parameter: position type, Scale: Ratio, Range: [0,N]
• Uneven amount• Mostly positive type• Divergent
Positions in decisions
96
Metric 7: Position type count Parameter: position type, Scale: Ratio, Range: [0,N]
Positions in decisions
97
Metric 7: Position type count Parameter: position type, Scale: Ratio, Range: [0,N]
• Sparse positions in over the decisions
• Long tail distribution for SAW
Consensus state vs. positions
98Metric 8: Consensus state Domain: Decision Scale: Ordinal, Range: {no positions, aligned, colliding, sealed}
Consensus state vs. positions
99Metric 8: Consensus state Domain: Decision Scale: Ordinal, Range: {no positions, aligned, colliding, sealed}
• Three parties collide• Possible herd behavior
in the long tail
104
HOW VOLATILE IS THE CONSENSUS OVER THE DECISIONS?
How volatile is the consensus over the decisions?
105
• Relative number of contributors (M3)
• Relative number of editors (M4)
• Time since last change (M5)
• Activity time (M6)
• Position count (M7)
• Time spent in particular consensus state (M11)
• Time spent in particular choice state (M12)
• Time since last position was stated (M13)
• Number of transitions of the consensus state (M14)
• Number of transitions of the choice state (M15)
Consensus state timespan
106
Metric 11: Relative consensus state timespanParameter: consensus state Domain: Decision Scale: Ratio Range: %
Consensus state timespan
107
Metric 11: Relative consensus state timespanParameter: consensus state Domain: Decision Scale: Ratio Range: %
• Two phases of decision making– Divergent – Convergent
• Little volatility after positions are set
Consensus state transitions
108
Metric 14: Consensus state transition count Domain: Decision Scale: Ratio Range: [0,N]
No positions Aligned Colliding
Consensus state transitions
109
Metric 14: Consensus state transition count Domain: Decision Scale: Ratio Range: [0,N]
• Forward architecting• Long tail distribution
in SAW
Choice state timespan
110
Metric 15: Choice state transition count Domain: Issue Scale: Ratio Range: [0,N]
Choice state timespan
111
Metric 15: Choice state transition count Domain: Issue Scale: Ratio Range: [0,N]
• Little volatility after choice is made
• SAW models are a bit more convergent
Choice state transitions
112Metric 15: Choice state transition count Domain: Issue Scale: Ratio Range: [0,N]
No Alternatives
Inconclusive Choice
Complete choice
Incomplete Choice
Conclusive Choice Warring Choice
No Positions
Choice state transitions
113Metric 15: Choice state transition count Domain: Issue Scale: Ratio Range: [0,N]
• EP models are more divergent
• Extended deliberation for SAW
Design workshop timeline
118
Design workshop timeline
119
Time since the last position
120
Metric 13: Time since last positionDomain: Issue, Decision Scale: Ratio Range: [0,T]
• Steady pace of decision making
• Slight maximum near the middle of the workshop
Design workshop timeline
121
Activity timespan
122
Metric 5: Activity timespanDomain: Project, Issue, Alternative, Position Scale: Ratio Range: [0,N]
Activity timespan
123
Metric 5: Activity timespanDomain: Project, Issue, Alternative, Position Scale: Ratio Range: [0,N]
• Time pressure• Quantity over quality
Activity timespan
126Metric 5: Activity timespanDomain: Issue Scale: Ratio Range: [0,N]
Activity timespan cut-off
127Metric 5: Activity timespanDomain: Issue, Decision Scale: Ratio Range: [0,N]
Activity timespan cut-off
128Metric 5: Activity timespanDomain: Issue, Decision, Decision Scale: Ratio Range: [0,N]
• Higher yield of significant decision items in SAW
131
HOW DEMOCRATIC ARE THE DECISIONS?
How democratic are the decisions?
132
• Number of issues (M1)
• Number of alternatives (M2)
• Activity time (M6)
• Position count (M7)
Number of contributors
133
Metric 3: Relative number of contributorsDomain: Issue, Alternative Scale: Ratio Range: %
Number of contributors
134
Metric 3: Relative number of contributorsDomain: Issue, Alternative Scale: Ratio Range: %
• Dominantly individual contributions
• Long tail distribution of contributions in SAW
Number of decision makers
137
Metric 4: Relative number of decision makersDomain: Decision Scale: Ratio Range: %
Number of decision makers
138
Metric 4: Relative number of decision makersDomain: Decision Scale: Ratio Range: %
Number of decision makers
139
Metric 4: Relative number of decision makersDomain: Decision Scale: Ratio Range: %
• Divergent decision process
• Long tail distribution for SAW
• Choice is more inclusive in SAW
Number of decision makers
140
Metric 4: Relative number of decision makersDomain: Decision Scale: Ratio Range: %
Number of decision makers
141
Metric 4: Relative number of decision makersDomain: Decision Scale: Ratio Range: %
Number of decision makers
142
Metric 4: Relative number of decision makersDomain: Decision Scale: Ratio Range: %
• Disagreement in groups of two and three
• Herd behavior
Activity timespan cut-off
143Metric 5: Activity timespanDomain: Issue, Decision Scale: Ratio Range: [0,N]
• Higher participation in significant decision items in SAW
Summary
145
Application of decision argumentation metrics to
empirical data
Comparative evaluation of the Software Architecture
Warehouse and the EtherPadSAW vs. EP
Threats to validity
146
• Internal validity– Parallel execution of the design workshops– Skillset of subjects balanced between the groups
• Construct validity– Provided by the interpretation model
• External validity– Limited to the naïve architecting mode– “Students are not different from professionals”, but
“some professionals behave VERY different from all other professionals and all students” *
* Hans van Vliet “Architecting as Decision Making” Keynote for ECSA 2014
148
IN CONCLUSION
149
Conclusion
Support for the low latency, structured architecture decision argumentation does have impact
on the quality of decision making process.
Even if our comparative evaluation in the classroom did not produce statistically significant results, it
proved the concept of decision analytics
Main Contributions
150
Design Issue
Design Alternative
Design Alternative
Design Alternative
Design Decision
Design Decision
Design Decision
Position: Positive
Position: Positive
Position: Positive
Position: Negative
Position: Negative
Architecture ArgumentationViewpoint
Decision Argumentation Metrics
Software Architecture Warehouse
Acknowledgments
151
• Evgenii Riabokon is a Masters graduate form University of Lugano. He developed Java-Script front-end module for interactive, side-by-side comparisons for project/issue/decision properties
• Masiar Babazadeh is a PhD student at University of Lugano. Contributed JavaScript and HTML5 knowledge visualization component during UROP project in the summer break 2010.
• Mark Pruneri is a Bachelor graduate from University of Lugano. Has actively developed 2nd generation prototype of SAW, contributing high quality HTML5/CSS and some Ruby On Rails coding to the Dynamic Types Management.
• Alessandro Trombini is a Bachelor graduate from University of Lugano. He developed high quality HTML + CSS user interface design for 2nd generation of SAW prototype during Software Atelier 3 course.
• Adnan Al Hariri as a Bachelor graduate from University of Lugano has developed parts of Dynamic Type Management system and foundations of collaborative content editing for 2nd generation of SAW prototype during Software Atelier 3 course.
Publication list
152
• "Team Situational Awareness and Architectural Decision Making with the Software Architecture Warehouse”, Marcin Nowak, Cesare Pautasso, Presented at: European Conference on Software Architecture (ECSA) 2013
• "The Design Space of Modern HTML5/JavaScript Web Applications", Marcin Nowak, Cesare Pautasso, Presented at: SATURN 2013
• "Software Architecture Warehouse: live and collaborative architectural decision making", Marcin Nowak, Cesare Pautasso, Presented in the tool demonstrations track of Working Conference on Software Architecture 2012
• "Reusable Decision Space for Mashup Tool Design”Saeed Aghaee, Marcin Nowak, Cesare Pautasso, Proceedings The fourth ACM SIGCHI Symposium on Engineering Interactive Computing Systems
• "Goals, Questions and Metrics for Architectural Decision Models”Marcin Nowak, Cesare Pautasso . In Proceedings of the Workshop on Sharing and Reusing Architectural Knowledge SHARK'11
• "Architectural Decision Modeling with Reuse: Challenges and Opportunities". Marcin Nowak, Cesare Pautasso, In Proceedings of the Workshop on Sharing and Reusing Architectural Knowledge SHARK'10
153
154
155
156
Spare slides
Position revoking
158
158
Team re-focusing
159
159
Team re-focusing
160
160
Constraining decision-process
161
161
Constraining decision-process
162
162
