Software Architecture Software Architecture Evaluation MethodologiesEvaluation Methodologies

Presented By:Presented By:

Anthony RegisterAnthony Register

Software architecture (SWA)Software architecture (SWA)

- - “The fundamental organization of a system embodied in “The fundamental organization of a system embodied in its components, their relationships to each other, and its components, their relationships to each other, and to the environment, and the principles guiding its to the environment, and the principles guiding its design and evolution”design and evolution” as defined by IEEE as defined by IEEE

SWA Quality determined by quality SWA Quality determined by quality attributes such as:attributes such as:

FlexibilityFlexibility

UsabilityUsability

PerformancePerformance

MaintenanceMaintenance

Evaluation methods:Evaluation methods:

SAAMSAAM : : SScenario-Based cenario-Based AArchitecture rchitecture AAnalysis nalysis MMethodethod

ALMAALMA : : AArchitecture rchitecture LLevel evel MModifiability odifiability AAnalysisnalysis

ATAMATAM : : AArchitecture rchitecture TTrade-off rade-off AAnalysis nalysis MMethodethod

PASAPASA : : PPerformance erformance AAssessment of ssessment of SSoftware oftware AArchitecturerchitecture

AssumptionAssumption

- Several of the most common SWA - Several of the most common SWA evaluation methods are similar in the evaluation methods are similar in the techniques applied, but have evolved to techniques applied, but have evolved to serve different purposes or viewpoints serve different purposes or viewpoints for evaluating the quality of a specific for evaluating the quality of a specific SWA.SWA.

Scenario BasedScenario Based Documented in OO design and user interface Documented in OO design and user interface

as a technique to solicit requirementsas a technique to solicit requirements

Used for comparing SWA optionsUsed for comparing SWA options

Simulate system events or changesSimulate system events or changes

Determine impact on SWADetermine impact on SWA

Result in brief description of a functional Result in brief description of a functional requirement that eventually may be requirement that eventually may be implemented into the system at design time or implemented into the system at design time or in the futurein the future

Scenario BasedScenario Based

Benefits:Benefits:

Break down and clearly describe the Break down and clearly describe the architectural description and analysis architectural description and analysis points points

Catalyst for generating more questionsCatalyst for generating more questions

Types of ScenariosTypes of Scenarios

DirectDirect : requires no change to system : requires no change to system architecturearchitecture

IndirectIndirect : requires change to system : requires change to system architecturearchitecture

Scenario ExampleScenario Example

Scenario-Based Architecture Analysis Scenario-Based Architecture Analysis

5 steps:5 steps:1.1. Describe the candidate architectureDescribe the candidate architecture

2.2. Develop scenariosDevelop scenarios

3.3. Perform scenario evaluationsPerform scenario evaluations

4.4. Reveal scenario interactionReveal scenario interaction

5.5. Overall SWA evaluation Overall SWA evaluation

http://www.sei.cmu.edu/pub/documents/00.reports/pdf/00tr004.pdf#search=%22ATAM%20%22

SAAM SAAM (Steps 1 & 2)(Steps 1 & 2)

Describe the candidate architectureDescribe the candidate architecture Use a common syntactic and semantic Use a common syntactic and semantic

notation that all designers and stakeholders notation that all designers and stakeholders can easily understand. can easily understand.

Develop scenariosDevelop scenarios Direct and indirectDirect and indirect

SAAM SAAM (Steps 3 & 4)(Steps 3 & 4)

Perform scenario evaluationsPerform scenario evaluations Evaluate quality attributes Evaluate quality attributes (flexibility, usability, (flexibility, usability,

performance, maintenance, etc.)performance, maintenance, etc.)

Reveal scenario interactionReveal scenario interaction Indirect scenario affects need to be at a Indirect scenario affects need to be at a

minimum or the component may be a risk to minimum or the component may be a risk to the SWAthe SWA

Scenario analysis indicates that many indirect Scenario analysis indicates that many indirect scenarios affect the same component, then scenarios affect the same component, then too much architectural level coupling may be too much architectural level coupling may be indicatedindicated

SAAM SAAM (Step 5)(Step 5)

Overall SWA evaluation Overall SWA evaluation

Weighted ranking or scoring performed for Weighted ranking or scoring performed for the overall SWA evaluation. the overall SWA evaluation.

A scoring technique using an appropriate A scoring technique using an appropriate weighting analysis can be applied to track weighting analysis can be applied to track the affects of the indirect scenarios on the the affects of the indirect scenarios on the SWA for future decision makingSWA for future decision making

SAAMSAAM

Benefits:Benefits: Disadvantages:Disadvantages:

Detecting issues earlyDetecting issues early Lacking in tool supportLacking in tool support for managing for managing large amounts of data for system and large amounts of data for system and scenario descriptions scenario descriptions

SWA documentationSWA documentation through the through the documentation of scenarios, documentation of scenarios, outcomes, and affects on the system outcomes, and affects on the system architecture architecture

Lacks architectural metricsLacks architectural metrics for for assisting designers when making assisting designers when making decisions between alternative designs decisions between alternative designs

Improved understandingImproved understanding of possible of possible SWA issuesSWA issues

SAAM StrengthsSAAM Strengths Evaluate or compare future and existing Evaluate or compare future and existing

systemssystems, such as using the low cost method , such as using the low cost method (tabular calculations and comparisons) for (tabular calculations and comparisons) for measuring change when choosing between measuring change when choosing between SWA designsSWA designs

Evaluation result enables the designers Evaluation result enables the designers and users to focusand users to focus on the details of the SWA on the details of the SWA and not be distracted from the less important and not be distracted from the less important areas areas

Provides a guided approach for evaluationProvides a guided approach for evaluation and and provides for open dialogueprovides for open dialogue between the between the stakeholders of the system under evaluationstakeholders of the system under evaluation

Architecture Level Modifiability Analysis Architecture Level Modifiability Analysis (ALMA)(ALMA)

Focuses on modifiability of the Focuses on modifiability of the SWASWA

Risk assessmentsRisk assessments Distinguishes multiple analysis Distinguishes multiple analysis

goalsgoals Explicit assumptionsExplicit assumptions Provides repeatable techniques Provides repeatable techniques

for performing the steps for performing the steps Predicting future maintenance Predicting future maintenance

costs costs Evaluating the flexibility of a Evaluating the flexibility of a

system at an architectural levelsystem at an architectural level Main quality inputs are the Main quality inputs are the

SWA specification and quality SWA specification and quality requirements requirements

http://www.sei.cmu.edu/pub/documents/00.reports/pdf/00tr004.pdf#search=%22ATAM%20%22

ALMA Process StepsALMA Process Steps

5 Steps:5 Steps:1.1. Set the goal and determine the focus of the Set the goal and determine the focus of the

analysisanalysis

2.2. Create a description of the SWACreate a description of the SWA

3.3. Create scenarios from the functional Create scenarios from the functional requirementsrequirements

4.4. Evaluate the effects of the scenariosEvaluate the effects of the scenarios

5.5. Analyze the results of the independent Analyze the results of the independent scenario evaluationsscenario evaluations

ALMA ALMA (Steps 1 & 2)(Steps 1 & 2)

Goal SettingGoal Setting Maintenance cost prediction, risk assessment, Maintenance cost prediction, risk assessment,

and SWA selectionand SWA selection

Description of SWADescription of SWA SWA information in order to derive an SWA information in order to derive an

architectural description of the systemarchitectural description of the system Activity evaluates the decomposition of the Activity evaluates the decomposition of the

system components and evaluates the system components and evaluates the relationships between the componentsrelationships between the components

ALMA ALMA (Steps 3 & 4)(Steps 3 & 4)

Create scenarios from functional requirementsCreate scenarios from functional requirements Determine the impact of change to the SWA Determine the impact of change to the SWA Providing a metric for analysis performed in the Providing a metric for analysis performed in the

next step for evaluating the different scenariosnext step for evaluating the different scenarios

Evaluate effects of scenariosEvaluate effects of scenarios Collection of information to be used in SWA impact Collection of information to be used in SWA impact

analysisanalysis Identify components affected by the change, Identify components affected by the change,

determine what the effects were, and determine the determine what the effects were, and determine the ripple effects on relational componentsripple effects on relational components

ALMA ALMA (Step 5)(Step 5)

Analyze the results of the independent Analyze the results of the independent scenario evaluationsscenario evaluations InterpretationInterpretation of the results needs to align of the results needs to align

with the goal set forth for the evaluationwith the goal set forth for the evaluation

Goals:Goals: Maintenance cost prediction, the scenarios Maintenance cost prediction, the scenarios

should cover future events of the systemshould cover future events of the system Risk assessment, the scenarios should Risk assessment, the scenarios should

provide complex change events in order to provide complex change events in order to determine and interpret the effects of the determine and interpret the effects of the changeschanges

ALMAALMA

Benefits: Benefits: Identification of SWA risksIdentification of SWA risks Measurement of the amount of effort required for Measurement of the amount of effort required for

changeschanges Deciding between available SWA optionsDeciding between available SWA options Reduction in the number of scenarios and a Reduction in the number of scenarios and a

process that provides guidance as to when to process that provides guidance as to when to stop generating scenariosstop generating scenarios All change categories explicitly consideredAll change categories explicitly considered New change scenarios do not affect the classification New change scenarios do not affect the classification

structurestructure

Architecture Trade-off Analysis Method Architecture Trade-off Analysis Method (ATAM)(ATAM)

Evaluates a SWA for quality attribute target goals, but Evaluates a SWA for quality attribute target goals, but focuses more on the trade-offs between the quality focuses more on the trade-offs between the quality attributes. attributes.

http://www.sei.cmu.edu/architecture/ata_method.html

ATAMATAMMain inputs:Main inputs:

business goalsbusiness goals software specificationssoftware specifications SWA descriptionSWA description

Main Outputs:Main Outputs: list of scenario sensitivity pointslist of scenario sensitivity points trade-off pointstrade-off points risksrisks different approaches to the SWAdifferent approaches to the SWA utility treeutility tree quality attribute questions with the responsesquality attribute questions with the responses

ATAM Process StepsATAM Process Steps

The 4 phases are:The 4 phases are: PresentationPresentation Investigation and AnalysisInvestigation and Analysis TestingTesting

ReportingReporting

ATAM ATAM (Phase 1 & 2)(Phase 1 & 2)

PresentationPresentation Activities:Activities:

Presenting the ATAMPresenting the ATAM Presenting the business requirementsPresenting the business requirements Presenting the architecturePresenting the architecture

Investigation and analysisInvestigation and analysis Activities:Activities:

Identify the architectural approaches suggested Identify the architectural approaches suggested by the architect before they are analyzedby the architect before they are analyzed

Create an attribute utility treeCreate an attribute utility tree Analyze the architectural approachesAnalyze the architectural approaches

ATAM – Utility TreeATAM – Utility Tree

An analytical method that provides a top-down An analytical method that provides a top-down approach for decomposing the quality attributes as approach for decomposing the quality attributes as designated by the ATAM goals.designated by the ATAM goals.

http://www.sei.cmu.edu/architecture/ata_method.html

ATAM ATAM (Phase 3 & 4)(Phase 3 & 4)

TestingTesting ActivitiesActivities

Brainstorming scenariosBrainstorming scenarios Analyzing the architectural approachesAnalyzing the architectural approaches Utilizes the designated high priority scenarios to be used Utilizes the designated high priority scenarios to be used

for test casesfor test cases The goal is targeted to identify any hidden architectural The goal is targeted to identify any hidden architectural

approaches, risks, sensitivity points, and tradeoff pointsapproaches, risks, sensitivity points, and tradeoff points

ReportingReporting ActivitiesActivities

Present the resultsPresent the results Presented to the stakeholders in the form of a final Presented to the stakeholders in the form of a final

analysis reportanalysis report

ATAMATAM

Benefits:Benefits: The quality attributed requirements are clarifiedThe quality attributed requirements are clarified Provides for improved SWA documentation that Provides for improved SWA documentation that

can be used in the foundation of future SWA can be used in the foundation of future SWA decisionsdecisions

Promote communication between the Promote communication between the stakeholders, such as customers, architects, and stakeholders, such as customers, architects, and testers.testers.

Identify system risks early in the solution life-Identify system risks early in the solution life-cyclecycle

Performance Assessment of Software Architecture Performance Assessment of Software Architecture (PASA)(PASA)

Evaluates performance issues for SWA systemsEvaluates performance issues for SWA systems Goal of PASA utilizes performance based scenarios Goal of PASA utilizes performance based scenarios UniqueUnique : the documentation extracted comes from the : the documentation extracted comes from the

developers and source code (since can be performed developers and source code (since can be performed during development cycle)during development cycle)

Only includes interaction with the development team Only includes interaction with the development team which is different from the other techniqueswhich is different from the other techniques

Comparison of Scenario-Based Software Architecture Evaluation Methods

PASA ProcessPASA Process

Starts by setting goalsStarts by setting goals, identifying required , identifying required information, understanding stakeholder information, understanding stakeholder expectations, and describing the method of expectations, and describing the method of approachapproach

Selects key performance scenariosSelects key performance scenarios as elicited as elicited from the developersfrom the developers

Focuses on the architectural styleFocuses on the architectural style or patterns or patterns usedused

The output of the processThe output of the process is a presentation of is a presentation of the results.the results.

Common Goals and Activities of all Methods Common Goals and Activities of all Methods

Common GoalCommon Goal : : evaluate and predict the quality attributes as evaluate and predict the quality attributes as applied to a SWA evaluation analysisapplied to a SWA evaluation analysis

Common ActivitiesCommon Activities : : EvaluatingEvaluating Planning and preparationPlanning and preparation Explanation of SWA approachesExplanation of SWA approaches Elicitation of quality sensitive scenariosElicitation of quality sensitive scenarios Analysis of SWA optionsAnalysis of SWA options Interpretation and presentation of the evaluation Interpretation and presentation of the evaluation

results for final SWA decision makingresults for final SWA decision making

Methods OverallMethods Overall

SAAMSAAM attempts to identify the potential risks to attempts to identify the potential risks to SWA and assess the modifiability. SWA and assess the modifiability.

ALMAALMA attempts to predict the modifiability attempts to predict the modifiability based on risk assessment, support costs, and based on risk assessment, support costs, and SWA comparisons.SWA comparisons.

ATAMATAM analyzes the sensitivity and trade-off analyzes the sensitivity and trade-off points to determine what may prevent realizing points to determine what may prevent realizing the best combination of quality attributes for a the best combination of quality attributes for a given SWA.given SWA.

PASAPASA evaluates the performance risks. evaluates the performance risks.

DifferencesDifferences Participants:Participants:

SAAM and ATAM involve the architects, designers, and the SAAM and ATAM involve the architects, designers, and the end usersend users

ALMA only mostly includes the architect designerALMA only mostly includes the architect designer PASA only includes the developersPASA only includes the developers

SAAM and ATAM are the only twoSAAM and ATAM are the only two methodologies that are methodologies that are close to success when providing details as to the costs associated close to success when providing details as to the costs associated with a SWA evaluation or resource requirementswith a SWA evaluation or resource requirements

ATAM is one of the fewATAM is one of the few if not the only evaluation method that if not the only evaluation method that provides sufficient process steps. The other evaluation methods provides sufficient process steps. The other evaluation methods provide descriptions of the required activities, but do not provide provide descriptions of the required activities, but do not provide enough granular detail.enough granular detail.

All of the methods are influenced by non-technical All of the methods are influenced by non-technical issuesissues, such as stakeholder interests and political factors, but the , such as stakeholder interests and political factors, but the ATAM process is the only method that provides instructions for ATAM process is the only method that provides instructions for detailed guidelines and techniques to manage the social issuesdetailed guidelines and techniques to manage the social issues

Original AssumptionOriginal Assumption

The most common SWA The most common SWA evaluation techniques are similar in the evaluation techniques are similar in the methodology applied, but have evolved to methodology applied, but have evolved to serve different purposes or perspectives serve different purposes or perspectives for evaluating the quality of a specific for evaluating the quality of a specific SWA.SWA.

Final AssumptionFinal Assumption

Slightly reversedSlightly reversed in truth as it has been in truth as it has been determined that the methodologies applied are determined that the methodologies applied are different and unique in their approaches, but different and unique in their approaches, but they all serve the common goal for they all serve the common goal for determining the best SWA option for a given determining the best SWA option for a given set of business scenariosset of business scenarios

No single evaluation techniqueNo single evaluation technique provides a provides a clearly decisive result by specifying a single clearly decisive result by specifying a single SWA selection out of multiple available SWA SWA selection out of multiple available SWA optionsoptions

ConclusionConclusion

The various techniques of SAAM, ALMA, The various techniques of SAAM, ALMA, ATAM, and PASA use different approaches and steps ATAM, and PASA use different approaches and steps in the quest for selecting the best SWA solution. in the quest for selecting the best SWA solution. Ultimately the common goal or purpose of all of the Ultimately the common goal or purpose of all of the techniques is the selection of the SWA that provides techniques is the selection of the SWA that provides maximum usability and satisfies all of the quality maximum usability and satisfies all of the quality attributes as defined by the business goals.attributes as defined by the business goals.

Questions and DiscussionQuestions and Discussion