CmpE 550 – Advanced Software Engineering
Selim Özyılmaz & Elif SürerSelim Özyılmaz & Elif Sürer
““Scenario-Based Assessment of Scenario-Based Assessment of Nonfunctional Requirements”Nonfunctional Requirements”
INTRODUCTION
Purpose: Validation of Requirements Specification using Scenarios
“Scenarios are applied to the analysis of non-functional requirements using dependency tables to assess the relationship between goals (functional and non-functional requirements) and the agents and tasks that achieve them in the i* language”
Introduction
Many NFR’s are influenced by human properties, they inherit the diverse nature of human characteristics (System Reliability is influenced by human characteristics such as ability)
This work is a revised version of a previous one that prompted designers with questions about potential problems in a scenario event sequence. (Psychology-based taxonomy of failure)
Intro(Cont’d)
The problem is that too many scenario variations were generated. To prevent, transform the taxonomy of human and system failures into a model to predict the errors in the system design.
Bayesian Belief Nets (BNs) are used to predict the reliability.
Intro(Cont’d)
Model-checking techniques have been used extensively to verify and validate requirements.
Communication problem occurs between user-stakeholders and the model developers.
Software Cost Reduction (SCR)
[ tabular representation ]
Related Work
Related Work
A combination of visualizations, examples, and simulations is necessary to explain complex requirements to end users.
Scenario-based representations and animated simulations help users see the implications of system behavior and, thereby, improve requirements validation.
Related Work
Related Work
Animation simulation tools -> by Dubois et al. in the ALBERT II language
KAOS language and supporting GRAIL tool which enable formal reasoning about dependencies between goal model, system behavior and constraints.
Animator-validator tool, TROLL, uses a formal object-oriented language for modeling information systems
Related Work
Related Work
What is a sufficient set of scenarios to enable validation to be completed with confidence?
While we believe there is no quick answer to this vexing problem, one approach is to automate the process as far as possible so more scenarios can be tested.
Related Work
Related Work
Intent specifications provide a hierarchical model to facilitate reasoning about system goals and requirements in safety critical systems.
Goals are decomposed in a means-ends hierarchy, widely practiced in requirements engineering.
Automated support for reasoning about conflicting system states and behavior is provided by the SpecTRM-RL tool.
Related Work
Related Work
Assessment of nonfunctional system requirements, such as system reliability, has to use probabilistic reasoning since the range of potential system behaviors is either unknown, in the early requirements phase, or too large to specify.
Bayesian Nets (BNs) have been developed to assess software quality from properties of the code and software engineering process.
Related Work
Related Work
BNs have been widely applied as a probabilistic reasoning technique in software engineering and other domains; however, previous work used single nets to evaluate a set of discrete states pertaining to a software product or development process.
More automated tools for scenario analysis of NFR conformance for requirements specifications with multiple BN tests have been developed.
Related Work
Modeling Uncertainity
There are four different techniques to model: Bayesian Probability Dempster-Shafer Fuzzy Sets Possibility Theory
Bayesian Probability offers an easier combination of multiple influences on probability than Dempster-Shafer and a sounder reasoning than Fuzzy sets.
Bayesian Probability provides a decision theory of how to act on the world in an optimal fashion under circumstances of uncertainity.
Modeling Uncertainty
Bayesian Belief Nets
Directed acyclic graphs of causal influences, where the nodes represent the variables and the arcs represent the relationships between variables.
Variables can have any number of states in a BN, so the choice of measurement is left to the analyst.
Network Probability Table (NPT)
BN (Cont’d)
Input evidence are propagated through the network updating the values of other nodes (Computationally complex but effective algorithms exist)
BN (Cont’d)
Three possible ways to compute the probability of each node in the network Input a posterior probability into a BN as a prior
observation each run should be assumed to be independent
Combine output probabilities from a sequential run using a summarizer probabilities of particular run has to be set initially
Output probability of each event is compared with a threshold value, if surpassed success else failure
Pinpoint some steps in the scenario which are weak in reliability
Sensitiviy analysis can be carried out with multiple BN runs by varying environmental variables
BN Model of System Reliability
Influencing factors are divided into two: Slips: Attention based lapses and omissions in
skilled behavior Mistakes: failures in plans and knowledge of
processing. System environmental variables have an indirect
effect on an individuals ability whereas organizational factors (management culture) have a direct effect.
High Cognitive Complexity more prone to mistakes High Physical Complexity more prone to slip errors
System Reliability
Remarks– Input variables are all discrete states– The BN is a run with a range of scenarios
that stress-test system design against operational variables.
– Scenarios can be either taken from domain-specific operational procedures or by interviewing with users
– Two different outputs: slips and mistakes
Operational Performance Time
Same variables with reliability case, the difference is that except two output nodes only one output node
The output is used to increase the best case task completion time to reflect the less than ideal properties of human and machine agents.
ET: estimated time BT: best task completion time To reflect the case of reverting to manual when an
automated technology fails, highly automated tasks worst completion time is set greater than the those of manual tasks
SRA System Architecture
Analysis starts with the selection of the i* model to be evaluated and creating the test scenarios.
Scenarios are narratives taken from real life experience describing operation of similar systems from which event sequences are extracted.
SRA Tool’s Components
The Session Controller implements the user command interface for selecting designs and scenarios and executes the algorithm that assesses a set of scenarios with the BNs. It calls the system reliability or operational performance BN assessors to execute the BN runs with all possible environmental combinations.
SRA Tool’s Components
The i* model editor allows interactive construction of i* models with typical CASE tool-type functions.
The Interactive Scenario Constructor produces test scenarios from the system model based on user directions. Scenarios are stored in a database in an array of tuples.
SRA Tool’s Components
The Model Controller controls the BN models. It selects the appropriate BN model for each task step, then populates the input nodes, runs the model, and receives the belief distributions of the output nodes.
The Model Controller also manages the back propagation of the BN model to identify required technology and agent characteristics.
SRA Tool’s Components
SRA Tool’s Components
The BN assessor modules run the net by calling the HUGIN algorithm for each task step and for each set of environmental variable combinations.
The output from each run is compared with the desired NFR threshold and the survivor runs are passed to the results visualizer.
SRA Tool’s Components
SRA Tool’s Components
The Visualizer provides a visual summary of all qualified BN runs for a set of scenarios for one or more system designs.
This enables different designs to be compared and problem areas in the requirements to be identified, i.e., task/technical component combinations which show low potential NFR assessments. The Visualizer displays results at three levels: System, Scenario, and Phase views.
SRA Tool’s Components
NFR Analysis Method
Remarks Scenarios are composed of a number of
phases and each phase is composed of a number of task-steps each modeled as <Agent,Task,Technology>
Phases are used to structure task sequences that fulfill a higher order goal.
The best design is generally the one who has more surviving BN runs.
The best design also needs to be resilient to environmental conditions.
NFR Analysis Method
NFR Analysis Method
Impact of Environmental Variables (1) Survivor runs with ER x set to best case (2) Total survivor runs for all settings
If an overall design or a particular task fails to meet the threshold back propagation alaysis is used to discover the necessary settingto achieve the NFR value. All input nodes are unconstrained
(calculation for all) One/few inputs are unconstrained
(calculation for unconstrained)
NFR Analysis Method
Case Study
The application of the SRA tool in validating the operational performance and system reliability of a complex socio-technical system.
The requirements question is to assess the impact of new automated technology on the task of loading weapons on to aircraft in an aircraft carrier.
Case Study
Tasks in Design 1 are manual or semiautomated, while, in Design 2, they are semi or fully automated.
The second design saves manpower since it can be operated by one WA and is potentially more rapid to operate, but it is more expensive.
Case Study
When the operational performance times are compared, Design 2 is quicker for nearly all tasks, which is not surprising since it has more automated tasks.
Case Study
Case Study
The critical environmental variables for both designs, shows that incentives, motivation, duty time concurrency, and time constraints were all marked as vulnerable for Design 1.
Design 2 in contrast, fares better with only motivation, concurrency, and maintenance marked as vulnerable.
Case Study
Case Study
After identifying the most appropriate design, the problematic tasks, and the critical environmental variables, the analyst investigates the improvements required for the Autoload palette component, which was the weakest link in Design 2.
Case Study
Validating the BN Models
Data mining techniques are used to test the assumptions in BN models.
All possible permutations are simulated and created a database of reliability and performance time predictions
Data mining Techniques:Relevance Analysis: ranks input parameters of
the model based on their relevance with output
Association Rules: describe how often two or more facts cooccur in a dataset and were employed to check the causal associations in our model.
Classification: partitions large quantities of data into sets with common characteristics and properties
Validating BN Models (Cont’d)
Results– Sea state had only a minor influence on
system error relevance analysis– IF (Duty-time=high) survived=fail– IF (Workload=high) survived=fail
(Association)– These rules indicate that causal
influences of these variables are higher than assumed, alter NPT settings to overcome.
– Crew motivation and agent ability problems classification
Validating the BN Models
Discussion & Conclusions
Automated testing of requirements specifications and designs for conformance to nonfunctional requirements using a set of scenarios and variations in the system environment have been developed.
Discussion & Conclusions
The SRA could be applied to any class of component-based problems where the selection of components needs to be optimized for nonfunctional requirement types of criteria.
The SRA tool was a development from previous BN requirements analyzer and has partially addressed the difficult problem of scenario-based testing
Discussion & Conclusions
Discussion & Conclusions
The SRA tool is aimed at requirements investigation in complex socio-technical systems and, hence, it complements model-checking tools which are more appropriate to later stages in development when specifications of agent behavior are available.
Discussion & Conclusions