1
IV&V Facility
FY 2002 Initiative IV&V of UML
Hany Ammar, Katerina Goseva-Popstojanova,V. Cortelessa, Ajith Guedem, Diaa Eldin Nassar, Walid AbdelMoez, Ahmad
Hassan, and Rania Elnaggar
LANE Department of Computer Science and Electrical EngineeringWest Virginia University
Ali Mili, Bo Yu
College of Computing ScienceNew Jersey Institute of Technology
Less risk, sooner-
A Catch Phrase by Coach Menzies
WVU UI: Architectural-level Risk Assessment
2
IV&V Facility
Outline
• Objectives
• What we can do
• Why UML
• UML & NASA
• Project Overview
• Architecture-Based Risk Analysis
• The Risk Assessment Methodology
• Performance – based risk
• Accomplishments
• Future Work • Publications
3
IV&V Facility
Objectives
• Automated techniques V&V of dynamic specifications– Performance and timing
analysis – Fault-injection based
analysis,• Less risk, sooner
– Risk assessment• Technologies:
– UML– Architectures– Risk assessment
methodology• Benefits:
– Find & rank critical • use cases, scenarios, • components, connectors
Before bad software
After bad software
The ARIANE 5explosion
4
IV&V Facility
What We Can Do
• Identify and rank critical components based on risk factors and severity classes• How?- details follow
Scitcs
Fritcs
Pfmc_
LT
Pfmc_
MT
Sch
State
_Man
Op_C_Q
App_C
_QN3_
1
N3_2
Rpcm
_LT
Rpcm
_MT
Single_LT
Single_MT
Dual_MT_Failed
Dual_LT_Failed
Dual
Retry_MT_Pump
Retry_LT_Pump
Retrt_Both_Pumps
Monitoring
0
0.1
0.2
0.3
0.4
Risk Factor
Components
Scenarios
Minor
Major
Critical
Catastrophic
Not contributing
5
IV&V Facility
Why UML
• Unified modeling language– Rational software
– The three amigos: Booch Rumbaugh, Jacobson.
• International standard in system specification
An international standardIn system specification
6
IV&V Facility
UML & NASA
• Increasing use at NASA• Informal (very) survey
– Google search:
– “rational rose nasa”
– 10,000 hits
– 3 definite projects, just in first ten
• We use a case study
based on the UML specs of a component of the
International Space Station
7
IV&V Facility
Project Overview
FY01• Developed of an automated simulation environment for UML dynamic specification,
suggested an observer component to detect errors • Conducted performance and timing analysis of the NASA case study
FY02• Develop a fault injection methodology
Define a fault-model for components at the specification level • Develop a methodology for architecture-based risk analysis
Determine critical use case ListDetermine critical component/connector list
(based on recent paper by Yacoub & Ammar on IEEE Trans. on Software Engineering, June 02)
FY03• Develop a methodology for performance-based/reliability-based risk assessment• Validation of the risk analysis methodology on several NASA projects
8
IV&V Facility
Architecture-Based Risk Analysis
• Develop architecture-based approach for risk assessment– Overall system/subsystem– Different use cases– Key scenarios associated with use cases
• Heavily used scenarios• Scenarios that are used infrequently but perform critical functions
• Develop components and connectors risk factors– Define components risk factors as
Normalized dynamic complexity * Severity – Estimate dynamic complexity measure based UML sequence diagrams and state charts– Estimate severity measure based FEMA and hazard analysis– Consistent with the NASA definition of risk
Probability of an undesired event * Consequences if that event should occur
– Define connectors risk factor as Normalized dynamic coupling * Severity
9
IV&V Facility
Risk Assessment Methodology
• For each use case– For each scenario
• For each component– Measure dynamic complexity– Assign severity based on FMEA and hazard analysis– Calculate risk factor
• For each connector – Measure dynamic coupling– Assign severity based on FEMA and hazard analysis– Calculate risk factor
• Construct Markov model• Calculate scenario level risk factor• Determine critical component/connector list
– Calculate use case level risk factors– Rank the scenarios based on risk factors, Determine critical scenarios list
• Calculate system level risk • Rank the use cases based on risk factors, Determine critical use case list • Determine critical component / connector list in the system scope
10
IV&V Facility
Risk Assessment Methodology
• For each use case– For each scenario
• For each component– Measure dynamic complexity– Assign severity based on FMEA and hazard analysis– Calculate risk factor
• For each connector – Measure dynamic coupling– Assign severity based on FEMA and hazard analysis– Calculate risk factor
• Construct Markov model• Calculate scenario level risk factor• Determine critical component/connector list
– Calculate use case level risk factors– Rank the scenarios based on risk factors, Determine critical scenarios
list • Calculate system level risk • Rank the use cases based on risk factors, Determine critical use case
list • Determine critical component / connector list in the system scope
11
IV&V Facility
NASA CASE STUDY Use Case Diagram
Mode_setting
Single_MT
<<uses>><<uses>>
Single_LT
<<uses>><<uses>>
Dual_LT_Fai led
<<uses>><<uses>>
Dual
<<uses>><<uses>>
Dual_MT_Failed
<<uses>><<uses>>
MT_Pump_Retry
LT_Pump_Retry
Retry_Both_Pumps
SFCA_MT
SFCA_LTPPA_MTPPA_LT
Warnig_for_Total_fai lure
Failure_Recovery
<<uses>><<uses>>
<<uses>><<uses>>
<<uses>><<uses>>
<<uses>><<uses>>
<<uses>><<uses>> <<uses>><<uses>>
<<uses>><<uses>>
<<uses>><<uses>>
Operator
Monitoring
12
IV&V Facility
Risk Assessment Methodology
• For each use case
– For each scenario• For each component
– Measure dynamic complexity– Assign severity based on FMEA and hazard analysis– Calculate risk factor
• For each connector – Measure dynamic coupling– Assign severity based on FEMA and hazard analysis– Calculate risk factor
• Construct Markov model• Calculate scenario level risk factor• Determine critical component/connector list
– Calculate use case level risk factors– Rank the scenarios based on risk factors, Determine critical scenarios
list • Calculate system level risk • Rank the use cases based on risk factors, Determine critical use case
list • Determine critical component / connector list in the system scope
13
IV&V Facility
Both Pumps Retry- scenario
RPCM_LT / rPCMR2 : RPCM
RPCM_MT / rPCMR1 : RPCM
/ pFMC_MTR1 : PFMC_MT
/ pFMC_LTR1 : PFMC_LT
/ fRITCSR1 : FRITCS
/ sCITCSR1 : SCITCS
Switch_CloseFailed Switch_CloseFailedDual_Mode_OOOODual
1: LT_Fai led1: LT_Fai led
1: MT_Failed1: MT_Failed
2: Pump_Retry(Retry)2: Pump_Retry(Retry)
PPAMT_F_PPALT_F_SFCAMT_O_SFCALT_O
3: Open_Switch (void)3: Open_Switch (void)
4: Open_Switch (void)4: Open_Switch (void)
DualSwitch_Open
PPAMT_F_PPALT_F_SFCAMT_O_SFCALT_O Switch_Open
5: Close_Switch (void)5: Close_Switch (void)
Switch_ClosePPAMT_F_PPALT_F_SFCAMT_O_SFCALT_O
6: Pump_Retry (void)6: Pump_Retry (void)
PPAMT_F_PPALT_F_SFCAMT_O_SFCALT_O Pump_Retry
7: Retry_Success (void)7: Retry_Success (void)
8: Pump_Retry_Success (void)8: Pump_Retry_Success (void)
Dual_Mode_OFOO
8: MT_Operating (void)8: MT_Operating (void)
Dual
Dual_Mode_OFOO Operating
9: Pump_Retry (Pump_Retry_Data{Retry_Type 1,Failure_Type 6})9: Pump_Retry (Pump_Retry_Data{Retry_Type 1,Failure_Type 6})
10: Pump_Retry (void)10: Pump_Retry (void)
Dual_Mode_OFOO Pump_Retry
11: Retry_Success (void)11: Retry_Success (void)
12: Pump_Retry_Success (void)12: Pump_Retry_Success (void)
Dual Dual_Mode_OOOO
12: LT_Operating (void)12: LT_Operating (void)
OperatingDual_Mode_OOOO
14
IV&V Facility
Risk Assessment Methodology
• For each use case– For each scenario
• For each component– Measure dynamic complexity– Assign severity based on FMEA and hazard analysis– Calculate risk factor
• For each connector – Measure dynamic coupling– Assign severity based on FEMA and hazard analysis– Calculate risk factor
• Construct Markov model• Calculate scenario level risk factor• Determine critical component/connector list
– Calculate use case level risk factors– Rank the scenarios based on risk factors, Determine critical scenarios list
• Calculate system level risk
• Rank the use cases based on risk factors, Determine critical use case list • Determine critical component / connector list in the system scope
15
IV&V Facility
Component Dynamic Complexity
Dynamic Complexity of a component Oi in scenario x is defined as :
Where
CCx(oi ) =ti – ci + 2
Is the cyclomatic complexity of a component Oi in scenario x
Cx(oi ) : A finite set of states for a component Oi
for a scenario x , ci is the cardinality of this set.
Tx(oi ) : A finite set of transitions from one state to another for Oi component in a scenario x,
ti is the cardinality of this set.
Ox : is the set of components collaborating during the execution of a scenario
( ( ))iDOC o
1
( )( )
( )x
x ix i O
x kk
CC oDOC o
CC o
16
IV&V Facility
Component Severity (FEMA)
Component Name Failure Mode Cause of Failure Effect of Failure Criticality of effects
SCITCS Failed to Synchronize with the FRITCS
Error in interpreting FRITCS message
Unable to follow up with the FRITCS recovery action
Major
FRITCS Failed to react to the failure of both pumps
Error in interpreting the message of the pumps or it is in wrong state
Unable to take the required failure recovery procedure of both pumps fail.
Catastrophic
PFMC_LT Failed to report the right status of the LT PUMP
LT PUMP sensor is malfunctioning
Unable to monitor or set the pump correctly
Critical
PFMC_MT Failed to report the right status of the MT PUMP
MT PUMP sensor is malfunctioning
Unable to monitor or set the pump correctly
Critical
RPCM_LT Failed to respond to the FRITCS commands
LT switch controller is malfunctioning
Unable to set the LT switch in the appropriate position
Major
RPCM_MT Failed to respond to the FRITCS commands
MT switch controller is malfunctioning
Unable to set the MT switch in the appropriate position
Major
17
IV&V Facility
Risk Assessment Methodology
• For each use case– For each scenario
• For each component– Measure dynamic complexity– Assign severity based on FMEA and hazard analysis– Calculate risk factor
• For each connector – Measure dynamic coupling– Assign severity based on FEMA and hazard analysis– Calculate risk factor
• Construct Markov model• Calculate scenario level risk factor• Determine critical component/connector list
– Calculate use case level risk factors– Rank the scenarios based on risk factors, Determine critical scenarios list
• Calculate system level risk
• Rank the use cases based on risk factors, Determine critical use case list • Determine critical component / connector list in the system scope
18
IV&V Facility
Dynamic coupling for connector between component Oi and component Oj
{ ( , )| , }( , ) x i j i j x i j
x
MT o o o o O o o
x i j MTEOC o o
Connector Dynamic Coupling
MTx(Oi ,Oj) : is the set of messages sent from component Oi to component Oj during the execution of scenario x
MTx : is the set of total messages exchanged between all components during the execution of scenario x .
is the percentage of the number of messages sent from Oi to OJ with respect to the total number of messages exchanged during the scenario x
19
IV&V Facility
Connector Dynamic Coupling
• Dynamic Coupling for Connector between component C1 and component C2 is
EOC(C1,C2) =2/5=0.4
M1
M2
M3
M4
M5
C1C2 C3
20
IV&V Facility
Connector Severity (FEMA)
Connector Name Failure Mode Cause of Failure Effect of Failure Criticality of effects
SCITCS-FRITCS Failure to keep the SCITCS and FRITCS synchronized
Wrong message or message has been lost
The FRITCS won’t be able to take appropriate action in case of a pump or valve failure
Minor
FRITCS-SCITCS Failure to keep the SCITCS and FRITCS synchronized
Wrong message or message has been lost
The SCITCS won’t be able to take appropriate action in a mode setting operation
Major
FRITCS-PFMC Unable to retry the pump
Incorrect interpretation of the sent message to the pump
The failure recovery may fail though this pump has not been retried
Critical
FRITCS-RPCM Failed to set the switch in the required setting
Incorrect interpretation of the sent message to the pump
The switch will not be in the position required to make the recovery procedure
Major
PFMC-FRITCS Failed to deliver the right pump state to FRITCS
Wrong message or message has been lost
The whole failure recovery scenario will not be initiated at all as there the case of both pumps fail won’t be detected
Catastrophic
RPCM-FRITCS Failed to report the current status of the switch to FRITCS
Wrong message or message has been lost
The Failure recovery is not responsible for reacting to different switch positions so it won’t be affected much with it
Minor
21
IV&V Facility
Risk Assessment Methodology
• For each use case– For each scenario
• For each component– Measure dynamic complexity– Assign severity based on FMEA and hazard analysis– Calculate risk factor
• For each connector – Measure dynamic coupling– Assign severity based on FEMA and hazard analysis– Calculate risk factor
• Construct Markov model• Calculate scenario level risk factor• Determine critical component/connector list
– Calculate use case level risk factors– Rank the scenarios based on risk factors, Determine critical scenarios list
• Calculate system level risk
• Rank the use cases based on risk factors, Determine critical use case list • Determine critical component / connector list in the system scope
22
IV&V Facility
The Markov Chain Model for Both Pumps Retry- scenario
FAILURE STATES OF VARIOUS SEVERITIES
PFMC_LT
Minor
Major
Critical
Catastrophic
FRITCS
SCITCS
RPCM_MT
RPCM_LT
PFMC_MT
S
T
23
IV&V Facility
Risk Assessment Methodology
• For each use case– For each scenario
• For each component– Measure dynamic complexity– Assign severity based on FMEA and hazard analysis– Calculate risk factor
• For each connector – Measure dynamic coupling– Assign severity based on FEMA and hazard analysis– Calculate risk factor
• Construct Markov model• Calculate scenario level risk factor• Determine critical component/connector list
– Calculate use case level risk factors
– Rank the scenarios based on risk factors, Determine critical scenarios list
• Calculate system level risk • Rank the use cases based on risk factors, Determine critical use case
list • Determine critical component / connector list in the system scope
24
IV&V Facility
Distribution of risk factors of each scenario over the severity classes
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Minor
Marginal
Crtical
Catastrophic
25
IV&V Facility
Risk Assessment Methodology
• For each use case– For each scenario
• For each component– Measure dynamic complexity– Assign severity based on FMEA and hazard analysis– Calculate risk factor
• For each connector – Measure dynamic coupling– Assign severity based on FEMA and hazard analysis– Calculate risk factor
• Construct Markov model• Calculate scenario level risk factor• Determine critical component/connector list
– Calculate use case level risk factors– Rank the scenarios based on risk factors, Determine critical scenarios
list
• Calculate system level risk
• Rank the use cases based on risk factors, Determine critical use case list
• Determine critical component / connector list in the system scope
26
IV&V Facility
Overall System risk distribution over the severity classes
MINORMARGINALCRITICALCATASTROPHIC
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
1
MINOR
MARGINAL
CRITICAL
CATASTROPHIC
MINOR MARGINAL CRTICAL CATASTROPHIC
0.3014 0.0103 0.2192 0.2879
The overall system risk factor is: 0.8189
27
IV&V Facility
Sensitivity analysis of components
00.10.20.30.40.50.60.70.80.910.65
0.7
0.75
0.8
0.85
0.9
0.95
1
Component risk factor
Sc
en
ari
o
ris
k
fa
cto
r
LTMTFRITCSRPLTRPMTSCITCS
28
IV&V Facility
Risk Assessment Methodology
• For each use case– For each scenario
• For each component– Measure dynamic complexity– Assign severity based on FMEA and hazard analysis– Calculate risk factor
• For each connector – Measure dynamic coupling– Assign severity based on FEMA and hazard analysis– Calculate risk factor
• Construct Markov model• Calculate scenario level risk factor• Determine critical component/connector list
– Calculate use case level risk factors– Rank the scenarios based on risk factors, Determine critical scenarios list
• Calculate system level risk
• Rank the use cases based on risk factors, Determine critical use case list • Determine critical component / connector list in the system
scope
29
IV&V Facility
Determine Critical Component/Connector List
Scitcs
Fritcs
Pfmc_
LT
Pfmc_
MT
Sch
State
_Man
Op_C_Q
App_C
_QN3_
1
N3_2
Rpcm
_LT
Rpcm
_MT
Single_LT
Single_MT
Dual_MT_Failed
Dual_LT_Failed
Dual
Retry_MT_Pump
Retry_LT_Pump
Retrt_Both_Pumps
Monitoring
0
0.1
0.2
0.3
0.4
Risk Factor
Components
Scenarios
Minor Major Critical CatastrophicNot contributing
30
IV&V Facility
• Performance failure is the inability of the system to meet its performance objective(s)
• Define components performance-based risk as Normalized component demand factor * Severity
Performance – based risk
X1 X2 X3
T11 T21
T22 T31
T23
T12
T32
D12
D21
D22
D23
D31
D32
D11 = 11 1
11 2
11 3
T r
T r
T r
d
d
d
kijrTd is demand for resource kr
(e.g, CPU, disk, etc.) in state Tij
(state j of component i )
Scaling vector ][ keSC the resource demands accordingly to the corresponding service times of the resources
scales
31
IV&V Facility
•Total demand of component xi in a scenario Sk is1
i k
l
X S ijj
D D
•Normalized demand factor of component xi in Scenario Sk
DFi = ( . SCT) / ( . SCT)i kX SD
kSD
where m is total number of components and l total number of states for a given component in a given scenario
• Overall demand of a scenario Sk is1 1
k
m l
S iji j
D D
Performance – based risk
32
IV&V Facility
Accomplishments
• Developed analytical techniques and a methodology for Architecture-Based Risk Analysis
• A lightweight approach based on static analysis of dynamic specifications is developed and automated
• A tool will be presented in the Tools session
• Applied the methodology and tool to the NASA case study
33
IV&V Facility
Future Work
• The main thrust of our future work will be in the development of a cohesive methodology for performance- based /reliability- based risk assessment
• Compare risk factors based on other Complexity and coupling metrics obtained from static analysis of UML dynamic specs.– COSMIC-Full Function Point measurement maybe a good complexity
predictor.
– COCOMO II’s effort prediction may be another good complexity predictor
• Validation of methodology using several NASA case studies
34
IV&V Facility
Publications
1. Sherif M. Yacoub, Hany H. Ammar , “A Methodology for Architecture-Level Reliability Risk Analysis,” IEEE Transactions on Software Engineering, June 2002, pp. 529-547
2. H. H. Ammar, T. Nikzadeh, and J. B. Dugan "Risk Assessment of Software Systems Specifications," IEEE Transactions on Reliability, September 2001
3. Hany H. Ammar, Sherif M. Yacoub, Alaa Ibrahim, “A Fault Model for Fault Injection Analysis of Dynamic UML Specifications,” International Symposium on software Reliability Engineering, IEEE Computer Society, November 2001
4. Rania M. Elnaggar, Vittorio Cortellessa, Hany Ammar, “A UML-based Architectural Model for Timing and Performance Analyses of GSM Radio Subsystem” , 5th World Multi-Conference on Systems, Cybernetics and Informatics, July. 2001, Received Best Paper Award
5. Ahmed Hassan, Walid M. Abdelmoez, Rania M. Elnaggar, Hany H. Ammar, “An Approach to Measure the Quality of Software Designs from UML Specifications,” 5th World Multi-Conference on Systems, Cybernetics and Informatics and the 7th international conference on information systems, analysis and synthesis ISAS July. 2001.
6. Hany H. Ammar, Vittorio Cortellessa, Alaa Ibrahim “Modeling Resources in a UML-based Simulative Environment”, ACS/IEEE International Conference on Computer Systems and Applications (AICCSA'2001), Beirut, Lebanon, 26-29 June 2001
7. A. Ibrahim, Sherif M. Yacoub, Hany H. Ammar, “Architectural-Level Risk Analysis for UML Dynamic Specifications,” Proceedings of the 9th International Conference on Software Quality Management (SQM2001), Loughborough University, England, April 18-20, 2001, pp. 179-190