Development of Security Metrics for a DistributedMessaging System

Reijo M. SavolaVTT Technical Research Centre of Finland

Oulu, FinlandReijo.Savola@vtt.fi

Habtamu AbieNorwegian Computing Center

Oslo, NorwayHabtamu.Abie@nr.no

Abstract—Carefully designed security metrics of practicalrelevance can be used to provide evidence of the securitybehavior of the system under development or operation. Thisstudy investigates a practical development of security metrics fora distributed messaging system based on threat and vulnerabilityanalysis and security requirements. Our approach is thusrequirement-centric. The high-level security requirements areexpressed in terms of lower-level measurable componentsapplying a decomposition approach. Both non-attack strategyoriented and attacker behaviour oriented metrics areinvestigated. The available on-line evidence information of thesecurity performance of the system is integrated with off-linemetrics to enable holistic decision-making for securitymanagement of the system.

I. INTRODUCTION

Evidence of the security performance of systems, servicesand products is needed for proactive security management ofthem. The current practice of security is a highly diversediscipline; widely accepted approaches for securitymeasurement are still missing. Clearly, the key problem is notonly the measurement but also the identification of exactlywhat to measure and how to prioritize issues to be focused on.A vast collection of security metrics without practicalimportance is useless.

In our earlier work [9], we have introduced a high-levelprocess for security metrics development, based on threat andvulnerability analyses and decomposition of securityrequirements. In this study, we enhance this process based onour practical experience from an example system – GEMOMmiddleware (Genetic Message Oriented Secure Middleware)[1]. In addition, a contribution of this study is to investigatedifferent sources of basic measurable components, and tointroduce the security monitoring mechanism developed forGEMOM. GEMOM is being developed in the GEMOM EUFP7 ICT project that focuses on end-to-end intelligence,adaptive security and the resilience of complex, distributedinformation systems.

Our hypothesis is that practical security metricsdevelopment can be based on the identification of basicmeasurable components in the system and taking into accountof the constraints due to the information flow architecture ofthe system under investigation. The methodology used in thisresearch is partly analytical and partly experimental. We apply

a theoretical security metrics development approach andidentify practical considerations on the development of securitymetrics. In this study, we do not present detailed securitymetrics. Rather, we investigate the sources and relationshipsbetween different basic measurable components. Detailedmetrics can be developed based on these components often in astraightforward way.

The rest of this paper is organized into the followingsections: Section II, which briefly introduces the system underinvestigation, GEMOM, Section III, which discusses oursecurity metrics development approach, Section IV, whichidentifies the origins of Basic Measurable Components (BMCs)obtained from using this approach and originating fromelsewhere, and discusses the use of them in GEMOM, SectionV, which presents an enhanced security metrics developmentprocess based on the observations in this study, Section VI,which presents related work, and finally, Section VII, whichsummarizes the study.

II. SYSTEM UNDER INVESTIGATION

MOM (Message Oriented Middleware) is used to enableapplications to exchange messages with other applications,without having to know details of the other applicationplatform and networking. GEMOM is a MOM based on thepublish/subscribe asynchronous messaging paradigm, which isthe most efficient way to integrate medium to high complexitydistributed systems.

A. Characteristics of the GEMOM SystemThe GEMOM [1] system is a resilient and scalable MOM

that supports adaptive security management by a monitoringfunctionality based on security and Quality of Service (QoS)metrics. Adaptive security in GEMOM refers to a securitysolution that learns and adapts to the changing threatenvironment without sacrificing too much of the efficiency,flexibility, reliability and security of the system. This involvesgathering contextual information both from within the systemand from the environment, analyzing the collected informationand responding to changes by adjusting security functions suchas encryption schemes, security protocols, security algorithms,and different authentication and authorization mechanisms.Information gathering is carried out by security and QoSmonitoring services and related administration. The GEMOMEU FP7 project investigates different use cases for the

978-1-4244-4740-4/09/$25.00 ©2009 IEEE

GEMOM system: a collaborative business portal, financialmarket data delivery, a road management system and moneytransfer banking.

Figure 1. Visualization of information flows to and from the GEMOMMonitoring Tool containing security metrics.

B. GEMOM System ArchitectureThe template is used to format your paper and style the text.

All margins, column widths, line spaces, and text fonts areprescribed; please do not alter them. You may notepeculiarities. For example, the head margin in this templatemeasures proportionately more than is customary. Thismeasurement and others are deliberate, using specifications thatanticipate your paper as one part of the entire proceedings, andnot as an independent document. Please do not revise any ofthe current designations.

The GEMOM system architecture is composed of a set ofcommunicating nodes (GEMOM Nodes, or G-Nodes). Some ofthese G-Nodes are operational (micro) nodes and somemanagerial (macro) nodes. The operational G-Nodes can beclassified as Message Brokers, Clients (either publishing orsubscribing messages), Authentication and AuthorizationModules, Anomaly Detector Security Measurement Managers,etc. They communicate with managerial nodes of differenttypes. The managerial G-Nodes can be classified as AdaptiveSecurity Managers, Audit and Logging Modules, andMonitoring Tools with associated Security Monitors and QoSMonitors, Resilience Managers, Security Anomaly Managers,etc. The managerial G-Nodes make decisions about the runtime operation of the system and require a wider perspectivethan the individual operational G-Nodes. In GEMOM, aMessage Broker is a package consisting of an applicationserver, numerous plug-and-play objects, configuration files,and database schemas.

C. Use of Security Metrics in GEMOMThe development of security metrics for the GEMOM

system is a central activity. Metrics are used for differentpurposes:

Security and QoS monitoring,Adaptive Security Management, andSecurity engineering and software security assurance ofthe system and service. In addition to technical securitymetrics, metrics designed for security engineering, e.g.SSE-CMM [13]), in software development lifecycle areused.

D. GEMOM Monitoring SystemThe GEMOM Monitoring Tool is responsible for the

collection of evidence of security and QoS and the maintenanceof appropriate metrics database in the GEMOM environment.There is one Monitoring Tool per Message Broker. TheGEMOM Monitoring Tool consists of the Monitor Core (MC)process and the Monitor Modules. The MC is runs in thebackground and the Monitor Modules are added during runtimeto the MC. The MC offers database and messaging services tothe Monitor Modules. It connects to a GEMOM Broker and themodules use it to publish and subscribe to certain monitoringtopics in a namespace.

E. Concept of State of SecurityIn practice, the GEMOM security monitoring and adaptive

security management utilize a holistic State of Security (SoS)concept. SoS is a time-dependent estimate of the system’ssecurity performance level based on the appropriate collectionof security metrics that is calculated initially and whentriggered. There are five steps in the estimation of the SoS :

1. Initial SoS: Define the initial SoS using appropriatesecurity metrics.

2. Current SoS: Measure the SoS whenever triggered by atimer, an attack, an anomaly or a manual request.

3. Compare: Past and Current SoS can be compared to offerinput to the trend estimation in decision-making.

4. Adapted SoS: The initial SoS can be adapted according todecisions made by the ASM functionality.

5. Predicted SoS: A future SoS can be predicted to enableproactive Adaptive Security Management.

Figure 2. A timeline visualization of SoS estimates.

Fig. 2 depicts the visualization of the role of different typesof SoS estimates. The predicted SoS is based on the analysis ofthe past history of the SoS levels and threat and vulnerabilitytrends. The predicted SoS is useful when carrying out proactiveoperations to ensure high resilience of the system.

III. SECURITY METRICS DEVELOPMENT APPROACH

The analytical security metrics development approachapplied in the GEMOM system, and more comprehensiveresults of the threat analysis of the GEMOM system and theidentification of BMCs are presented in [9]. In the following,we review the development approach briefly.

A. Original Security Metrics Development ProcessSavola and Abie [9] define the following holistic security

metrics development process:

1. Carry out threat and vulnerability analysis of the systemunder investigation and its use environment (use cases).Identify known or suspected vulnerabilities. Furthermore,carry out appropriate (business) impact and risk exposureanalyses. Both impact and risk exposure estimates have asignificant effect on the prioritization.

2. Define and prioritize security requirements in a holisticway based on the threat and vulnerability analysis. Themost critical security requirements should be given themost attention.

3. Identify Basic Measurable Components from the higher-level requirements using a decomposition approach.

4. Define measurement architecture for on-line metrics andevidence collection mechanisms for off-line metrics.

5. Select BMCs based on feasibility and importance.6. Develop appropriate off-line and on-line security metrics,

and the functionalities and processes where they are used.

The process is highly iterative and the sequence of the stepscan be varied. Steps 1 and 2 should be started as early aspossible in the system development process and elaboratediteratively as the system design becomes more mature.Identification of BMCs and measurement architecturedefinition can be carried out in parallel to each other. The lattercan also be partially initiated during the architectural designphase. The process is based on the approach by Savola [8][10],enhanced with decomposition, evidence collection andemphasis on feasibility and importance [9].

B. Security Requirement DecompositionThe core activity in the security metrics development

process is in the decomposition of the security requirements.The following decomposition process, based on Wang andWulf, [14] is used to identify measurable components from thesecurity requirements:

1. Identify successive components from each securityrequirement (goal) that contribute to the achievement ofthe goal.

2. Examine the subordinate nodes to see if furtherdecomposition is needed, and if so, repeat the process

with the subordinate nodes as current goals, breakingthem down to their essential components.

3. Terminate the decomposition process when none of theleaf nodes can be decomposed any further, or whenfurther analysis of these components is no longernecessary.

When the decomposition terminates, all leaf nodes shouldbe measurable components.

As an example, the model depicted in Fig. 3 can be used forthe authentication decomposition [14] during the process ofidentifying potential metrics for the authentication strength.

Figure 3. A decomposition of authentication.

IV. SOURCES OF BMCS IN GEMOMIn this section, we identify security-related BMCs in the

GEMOM system. In the security metrics developmentapproach of [9], the BMCs are derived from the decompositionof security requirements. In this section, we present additionalsources of BMCs: anomalies, attacks, vulnerabilities,compromises, trust, reputation, and software quality.

A. BMCs from Requirement DecompositionIn the following, we list some BMCs of the GEMOM

system obtained by the decomposition process. See [9] for amore detailed discussion of GEMOM security requirementsand their decomposition. The high-level classification ofGEMOM security requirements is shown in Fig. 4. Most of themetrics listed here can be used in the definition of the initialand adapted SoS.

Figure 4. Classification of GEMOM security requirements.

The adaptive security in GEMOM is based on variousadaptive trust metrics emphasizing the different securitydimensions (numbers 2 to 7 in Fig. 4). The adaptive security inGEMOM allows the system to choose among differentauthentication and authorization methods and encryptionstrength based on trust, reputation and security metrics. Input

from Security and QoS monitoring systems are used fordecision-making.

TABLE I. BMCS OF AUTHENTICATION

Symbol Basic Measurable Component

U(id) Uniqueness of identityS(id) Structure of identityI(id) Integrity of identityR(am) Reliability of authentication mechanismI(am) Integrity of authentication mechanism

The confidentiality requirements define message, log andmetadata confidentiality and storage confidentiality, includingthe cryptographic strength requirements. Integrity requirementsaddress especially message, log and metadata integrity,environment integrity and persistent data integrity. Non-repudiation requirements address the non-repudiation ofpublished and received messages, non-repudiation of origin orreceipt and non-repudiation of triggered push.

Based on the results of the GEMOM threat analysis [9],DoS (Denial of Service) attacks are seen a very significantthreat to the availability of the GEMOM system. The impact ofavailability threats in the system is high because the system, ora core part of it, is especially in its most vulnerable state duringavailability attacks. Exploiting the high vulnerability timewindow, the attacker can execute their strategy to reach theirgoals, potentially also causing threats to other securitydimensions. The intruder can even seize the system using thisstrategy. In GEMOM, DoS attacks are monitored by the DoSAnomaly Detector Monitor. This functionality uses modelsbased on rates of messages, routing paths and time intervaldistributions. Traditionally, availability is the percentage oftime the target system is up (the time-averaged, binary-view ofsystem state – up or down) or information is available.However, this notion does not capture degraded states, a non-binary spectrum between “up” and “down”.

B. Anomaly and Misuse Models as BMCsIn GEMOM, anomaly monitoring consists of anomaly

detection processes and a hierarchy of anomaly correlationprocesses that combine the output anomalies from a set ofanomaly detectors. Attacks and faults often demonstrateanomalous characteristics in different ways and in differentparts of the system. Anomaly detectors are operated bycombining outputs of a set of models. The models describe thevarious aspects of the normal patterns of the system. Thedetectors are used to train a set of models, to detect anomaliesbased on the trained models and to aggregate the results, e.g.with simple weighted sums or naïve Bayesian networkapproaches. Each anomaly is assigned a probability or score.The difference between anomaly and misuse models is that theformer depict normal behaviour and the latter attack, “misuse”behaviour. Both kinds of models are used in the GEMOMMonitoring System. Authentication metrics based on anomalyand misuse models depict, e.g., the number of authentication

failures, the proportion of failed authentications andauthentication trends. False positives in authentication indicateattackers falsely permitted access and false negatives indicateauthorized users who are prevented from accessing the systemthey should be able to use. In the federated identitymanagement and single sign-on, normal use patterns are basedon the use case description or recording from the system. Inauthorization, there are many possibilities for attacker-orientedmetrics. Most authorization metrics can be based on the log andmetadata information of the users and objects they access or tryto access. This data can be used to investigate usage trends inauthorization and to track extraordinary user behaviour.

C. Availability Metrics Based on Application Performanceand QoSQoS metrics concentrate typically on the application

performance of the system. From the security performancepoint of view, application performance and other availabilityissues are of utmost importance. QoS monitoring in GEMOMis implemented by defining appropriate anomaly and faultmodels to be utilized in the detection.

D. Cryptographic Strength MetricsCryptographic strength metrics [5] are a special family of

algorithm-oriented security metrics that can be utilized in manycases, and especially in integrity, confidentiality andauthentication. Cryptographic strength metrics include keylength metrics, attack time metrics, rounds metrics andalgorithm strength metrics. In GEMOM, cryptographic metricsare used to set the initial SoS estimates to the strength ofauthentication, integrity, confidentiality algorithms and non-repudiation algorithms, and to adapt the estimates whenalternatives algorithms are in use.

E. Additional Security Oriented MetricsIn addition to metrics representing the security behaviour of

the system, there are a variety of security-oriented metrics. Thescope of all these security-oriented metrics is broader than thesecurity performance of the system. In addition to securitymetrics, the GEMOM Adaptive Security Managementfunctionality utilizes trust and reputation metrics to makeinformed decisions about the SoS. Software quality metrics is ahorizontal family of metrics, addressing not only security flawsbut the overall software quality. An important part of softwarequality metrics is the family of (software) vulnerability metrics.CWE (Common Weakness Enumeration) [3] is a promisingsystematic collaboration effort to classify vulnerabilities.

F. System Endemic Security MetricsIn practical systems, there can be a range of endemic

measurement activities that can be reused for the purposes ofsecurity and QoS measurements.

In GEMOM, some averaged counters measuring machinerelated information are being calculated and updated in certaintime intervals. Counters utilized in for the GEMOM securityand QoS measurements are listed in Table II and Table III. Thecounters of Table III can be used as a component foravailability metrics and DoS attack detection. Increased

bandwidth utilization and latency between different machinesare symptoms of such attacks. Publish/subscribe activity relatedcounters can be monitored to see if there is actual functionalityin the system during that time.

TABLE II. MACHINE RELATED COUNTERS

Symbol Counter

cpuI, cpuP CPU idle time, CPU processing timeaMEM Available memorypACT Paging activitybU, bM Bandwidth utilization, maximum bandwidthlM, vM Latency, visibility between two machines

TABLE III. PUBLISH/SUBSCRIBE ACTIVITY RELATED COUNTERS

Symbol Counter

pN, pT Publications per namespace, topicpB, pC Publications per Broker, Client

mB, mCMessages per Broker (publications andsubscriptions), Client

mCLI Messages per ClientdB, dC Amount of data per Broker, ClientptCLI Number of protocol breaches per ClientnN, nT Number of namespaces, topics

TABLE IV. SOURCES OF SECURITY AND RELATED BMCS IN GEMOM

Symbol 1 2 3 4 5 6 7Countermeasuremechanism performancemetrics (by requirementdecomposition)

× × × × × × ×

Metrics based onanomaly and misusemodels (attacker-oriented weaknessmetrics)

× ×

Cryptographic strengthmetrics × × × ×

Availability metricsbased on QoSapplication performancemetrics

×

Trust metrics ×Reputation metrics ×Software quality metrics × × × × × × ×Vulnerability metrics × × × × × × ×System endemicsecurity-relevant metrics ×

G. Holistic View of the Sources of BMCsTable IV summarizes the origin of BMCs for security

metrics and other security-relevant metrics in GEMOM. The

numbers in the columns represent the security dimensions ofFig. 4 (1 = adaptive security, 2 = authentication, 3 =authorization, 4 = confidentiality, 5 = integrity, 6 =availability and 7 = non-repudiation). “×” in the relevant boxindicates the use of the security metrics of the respective BMCtype in GEMOM.

Until recently, QoS and security technologies and metricshave lived in separate worlds. Some security attacks affectapplication performance – and ensuring applicationperformance is the most important goal of QoS. During thepractical development of the GEMOM monitoring approach,these two technologies and their respective metrics have beenintegrated.

V. ENHANCED SECURITY METRICS DEVELOPMENTPROCESS

Obviously, our original security metrics developmentprocess [9] must be updated in accordance with the results ofthis study. Security requirement decomposition is not the onlysource of potential BMCs. The BMCs originating from securityrequirements concentrate on the strengths of securitymechanisms and algorithms. Attacker strategies are not takeninto account explicitly. On the basis of the results of this study,we can improve the security metrics development process of[9] in the following way. (a), (b), and (c) represent parallelactivities:

1. Carry out threat and vulnerability analysis of the systemunder investigation and its use environment (use cases).Identify known or suspected vulnerabilities. Furthermore,carry out appropriate (business) impact and risk exposureanalyses. Both impact and risk exposure estimates have asignificant effect on the prioritization.

2. (a) Define and prioritize security requirements in aholistic way based on the threat and vulnerability analysis.The most critical security requirements should be giventhe most attention. (b) Model relevant attack strategies inprioritized order and carry out cost-benefit analysis fromthe point of view of the attacker. (c) Select appropriateQoS metrics for the security-oriented availability metrics.

3. (a) Identify Basic Measurable Components from thehigher-level requirements using a decompositionapproach. (b) Develop anomaly and misuse models basedon the attack strategies for anomaly detection. (c)Integrate chosen QoS metrics in the availability BMCsobtained using decomposition.

4. Define measurement architecture for on-line metrics andevidence collection mechanisms for off-line metrics. Payattention to readily available counters, measurementpoints, etc.

5. Select BMCs based on feasibility and importance.Estimated threat impacts and exposures affect to theimportance.

6. Apply suitable metrics ontologies to the chosen basicmeasurable components to plan the actual metricsdevelopment. See [6] for an example.

7. Develop appropriate detailed off-line and on-line securitymetrics, and the functionalities and processes where they

are used. In parallel, integrate metrics from other sourceslike trust metrics, reputation metrics, software qualitymetrics, vulnerability metrics and system-endemic metricsto the metrics collection.

VI. RELATED WORK

Heyman et al. [4] utilize a security objective decompositionapproach to define a security metrics framework. Theyassociate security patterns with respective security metrics andexploit the relationships between them to allow theinterpretation of measurements. CVSS [11] (CommonVulnerability Scoring System) is an initiative that provides anopen method for rating vulnerabilities. The U.S. NIST(National Institute of Information Standards and Technology)SAMATE (Software Assurance Metrics and Tool Evaluation)project [2] develops tools and metrics for software assurance.OWASP (Open Web Application Security Project) [7] carriesout collaborative development of security metrics and containsan active discussion forum for developers. CWE (CommonWeakness Enumeration) [3] is an example of systematicvulnerability classification that can be used as a basis forvulnerability metrics. Seddigh et al. introduce informationassurance metrics taxonomy for IT Network assessment in[12]. Their taxonomy divides the metrics space into threecategories – Security, Quality of Service (QoS) andAvailability – based on their novel definition of informationassurance. Under each of these three they consider technical,organizational and operational metrics. According to them,technical security metrics include sub-categories for productrating, incident statistics and security testing. In the QoScategory, they only propose technical metrics, includingproduct capabilities, network capabilities and QoS tests.

VII. CONCLUSIONS

In the GEMOM system, the security monitoring approachand adaptive security management are based on a State ofSecurity (SoS) concept, where the security performance levelof the system can be estimated, compared and predicted basedon security metrics. In addition, security metrics are usedduring the GEMOM system development for securityengineering and software security assurance purposes. InGEMOM, the Basic Measurable Components for securitymetrics originate from applying a security requirementdecomposition approach. Most of the metrics obtained usingthis method form the initial and adapted SoS estimate. Most ofthe non-attacker strategy oriented security metrics have beendeveloped using this method. On the other hand, attackerbehaviour oriented metrics have been developed using anomalyand misuse models. Availability in GEMOM, as often in webapplications, is critical to the system as DoS attacks are themain threat. It is noteworthy that availability metrics originatefrom both requirement decomposition and from applicationperformance metrics developed for the QoS managementsystem. Our practical security metrics development in theGEMOM system has shown clearly that security and QoS

mechanisms and related metrics are closely interrelated andshould be exploited for both purposes. Obviously, moreresearch work on coherent merging of security and QoSmetrics is needed. Furthermore, system endemic measurementshould be exploited for the purposes of security measurements.As our experience with the GEMOM system indicates,availability metrics are often easily available in practicalsystems.

ACKNOWLEDGMENT

The work presented in this paper has been carried out in theGEMOM FP7 research project, Grant Agreement No. 215327,partly funded by the European Commission.

REFERENCES

[1] Abie, H., Dattani, I., Novkovic, M., Bigham, J., Topham, S. and Savola,R. 2008. GEMOM – Significant and measurable progress beyond thestate of the art. In Proc. of ICSNC 2008, Sliema, Malta, Oct. 26-31, pp.191-196.

[2] Black, P. E. 2006. SAMATE’s contribution to information assurance,”IAnewsletter, Vol. 9, No. 2.

[3] CWE: Common Weakness Enumeration. http://cwe.mitre.org/[4] Heyman, T., Scandariato, R., Huygens, C., Joosen, W. 2008. Using

security patterns to combine security metrics. In Proc. 3rd Int. Conf. onAvailability, Reliability and Security (ARES), pp. 1156-1163.

[5] Jorstad, N. and Landgrave, T. S. 1997. Cryptographic algorithm metrics.In Proc. 20th National Information Systems Security Conference,Baltimore, MD, Oct. 1997.

[6] Niemelä, E., Evesti, A. and Savolainen P. 2008. Modeling qualityattribute variability. In Proc. of the 3rd Int. Conference on Evaluation ofNovel Approaches to Software Engineering, 4-7 May, Funchal, Madeira,Portugal, Institute for Systems and Technologies of Information, Controland Communication (INSTICC), pp. 169-176.

[7] OWASP: Open Web Application Security Project.http://www.owasp.org/

[8] Savola, R. 2007. Requirement centric security evaluation of softwareintensive systems, In Proc. of 2nd Int. Conf. on Dependability ofComputer Systems DepCOS-RELCOMEX 2007, Szklarska Poreba,Poland, June 14-16, pp. 135-142.

[9] Savola, R. and Abie, H.. 2009. Identification of basic measurablesecurity components for a distributed messaging system. In Proc. ofSECURWARE 2009, Athens, Greece, June 18-23, 8 p. (in press)

[10] Savola, R. and Abie, H.. 2009. On-Line and off-Line securitymeasurement framework for mobile ad hoc networks”, Journal ofNetworks, Special Issue on Security of Wireless CommunicationSystems, 15 p. (in press)

[11] Schiffman, M., Eschelbeck, G., Ahmad, D., Wright, A. and Romanosky,S. 2004. CVSS: A Common Vulnerability Scoring System. NationalInfrastructure Advisory Council.

[12] Seddigh, N., Pieda, P., Matrawy, A., Nandy, B., Lambadaris, I., Hatfield,A. 2004. Current trends and advances in information assurance metrics.In Proc. of the 2nd Ann. Conf. Privacy, Security and Trust (PST), 2004.

[13] SSE-CMM: Systems Security Engineering – Capability Maturity Model.http://www.sse-cmm.org/

[14] Wang, C. and Wulf, W. A. 1997. Towards a framework for securitymeasurement. In Proc. of 20th Nat. Information Systems Security Conf.,Baltimore, MD, Oct., pp. 522-533.