7
MAAP: Mission Assurance Analytics Platform Thomas Llans´ o Johns Hopkins University Applied Physics Laboratory Laurel, Maryland 20723 Email: [email protected] Telephone: (443) 778–6343 Peter A. Hamilton Johns Hopkins University Applied Physics Laboratory Laurel, Maryland 20723 Email: [email protected] Telephone: (443) 778–2163 Michael Silberglitt Johns Hopkins University Applied Physics Laboratory Laurel, Maryland 20723 Email: [email protected] Telephone: (443) 778–8261 Abstract—This paper describes the Mission Assurance Analyt- ics Platform (MAAP), an open, experimental software framework that provides analysts with an environment for systematically studying the link between cyber attack and the resulting impact on operational missions that are supported by a cyber system. MAAP directly informs both risk decisions and mitigation prioritization. Index Terms—Mission, Cyber, Mission Assurance, Risk, Ar- chitecture, Analytics, Framework, Mitigations, MBA, MIRA I. I NTRODUCTION As civilian and government information technology (IT) systems become more complex, mobile, and interconnected, cyber adversaries are presented with ever expanding opportu- nities to impact the missions that those systems support by exploiting the potentially large attack surfaces that vulnerable IT architectures expose. Because traditional government and civilian cyber security processes are only weakly tied to the larger mission, information assurance (IA) professionals often prescribe security controls without a clear understanding as to their mission value and contribution to risk reduction. Against this backdrop, this paper presents the Mission Assurance Analytics Platform (MAAP), an open, experimental software framework that provides analysts with an environ- ment for systematically studying the link between cyber attack and the resulting impact on missions that are supported by a target cyber architecture. MAAP is constructed around a methodology called Mission Information Risk Analysis (MIRA). MIRA can be applied to systems in the design stage or to systems already deployed. MIRA is itself an information-centric variant of a more general mission as- surance methodology called Mission-Based Analysis (MBA). MBA is a process for helping mission owners understand the degree to which their missions are imperiled by various threats, such as cyber and kinetic threats, and how to prioritize and pinpoint mitigations for maximum benefit in the face of these threats. Experience applying MIRA has demonstrated the strong need for automation assistance throughout the five major activ- ities of MIRA, especially for analysis of large cyber systems in which the combinatorics of certain activities preclude the use of manual methods. MAAP supports analysis automation in an extensible fashion by providing the ability to plug in and orchestrate the use of different analytic methods. The paper is organized as follows. Section II discusses related mission assurance work and the role MAAP could play in this type of analysis. Sections III and IV describe MIRA and MAAP respectively. Section V describes experiences experimenting with and applying MAAP in practical settings. The last two sections cover future work and conclusions. II. RELATED WORK There have been several examples of mission-focused cyber methodologies in the past. Buckshaw, et al [1] introduced an early example of cyber mission impact analysis, Mission Oriented Risk and Decision Assessment (MORDA). MORDA involves the creation of a mission impact utility function that can be applied to all data exchanges. MORDA requires a detailed technical and quantitative survey of all aspects of the network analyzed. Burris, et al [2] have researched a methodology for assessing the impact of attack on particular networks with respect to mission capability. This technique has historically focused on availability and requires detailed estimates on network supply and demand to perform the mission. Mudge and Lingley [3] proposed a technique for capturing the operational behaviors and impacts that result from a cyber compromise. Their technique is centered on live exercises. Musman, et al [4] have developed techniques for estimating mission impact as a result of cyber attacks. Their approach is concerned with target responses to attacks while MIRA is geared towards identifying not only what responses could be enacted (e.g., tactics, procedures), but also what technologies or architectural changes could be employed to mitigate impact. Cyber Investment Analysis Methodology (CIAM) [5] pro- vides a technique for prioritizing the use of security controls based on consideration of multiple factors, including business impact, attack type, frequency, and security control acquisition and operational costs. What is common across all of the above approaches is that they have not been integrated into a more general mission assurance framework, such as the one that MAAP provides for MIRA. That said, MAAP may be able to incorporate tech- niques from these related methodologies as distinct analytic plug-ins. In fact, an analytic plug-in for CIAM has already been developed and is useful during the mitigation derivation activity of MIRA. 978-1-4673-2709-1/12/$31.00 ©2012 IEEE 549

[IEEE 2012 IEEE International Conference on Technologies for Homeland Security (HST) - Waltham, MA, USA (2012.11.13-2012.11.15)] 2012 IEEE Conference on Technologies for Homeland Security

  • Upload
    michael

  • View
    216

  • Download
    3

Embed Size (px)

Citation preview

Page 1: [IEEE 2012 IEEE International Conference on Technologies for Homeland Security (HST) - Waltham, MA, USA (2012.11.13-2012.11.15)] 2012 IEEE Conference on Technologies for Homeland Security

MAAP: Mission Assurance Analytics PlatformThomas Llanso

Johns Hopkins UniversityApplied Physics Laboratory

Laurel, Maryland 20723Email: [email protected]

Telephone: (443) 778–6343

Peter A. HamiltonJohns Hopkins University

Applied Physics LaboratoryLaurel, Maryland 20723

Email: [email protected]: (443) 778–2163

Michael SilberglittJohns Hopkins University

Applied Physics LaboratoryLaurel, Maryland 20723

Email: [email protected]: (443) 778–8261

Abstract—This paper describes the Mission Assurance Analyt-ics Platform (MAAP), an open, experimental software frameworkthat provides analysts with an environment for systematicallystudying the link between cyber attack and the resulting impacton operational missions that are supported by a cyber system.MAAP directly informs both risk decisions and mitigationprioritization.

Index Terms—Mission, Cyber, Mission Assurance, Risk, Ar-chitecture, Analytics, Framework, Mitigations, MBA, MIRA

I. INTRODUCTION

As civilian and government information technology (IT)systems become more complex, mobile, and interconnected,cyber adversaries are presented with ever expanding opportu-nities to impact the missions that those systems support byexploiting the potentially large attack surfaces that vulnerableIT architectures expose. Because traditional government andcivilian cyber security processes are only weakly tied to thelarger mission, information assurance (IA) professionals oftenprescribe security controls without a clear understanding as totheir mission value and contribution to risk reduction.

Against this backdrop, this paper presents the MissionAssurance Analytics Platform (MAAP), an open, experimentalsoftware framework that provides analysts with an environ-ment for systematically studying the link between cyber attackand the resulting impact on missions that are supported bya target cyber architecture. MAAP is constructed arounda methodology called Mission Information Risk Analysis(MIRA). MIRA can be applied to systems in the designstage or to systems already deployed. MIRA is itself aninformation-centric variant of a more general mission as-surance methodology called Mission-Based Analysis (MBA).MBA is a process for helping mission owners understand thedegree to which their missions are imperiled by various threats,such as cyber and kinetic threats, and how to prioritize andpinpoint mitigations for maximum benefit in the face of thesethreats.

Experience applying MIRA has demonstrated the strongneed for automation assistance throughout the five major activ-ities of MIRA, especially for analysis of large cyber systemsin which the combinatorics of certain activities preclude theuse of manual methods. MAAP supports analysis automationin an extensible fashion by providing the ability to plug in andorchestrate the use of different analytic methods.

The paper is organized as follows. Section II discussesrelated mission assurance work and the role MAAP could playin this type of analysis. Sections III and IV describe MIRAand MAAP respectively. Section V describes experiencesexperimenting with and applying MAAP in practical settings.The last two sections cover future work and conclusions.

II. RELATED WORK

There have been several examples of mission-focused cybermethodologies in the past. Buckshaw, et al [1] introducedan early example of cyber mission impact analysis, MissionOriented Risk and Decision Assessment (MORDA). MORDAinvolves the creation of a mission impact utility function thatcan be applied to all data exchanges. MORDA requires adetailed technical and quantitative survey of all aspects of thenetwork analyzed.

Burris, et al [2] have researched a methodology for assessingthe impact of attack on particular networks with respect tomission capability. This technique has historically focused onavailability and requires detailed estimates on network supplyand demand to perform the mission.

Mudge and Lingley [3] proposed a technique for capturingthe operational behaviors and impacts that result from a cybercompromise. Their technique is centered on live exercises.

Musman, et al [4] have developed techniques for estimatingmission impact as a result of cyber attacks. Their approachis concerned with target responses to attacks while MIRA isgeared towards identifying not only what responses could beenacted (e.g., tactics, procedures), but also what technologiesor architectural changes could be employed to mitigate impact.

Cyber Investment Analysis Methodology (CIAM) [5] pro-vides a technique for prioritizing the use of security controlsbased on consideration of multiple factors, including businessimpact, attack type, frequency, and security control acquisitionand operational costs.

What is common across all of the above approaches is thatthey have not been integrated into a more general missionassurance framework, such as the one that MAAP providesfor MIRA. That said, MAAP may be able to incorporate tech-niques from these related methodologies as distinct analyticplug-ins. In fact, an analytic plug-in for CIAM has alreadybeen developed and is useful during the mitigation derivationactivity of MIRA.

978-1-4673-2709-1/12/$31.00 ©2012 IEEE 549

Page 2: [IEEE 2012 IEEE International Conference on Technologies for Homeland Security (HST) - Waltham, MA, USA (2012.11.13-2012.11.15)] 2012 IEEE Conference on Technologies for Homeland Security

III. MISSION INFORMATION RISK ANALYSIS (MIRA)

This section provides an overview of MIRA. Having abasic understanding of MIRA is critical to understandingMAAP, as it is a platform for implementing MIRA activities.MIRA systematically analyzes an operational mission, cyberthreats to the mission, and the IT infrastructure that supportsthe mission. Over the last three years, MAAP has beensuccessfully applied to existing systems as well as systems inthe requirements/design phases, including military planning,satellite and ship-board cyber-intensive systems. The goal ofMIRA is to find answers to the following questions:

1) What is the mission impact of different classes ofcyber attack that might be expected from the cyberadversaries?

2) What is the estimated level of effort (LOE) for anadversary to carry out the different attack classes?

3) What mitigations are possible for those attacks that areboth highly mission impacting and are estimated to berelatively easy for an adversary to carry out?

A. Models

As described below, MIRA employs three underlying mod-els: mission, system architecture, and adversary. The level ofabstraction of each model is driven by the desired fidelity levelof results as well as the amount of time and resources affordedfor performing the analysis.

The mission model, MM, is a five tuple (M,A,B, F,D).M is a set of distinct mission types supported by the targetIT system. A is a set of quantitative mission Measures ofEffectiveness (MOEs) that describe critical performance re-quirements that must be met to achieve missions in M . Bis a set of quantitative system-level MOEs required to realizethe mission-level MOEs in A. F is a set of mission essentialfunctions whose invocation directly impacts MOEs in B and,transitively, in A. Finally, D is a set of required informationelements acted upon by the functions in F . In general, thereis a many-to-many mapping of MM elements.

The system architecture model, SM, is a four tuple,(T,N,L,C). T is a set of node types, instances of whichare found in the system. A node is defined as an active entitycapable of carrying out computation and/or communicationoperations (e.g., router, switch, desktop PC, laptop, server,wireless device). N is a set of those node instances in thearchitecture, L is a set of links between node instances, andC is a mapping of data types from D in the mission modelto N . A given link represents connectivity, typically in anetwork/communication context, between two node instances.

The adversary model, AM, consists of an estimate of themaximum LOE that the anticipated worst-case cyber adversarymight muster against the target system, recognizing that notevery system is of interest to high-capability cyber adversaries.The value ranges from 1 to 10, where 1 indicates minimalexertion on the part of the attacker and 10 indicates maximallevels of exertion corresponding to the capabilities possessedby a top nation state attacker, with years of specialized R&D,

high tolerance for failure, and need for millions of dollars infunding.

B. Activities

Figure 1 lists the names of the MIRA activities and relateddata flows that help arrive at answers to the key questionsraised earlier. As MIRA is designed as an agile process, newinformation learned in later activities can be usefully fed backinto earlier activities. Thus MIRA is depicted as a spiralallowing for an incremental, iterative work style. Below wedescribe the typical MIRA realization of these activity names.

In the Derive Models activity, analysts populate MM, SM,and AM. They can do so via a number of techniques, suchas reviewing relevant documentation (if it exists), interviewingmission and system experts, and, if permitted for an existingsystem, running automated discovery analytics (e.g., trafficanalysis and IP/port/service scanning tools) against operationalmission-system environments.

Fig. 1. MIRA Process Overview

In detailing the models, analysts also capture how missiondata flows over the system nodes in the context of differentmission essential functions. To characterize the adversary, theanalyst considers the mission in the context of different kindsof potential adversaries who might wish to harm the mission.The expected worst-case adversary from this list then becomesthe focus, and an associated LOE capability is estimated.

In the Perform Scoring activity, analysts derive two distincttypes of scores: mission impact scores and attack LOE scores.In the most detailed case, a mission impact score is assignedfor each viable tuple from (M,A,B, F,D,N,CT ), where M ,A, B, F , D, and N were defined earlier and CT stands forCompromise Type. CT is one of: Confidentiality, Integrity,or Availability. A mission impact score is an ordinal value inthe range of 1 to 5, where 1 means “fully mission capable”(i.e., no mission impact) and 5 means “not mission capable”

550

Page 3: [IEEE 2012 IEEE International Conference on Technologies for Homeland Security (HST) - Waltham, MA, USA (2012.11.13-2012.11.15)] 2012 IEEE Conference on Technologies for Homeland Security

(i.e., mission fails). An LOE score is assigned for each viabletuple, (N,D,CT,AV ) where N , D, and CT were definedearlier and AV is Attack Vector. Typical attack vectors areNetwork, Insider, and Supply Chain Implant. There are variousapproaches to determining each type of scoring, each withits strengths and weaknesses. Analytic plug-ins allow for theuse of alternative scoring approaches. A detailed survey anddiscussions of scoring procedures is omitted in this paper dueto length constraints.

In the Estimate Risk activity, analysts typically create atype of risk matrix called a heat map (Figure 2). The x-axisof the heat map depicts mission impact scores; the y-axisin turn represents LOE scores, with the highest LOE scoresarranged to be closest to the origin. In this arrangement, thecyber attacks that are both highly mission impacting and whichrequire a relatively low LOE appear in the upper right-handquadrant of the heat map. Such attack contexts represent thehighest risk to the mission, and they become key inputs to theremaining activities.

Fig. 2. MBA Heat Map

In the Derive Mitigations activity, analysts consider differ-ent combinations of security controls, such as those describedin NIST 800−53 [6], and the effectiveness of those controls atcountering the prioritized attack contexts identified in the heatmap. Mitigation derivation has traditionally been performedmanually by analysts, though techniques such as CIAM canhelp make the process more rigorous and objective. As wasmentioned earlier, MAAP has a CIAM plug-in to assistanalysts with this activity. Once mitigations have been derived,one turns next to evaluating the mitigations.

In the Evaluate Mitigations activity, analysts refactor thesystem architecture model accordingly with the derived mit-igations in mind, and re-score the LOE values of affectedattack contexts. A key goal in this activity is to increase theLOE values via well-placed mitigations. Such increased LOEvalues can help reduce overall mission risk, particularly if therecomputed LOE values exceed the estimated capabilities ofthe cyber adversary in AM. In the end, analysts produce a setof recommendations based on the overall analysis, includingmitigations. For existing systems, additional evaluation and

validation of the mitigations can take place by suitably skilledpenetration testers once the mitigations have been put intoplace in the actual cyber architecture.

C. Workflow and Analytics

As was mentioned earlier, MIRA can be applied acrossa variety of different mission/system/threat contexts as wellas systems at different points in their lifecycle (e.g., design,deployment). This variety combined with the time/fidelityrequirements of different analyses calls for flexibility in thetypes of analytics that analysts can select and employ togetherin orchestrated workflows. Examples of analytics apper inTable I.

TABLE IANALYTIC PLUG-IN EXAMPLES

MIRA Activity Analytic Plug-in Example

Derive ModelsFlat-file data translation and ingestAuto-discovery via external scans

Perform Scoring

Cyber modeling and simulation withMonte Carlo-driven cyber attacksAttack path analysisPetri Net attack progression

Estimate Risk Attack likelihood estimation

Derive MitigationsCIAM security control prioritizationIndex portfolio experimentation

To describe one example from Table I, consider the “Per-form Scoring” MIRA activity. Historically MIRA analysts inconsultation with mission experts have performed mission-impact scoring manually. Unfortunately, the manual approachdoes not scale to common instances of MM (e.g., cardinalitiesof 5, 7, 7, 10, 20 for the respective components of MM,together with 3 attack types (confidentiality, integrity, avail-ability) and 50 nodes from SM). This example implies theneed to assign 7.35 million distinct mission impact scores.Identification of equivalence classes may reduce this number,but manual, fine-grained mission impact scoring is in generalinfeasible.

A modeling and simulation approach may be capable ofaddressing the combinatorial explosion implicit in missionimpact scoring, automatically providing a robust set of missionimpact scores without extensive manual input. A possible setupmight include a mission model, cyber model, network model,attacker model, and defender model working within a missionsimulation. These models can allow for automated scoring, asfollows: A Monte Carlo cyber attacker repeatedly samples atotal space of attacks by randomly choosing a target cybernode, attack type, and mission data element. The attack iscaried out in the simulation by updates to the cyber model.The impact of these attacks show up in changes to variousmeasures of effectiveness in the mission model that relies onthe cyber model. These impacts are automatically translatedto mission impact scores.

An automation platform is required to execute the analyticworkflows, some of which are both computation and data

551

Page 4: [IEEE 2012 IEEE International Conference on Technologies for Homeland Security (HST) - Waltham, MA, USA (2012.11.13-2012.11.15)] 2012 IEEE Conference on Technologies for Homeland Security

intensive. The next section describes MAAP, a platform cur-rently being developed and applied on an experimental basisfor this purpose.

IV. MISSION ASSURANCE ANALYTICS PLATFORM(MAAP)

MAAP is an experimental automation platform designed to(1) reduce the manual workload of cyber analysts conductingMIRA studies and (2) help increase the fidelity of the analysisresults by facilitating the use of various analytic plug-ins.The sections below describe the overall MAAP architecture,data management, the plug-in architecture, and interactivevisualizations.

A. Architecture

The MAAP architecture (Figure 3) employs a set of lay-ered components that share an underlying database. Workingfrom the bottom of the figure up, the persistence componentmanages MAAP data via the underlying repository, currentlyrealized as a relational database. The workflow componentallows for execution of named workflows, which are orderedsets of analytics that implement MIRA activities. The MAAPkernel connects and coordinates the various layered compo-nents in MAAP. The analytics manager component managesanalytic plug-ins and their allocation to named workflows. Thepresentation component provides a windowed user interfaceenvironment to display visualization results and to allow foranalytic plug-in management and execution.

Fig. 3. MAAP Architecture

For the sake of portability and extensibility, MAAP softwareis written in the Java programming language. The MySQLdatabase [7] is currently plugged in beneath the Persistencecomponent to store data. Minimal changes are required withinthe Persistence component to accommodate a different rela-tional database system.

B. Data Management

The underlying database stores the models MM, SM, andAM, intermediate processing results of analytic plug-ins, theultimate results processed by those same plug-ins (e.g., scoringdata), and other general MAAP configuration information,such as name analytic workflows and their mappings to

analytic plug-ins. The database construct alleviates the needfor large, manually updated spreadsheets of data that previousMIRA analyses relied upon, giving the analyst more flexibilityand time to perform the analysis. MAAP components accessthe database via the Persistence component. Upon initializa-tion, MAAP prompts the user for database login information.MAAP allows users to view the database connection infor-mation at any time as well as switch to a different databaseinstance at runtime.

C. Plug-in Framework

MAAP allows analysts to create sets of analytics as namedworkflows for execution at any time. The plug-in interfacegrants plug-ins access to the underlying model database.In general, each plug-in is associated with a single MIRAactivity, supporting some or all of the steps defined for thatactivity. The MAAP presentation component provides plug-in management that allows the user to register and deregisterplug-ins dynamically. Users can specify an ordered set of plug-ins suited to their task, save them as reusable name workflows,and execute the workflow on demand. To preserve flexibilityand independence from the core MAAP framework, plug-insare detected and loaded dynamically at run time.

A number of MAAP plug-ins have been written to date.To illustrate the flexibility and utility of the plug-in architec-ture, two such plug-ins are described below, the Attack PathAnalysis Plug-in (APA) and the Cyber Investment AnalysisMethodology (CIAM) Plug-in.

1) APA Plug-in: The APA plug-in leverages the systemarchitecture defined in the Derive Models activity and conductsa series of attack path analyses in support of the PerformScoring and Estimate Risk MIRA activities. The plug-in outputis a set of system components and data element pairs that arevulnerable to attack (i.e., hot spots), along with the specificattack path and conditions under which such an attack couldoccur.

The simplest form of APA conducts a basic analysis,enumerating all possible access paths from the set of viablestarting attack nodes to all possible target nodes processingmission data, individually computing the LOE values for allpossible attacks utilizing those paths and determining, foreach attack source/target pair, the minimum attack LOE. Thisminimum LOE value is obtained by comparing (1) the LOEvalue associated with initiating the attack, utilizing a specificattack vector on the source attack node, (2) the maximum fromthe set of LOE values representing the traversal difficulty foreach intermediary node along the attack path (i.e., how hardit is to pass through a node during an attack), and (3) theLOE to carry out a specific compromise on the target node.The maximum of these three difficulty measures representsthe most work an attacker must be capable of to carry out asuccessful attack. For a pair of attack source and target nodes,the minimum attack path LOE value represents the easiestattack for an adversary, given a specific attack vector andcompromise to be carried out on a specific data element.

552

Page 5: [IEEE 2012 IEEE International Conference on Technologies for Homeland Security (HST) - Waltham, MA, USA (2012.11.13-2012.11.15)] 2012 IEEE Conference on Technologies for Homeland Security

The power of the APA plug-in lies in the capability to swapout path processing logic. While the full path enumerationprovides a system-wide view of all possible avenues of attack,a specific analysis may not need that level of detail. Forexample, the set of paths could be reduced to the set of shortestpaths using a variant on a shortest-paths algorithm with thescoring function adjusted to account for the differing LOEscores on individual nodes and links. This modification in thepath processing logic would yield only those paths associatedwith the worst-case attacks (i.e., the attacks that are easiest),making a tradeoff between shorter execution times and theadditional data processing required by a full path enumeration,which could shave hours to days of time from the analysis.

Additional modifications could be made beyond simplefiltering on the shortest paths; an APA variation could purse themiddle ground on the path processing issue, focusing on theset of paths supporting attacks above a specific LOE threshold,for example. As a MAAP plug-in, APA provides flexibility toMIRA, supporting the evolving needs of the analyst.

2) CIAM Plug-in: The CIAM plug-in is useful during theDerive Mitigations activity of MAAP. The plug-in analyzes acollection of datasets pertaining to the history of cyber-securityattacks on a target cyber infrastructre, vulnerabilities exploitedin those attack steps, the estimated generic business impactof those attacks (derived from the Common VulnerabilityScoring System [8] score for the associated vulnerabilities),and the applicability and cost of a given set of securitycontrols to the attack steps derived from each attack. Using thisinformation, CIAM computes a prioritized ranking of securitycontrols which analysts can consider when making detailedcybersecurity investment decisions. As output, the CIAM plug-in provides the user with a simple interface to view the inputdata in a tabular format and the generated results in variousgraphs (Figure 4).

Fig. 4. CIAM Plug-in

The core CIAM software is manifested as an applicationthat can be invoked independently of MAAP, highlighting theflexibility of the plug-in framework. Plug-ins do not have to bedirectly tied into the model database provided by the MAAPframework. In fact, the CIAM plug-in employs data in flatfiles outside and independent of the MAAP database.

D. Interactive Visualizations

In addition to the core model database and the associatedplug-in architecture, MAAP provides several default visualiza-tions that provide customizable views of model and analysisdata. These visualizations aid heavily in a MIRA analysis,eliminating much of the need for analysts to pour over datain spreadsheet form. Two visualizations, mission model viewand heat map view, are discussed below. The figures are notscaled for detailed viewing but are intended to provide quickimpressions.

Fig. 5. Mission Map Visualization

The first visualization (Figure 5) is a rendition of the missionmodel, called a mission map. The mission map presents thelinks between data, mission functions, MOEs, and missionsin a hierarchical nodes-and-links display. Users can interac-tively expand individaul mission components to see underlyingdetails of each mission element, while also highlighting therelationships between different pieces of the model.

Fig. 6. Inverted Risk Matrix Variant

The second visualization (Figure 6) presents the output ofthe Estimate Risk activity as an interactive heat map thatdisplays the hot spots produced by the path analyses conductedduring the Perform Scoring activity. Figure 6 is the MAAPversion of the general heat map shown in Figure 2. Systemvulnerabilities can be filtered based on various attributes (e.g.,

553

Page 6: [IEEE 2012 IEEE International Conference on Technologies for Homeland Security (HST) - Waltham, MA, USA (2012.11.13-2012.11.15)] 2012 IEEE Conference on Technologies for Homeland Security

nodes affected, mission impacted). Each point in the matrixrepresents a set of hot spots that share the same attack LOEand data criticality score; selecting an individual point bringsup a separate display, presenting a tabular breakdown of allindividual vulnerabilities represented by a single point in thematrix.

V. APPLYING MAAP

As an experimental cyber-security analysis tool designed tofacilitate MIRA studies, MAAP needs exposure to analysts andsubject matter experts (“users” below) who have experienceapplying MIRA via mostly manual methods. Their feedbackprovides valuable input to future development plans.

Initial feedback from the first round of analyst testing hasgenerally been positive. However, several issues have beenidentified and are discussed below.

One area of feedback lies in the lack of an extensivedata management capability built into MAAP (e.g., data inputscreens). While certain analytics can populate the databasewith initial model data, a user interface to view and editthe data directly in the database is lacking. In the meantime,generic data management tools for databases exist and canbe leveraged to fill the need outside of MAAP. Additionally,the plug-in framework is an ideal place to provide customizedinterfaces for data management.

Another MAAP usability concern relates to the small num-ber of plug-in analytics thus far, consequently limiting theapplicability of MAAP. Plug-ins to date are focused on certainvisualizations, data ingest, mitigation prioritization (CIAM),and attack path processing. Without a broader set of analytics,MAAP is limited in its applicability across a wide variety oftarget system types. There are plans to expand the plug-insavailable.

Additional usability feedback pertains to MAAP visual-izations. Some analysts suggested alternative methods fordisplaying the analysis data, providing additional ways ofhighlighting features that relate to important security details.One suggestion under consideration involves integrating fea-tures of the existing model and heat map views.

A final usability item raised concerns with the granularityat which LOE scores are derived. The underlying databaseschema is currently designed to accommodate a highly granu-lar mode of recording scores. More coarse grained approachescan be useful in certain engagements, but the path analysisplug-in and the database schema require changes to accomo-date this flexibility.

In addition to the above criticisms, several provided featureswere well received. The accessibility of the database and itsuse beyond MAAP free the analyst to explore alternativeautomation methods. The interactive visualizations alleviatethe overhead previously required to document analysis results.Finally, the analytic workflow construct provides analysts witha modular and flexible testbed to test their hypotheses andmanage multiple analyses.

VI. FUTURE WORK

Ideas for future work include: enhancing database supportand the underlying database schema, investigating and adapt-ing new plug-in analytics, incorporating validation techniquesfor both MIRA and MAAP, refactoring the current architec-ture, and incorporating further improvements from MIRA asthe methodology continues to evolve.

Continued development of plug-in analytics for the MAAPplug-in framework is critical to the future success of thetool. Potential plug-in analytics include: automated systemarchitecture detection from live networks using preexistingtools, mission information extraction from documentation, fullor partial automation of intensive computational activities inMIRA, and modeling and simulation integration to facilitateempirical evaluations of cyber-attack impact. Current work isalready addressing some of these areas.

The MAAP team is interested in experiments to evaluatemetrics related to MAAP and MIRA. Two examples includethe evaluation of metrics related to the fidelity of analytics andthe estimation of time saved using particular analytic plug-ins.

While many improvements can be made to MAAP, it mustalso reflect future enhancements made to MIRA. For exam-ple, the AM is simplistic in its current instantiation; addingrobustness to this model, by including a detailed breakdownof attack capability for example, improves the fidelity ofthe analyses leveraging MIRA and MAAP. Current work isalready underway to further refine the methodology.

VII. CONCLUSION

As an extensible framework for conducting MIRA studies,MAAP can directly inform mission-cyber risk decisions andmitigation selection and prioritization. With the continueddevelopment and incorporation of additional plug-ins intoreusable analytic workflows, the applicability and utility ofMAAP will grow. The end goal of this work is to provideMAAP as an open framework to the cyber security community.As developments continue to be made in the area of missionassurance, MAAP and other frameworks like it will be essen-tial components to the cyber analyst tool set.

ACKNOWLEDGMENT

The authors wish to acknowledge the contributions ofWende Peters, Shaun Hutton, Charlie Frick, Paul Boudra, JulieTarr, and Brad Siegler. Their work on MBA and MIRA andon suggesting improvements to MAAP have been invaluable.

REFERENCES

[1] D. L. Buckshaw, G. S. Parnell, W. L. Unkenholz, D. L. Parks,J. M. Wallner, and O. S. Saydjari, “Mission Oriented Risk andDesign Analysis of Critical Information Systems,” Military OperationsResearch, vol. 10, no. 2, pp. 19–38, Mar. 2005. [Online]. Avail-able: http://openurl.ingenta.com/content/xref?genre=article&issn=1082-5983&volume=10&issue=2&spage=19

[2] C. M. Burris, J. G. McEver, H. W. Schoenborn, and D. T.Signori, “Steps Toward Improved Analysis for Network MissionAssurance,” in 2010 IEEE Second International Conference on SocialComputing. IEEE, Aug. 2010, pp. 1177–1182. [Online]. Available:http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5591922

554

Page 7: [IEEE 2012 IEEE International Conference on Technologies for Homeland Security (HST) - Waltham, MA, USA (2012.11.13-2012.11.15)] 2012 IEEE Conference on Technologies for Homeland Security

[3] R. S. Mudge Capt USAF and S. Lingley, “Cyber and Air Joint Ef-fects Demonstration (CAAJED),” AFRL-RI-RS-TM-2008-12, Tech. Rep.March, 2008.

[4] S. Musman, A. Temin, M. Tanner, D. Fox, and B. Pridemore, “Evaluatingthe Impact of Cyber Attacks on Missions,” in Proceedings of the 5thInternational Conference of Information Warfare and Security, 2010, pp.446–456.

[5] T. Llanso, “CIAM: A data-driven approach for selecting and prioritizingsecurity controls,” in 2012 IEEE International Systems ConferenceSysCon 2012. IEEE, Mar. 2012, pp. 1–8. [Online]. Available:http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6189500

[6] “NIST SP 800-53 Rev. 3, Recommended Security Controls for FederalInformation Systems and Organizations,” National Institute of Standardsand Technology, Tech. Rep. August, 2009.

[7] Oracle Corporation, “MySQL: The world’s most popular open sourcedatabase,” 2012. [Online]. Available: http://www.mysql.com/

[8] FIRST, “Common Vulnerability Scoring System (CVSS-SIG),” 2012.[Online]. Available: http://www.first.org/cvss

555