7th Annual Secure and Resilient Cyber Architectures ...acquisition. In addition, vendor booths and representatives displayed leading-edge cyber resiliency offerings. Recent Year: 2017

1 ©2018 The MITRE Corporation, All rights reserved. Approved for Public Release; Distribution Unlimited. Case

Number 17-3229-17

Annual Secure and Resilient Cyber Architectures Invitational

& Training Event

Overview May 2017 marked the seventh year in which approximately 120 subject matter experts (SMEs) in cyber resiliency from government, industry, and academia came together in McLean, VA, for collective work on topics of common policy and engineering concern. For two days, the 7th Annual Secure and Resilient Cyber Architectures Invitational & Training Event accelerated recognition and adoption of cyber resiliency with a focus on organizations.

Background

Prior Years: 2010 - 2016 The first workshop, held in October 2010, established the initial community and shared architectural, technical, and policy perspectives on cyber resiliency. The second workshop, held in May 2012, focused on collaborating to develop a communal view of resiliency frameworks, engineering principles, and metrics [1]. The third workshop, held in June 2013, centered on identifying favorable conditions for use of specific resiliency techniques, assessing the use of techniques in enterprise architectures, and developing use cases [2]. The fourth meeting, now renamed “Invitational” and held in May 2014, emphasized applying cyber resiliency to space-based systems and critical infrastructure, designing a cyber resiliency challenge, and identifying roles played by cyber resiliency throughout the systems engineering life cycle [3]. The Fifth Annual Secure and Resilient Cyber Architectures Invitational, held in May 2015, concentrated on taking stock of the state of cyber resiliency: the lessons learned and the remaining challenges to overcome. It sought community consensus on the theme of Cyber Resilience: Looking Backward (What Has Worked? What Has Not?), Looking Forward (What New Challenges Must Be Faced?). Keynote speakers included representatives from the National Institute of Standards and Technology (NIST), US Navy, Indiana University, and Bit9 + Carbon Black [4]. The Sixth Annual Secure and Resilient Cyber Architectures Invitational, which took place on 18–19 May 2016, centered on the theme of Institutionalizing Cyber Resiliency [5]. Four keynote speakers were followed by panel discussions inclusive of industry leaders. Three working groups furthered knowledge sharing by focusing on: cyber resiliency and system security engineering cyber resiliency and an organization’s cybersecurity program, and cyber resiliency and

7th


Number 17-3229-17

acquisition. In addition, vendor booths and representatives displayed leading-edge cyber resiliency offerings.

Recent Year: 2017 The rest of this report focuses on the 7th Annual Secure and Resilient Cyber Architectures Invitational and Training Event (the last part added to acknowledge the tutorials held the day before the main event). These proceedings present a summary of the keynote talks, the panel discussion, and working group tracks. The Cyber Resiliency Invitational Committee believes the Invitational serves a larger mission: to advance the field of cyber resiliency for our sponsors and nation. Additional materials from the invitational and briefings can be found at http://www.mitre.org/cyberworkshop. The committee welcomes comments from readers through the contact email address: [email protected].

The Cyber Resiliency Invitational Committee April 2018

http://www.mitre.org/cyberworkshop.

mailto:[email protected]


Number 17-3229-17

Table of Contents

Overview ......................................................................................................................................... 1

Background ..................................................................................................................................... 1

Prior Years: 2010 - 2016 ............................................................................................................ 1 Recent Year: 2017 ...................................................................................................................... 2

Introduction ..................................................................................................................................... 4

Keynote Presentations and Panel .................................................................................................... 6

Mark Maybury, Director of Cyber Security Implementation, The MITRE Corporation ........... 6 How We’ll Solve Cybersecurity in the Next Five Years – Sounil Yu Senior Vice President, Director of Security Innovation................................................................................................... 6

Quantifying and Measuring Cyber Resiliency – Dr. George Cybenko, Dorothy and Walter Gramm Professor of Engineering, Dartmouth College ............................................................... 9

Engineering Cyber Resilient Weapon Systems – Ms. Kristen J. Baldwin, Acting Deputy Assistant Secretary of Defense for Systems Engineering (DASD(SE)), Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics (OUSD(AT&L)) ............... 10

Cyber Resiliency in Financial Community Panel ..................................................................... 12

Working Groups............................................................................................................................ 13

Track 1: Cyber Resiliency and Architecture ............................................................................ 13 Track 2: Measuring the Effectiveness of Cyber Resilience ..................................................... 15

Track 3: Different Types of Resilience .................................................................................... 17

Track 4: Table Top Exercise .................................................................................................... 19

References ..................................................................................................................................... 22


Number 17-3229-17

Introduction The 7th Annual Secure and Resilient Cyber Architectures Invitational & Training Event included four presentations, a panel, four facilitated working groups, and selected vendors. Section 2 summarizes the four keynote addresses and one panel, as follows:

• Keynote Presentation and Introduction to Invitational, Dr. Mark Maybury, Director of Cyber Security Implementation, The MITRE Corporation

• “How We’ll Solve Cybersecurity in the Next Five Years,” Sounil Yu Senior Vice

President, Director of Security Innovation in financial services

• “Quantifying and Measuring Cyber Resiliency,” Dr. George Cybenko, Dorothy and Walter Gramm Professor of Engineering, Dartmouth College

• “Engineering Cyber Resilient Weapon Systems,”Ms. Kristen J. Baldwin, Acting Deputy

Assistant Secretary of Defense for Systems Engineering (DASD(SE)), Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics (OUSD(AT&L))

Financial Panel on Cyber Resiliency –Four senior-level panelists from major financial institutions comprised the group. At the request of the panelists and for security reasons, their names and affiliations are not recorded in this report. Section 3 summarizes the working groups:

• Cyber Resiliency and Architecture, led by Mitch Miller USAF and Mindy Rudell, MITRE Corporation

• Measuring the Effectiveness of Cyber Resiliency, led by Dayton Marchese, US Army Research and Development Center and Deb Bodeau, MITRE Corporation

• Cyber Resiliency and its Relationship to Other Types of Resiliency, led by Craig Jackson, Indiana University and Rich Graubart, MITRE Corporation

• Table Top Exercise: Defending a Resiliency-Enabled Enterprise, led by Nick Multari, Project Manager, Pacific Northwest National Lab (PNNL)

In parallel to the presentations and working groups, during the first day there were vendor booths presenting cyber resiliency enabling products from the following vendors: DigitalPersona, Inc., Elasticsearch, Inc., Intelligent Waves, Inc., Phantom Corp., Privoro, LLC Rambus-Cryptography Research, Inc., SteelCloud, LLC, Tresys Technology, LLC., Verodin, Inc.


Number 17-3229-17


Number 17-3229-17

Keynote Presentations and Panel Mark Maybury, Director of Cyber Security Implementation, The MITRE Corporation

Mark Maybury opened the first keynote by speaking about his recent trip to Japan, the Toji pagoda design, and its resilience to earthquakes – only two pagodas have ever been toppled by an earthquake and some date back to 607. Pagodas have redundant support pillars; each floor and the individual walls of each floor are isolated from each other and can move independently. The central pole is suspended so that it absorbs energy as the pagoda moves back and forth in response to earthquakes. The resilience techniques (redundancy, isolation and adaptive

response) are the reasons for the pagodas’ ability to withstand such “attacks’, i.e., earthquakes.

Dr. Maybury then pivoted to focus on cyber resiliency and the challenges in the cyber domain, noting that $445 billion dollars was lost in 2014 due to cyber-attacks. He associated this with the brittleness of cyber systems. He discussed how cyber resiliency should be a holistic approach to the challenges faced by complex systems engineering. He also discussed the importance of scale– both scaling up (in terms of the Internet of Things) and scaling down (in terms of applying cyber resiliency to smaller operations, like one-person businesses). Other examples of complexity included:

• Approval cycles such as FDA approval being on the order of years but the patching and update cycles being on the order of months

• Embedded cyber environments like police cars that functions as mobile jail cell, temporary storage locker for evidence and weapons

Dr. Maybury concluded his keynote by addressing what success looks like. His vision of success includes a model of cyber resiliency management, steps that align cyber resiliency with the mission environment, an acquisition process that incorporates cyber resiliency, and the ability to actively mitigate risks even in challenging environments faced by our sponsors.

How We’ll Solve Cybersecurity in the Next Five Years – Sounil Yu Senior Vice President, Director of Security Innovation Mr. Yu’s keynote presented the challenge of solving cyber security in the next five years. The premise to an overall solution, he suggested, is to get inside the OODA (Observe, Orient, Decide


Number 17-3229-17

and Act) loop. By doing so, cyber security professionals can hope to outpace an attacker who has the initial advantage. Mr. Yu reviewed the history of IT and security as follows:

• The 1980’s were associated with “identify. In general, the IT inventory was inexpensive and the industry was not focused on prioritizing the security of assets. Small solutions included system management tools and scanners. This period was characterized by little to no tension between IT and security because IT security was in its infancy.

• The 1990’s were associated with “protect. Because of an increased networked landscape, viruses and attacks against our systems became realities. Solutions included Antivirus, firewalls, secure configurations and application security. There was now some tension between IT and security because security was breaking application functionality. At this point, most security teams were small and ad hoc, mostly focused on vulnerability management.

• The 2000’s were associated with “detect.” Client-side attacks were becoming common and newsworthy. Solutions began to focus on audit log analyses and management with a focus on IDS and SIEMs. The tensions between IT departments and services and Security teams grew as the teams focused on security operations and threat management.

• Now, the 2010’s are associated with “respond.” Organizations work under the assumption they have been or will be breached Solutions have expanded and include incident response, hunting endpoint detection and response (EDR) and identify and access management (IdAM). There are increased tensions between IT and Security. By now, Security has become a dedicated business unit and risk management organization answering to the highest levels of an enterprise.

Mr. Yu noted that we are currently putting significant effort into prevention (e.g., hardening). But he pointed out that prevention is only effective at reducing the likelihood of a breach occurring. It does not address the ability of an organization to recover from a breach.

Therefore, Mr. Yu predicted that the next stage will be “recovery”. He posited that the challenge in the 2020’s will be our ability to recover in an environment where the harm caused by attacks may be irreversible. Examples of irreversible harm are:

• WikiLeaks – permanently destroyed confidentiality of the data • Ransomware – permanently destroys trust in the integrity of the data and systems • Permanent DOS, BR Wiper, bricking firmware – permanently destroys the availability of

the resource


Number 17-3229-17

Mr. Yu then shifted his presentation to address various architectural solutions that would help make systems more resilient. Moreover, he noted how many of the solutions are architectural approaches that were developed with the intent of providing some operational capability (e.g., greater speed) but actually provide security and resiliency. Examples he cited include:

• Copy on write, block chain – Declarative policy through Infrastructure as Code actually makes any unauthorized activity more visible and immediately suspect.

• Serverless architectures- These are intended to facilitate rapid rebuilds. While not originally designed to counter entrenched adversaries, they do provide the ability to create non-persistent environment, and effectively flush out an entrenched adversary.

• IaaS – while not developed with resiliency in mind, by default it provides greater visibility and logging of user and software actions, both of which will facilitate detection of an adversary.

Mr. Yu went on to note how traditional solutions for security commonly make the business OODA loop move slower than the attacker. As a result, the business tries to circumvent the security which may only aid the adversary. Mr. Yu argued for the use of technology and best practices which allow businesses to outpace the adversary and be more resilient, thus naturally shortening the OODA loop.

He went on to discuss how the 1980s, 1990s, 2000s, and 2010 each had their own security focus. In the 1980s the focus was Identify, in the 1990s it was Protect, in the 2000s it was Detect, and in the 2010s, it was Respond. He believes that the focus of the 2020s will be Recover. Mr. Yu concluded by noting:

• Known attack methods only get better with time when against static systems • The next era in IT and Security will have to deal with a growing number of irreversible

attacks that challenge and undermine our ability to RECOVER • Improvements in PROTECT, DETECT, and RESPOND capabilities may reduce

occurrences of malicious events but are insufficient against well-executed destructive scenarios; the bad guys will still get into systems and organizations will still need ways to continue missions while under attack.

• Our best countermeasure is better design and proper use of existing technologies

The talk closed with a challenge from Mr. Yu to the audience: “Elimination of poor designs will happen – either by intentional decommissioning or by adversarial destruction. Which would you rather count on?”


Number 17-3229-17

Quantifying and Measuring Cyber Resiliency – Dr. George Cybenko, Dorothy and Walter Gramm Professor of Engineering, Dartmouth College

Dr. Cybenko, like Mr. Sounil Yu, noted that there were multiple eras in cybersecurity. His three eras of cyber security were: prevention (pre-1990), detection (1990-2010), and response and recovery (post 2010).

Dr. Cybenko employed the definition of resilience from PPD-21: the ability to prepare for and adapt to changing conditions and withstand and recover rapidly from disruptions. Resilience includes the ability to withstand and recover from deliberate attacks, accidents, or naturally occurring threats or incidents.

As a hypothesis, Dr. Cybenko posited that cyber resiliency is more important to measure than security. A primary reason for his view was that given the sophistication and number of cyber-attacks adversaries will compromise a system, so it is essential that organizations are able to operate despite the persistent adversary presence.

The speaker offered a cyber resilience review self-assessment guide, from US-CERT which listed 7 actions to take:

1. Services are identified and prioritized. 2. Assets are inventoried, and the authority and responsibility for these assets is established. 3. The relationship, between assets and the services they support, is established. 4. The asset inventory is managed. 5. Access to assets is managed. 6. Information assets are categorized and managed to ensure the sustainment and protection

of the critical service. 7. Facility assets supporting the critical service are prioritized and managed.

Dr. Cybenko moved on to the heart of his presentation which was quantitative cyber resiliency. His premise was that reliability engineering is an established field whose approaches can produce quantifiable measurements of cyber resiliency. Underlying his proposition is the need and ability to measure performance. There are two values of performance: 1) objective values, such as, desired operational goals, - and threshold values, where performance is not effective operationally or suitable)

Dr. Cybenko further noted that there are three security performance value areas – confidentiality, availability, and integrity. Mission performance could be measured with regards to loss of any of the three. Of the three, availability is the easiest to measure. Integrity measurements are more difficult and confidentiality measurements are the most difficult to quantify and obtain.


Number 17-3229-17

Engineering Cyber Resilient Weapon Systems – Ms. Kristen J. Baldwin, Acting Deputy Assistant Secretary of Defense for Systems Engineering (DASD(SE)), Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics (OUSD(AT&L))

Ms. Baldwin discussed the importance of cyber resilience in the defense acquisition process. She noted that from an acquisition point of view, we need to embed cyber resilience into everyday processes and into the designs of systems.

One of the major challenges is trying to understand and address the threat throughout the acquisition lifecycle and from the various perspectives of all stakeholders, e.g., government, prime subtractors, vendors, and 3rd party tests of certifications.

Ms. Baldwin noted how the threat surface in acquiring programs and systems information is broadened by the unique ways we are using open technology in designs, allowing an adversary to disrupt/degrade system performance, and obtain or alter a US capability. She also noted that vulnerabilities are found in all levels – programs, organizations, personnel, hardware, and software. She also noted that systems’ vulnerabilities may arise from inherent weaknesses in their design or processes) or from weaknesses intentionally inserted. Both must be addressed.

If the adversary is successful in compromising systems, the consequences would be severe and include:

• Loss of technological advantage • System impact (corruption and disruption) • Mission impact

Ms. Baldwin discussed what is being done to improve cyber resilience within the acquisition process. She discussed the Program Protection and Cybersecurity Instruction and associated guidelines and instructions. She noted how collectively these documents address the threat from technology, components, and information perspectives. An example of a technology protection is anti-tamper. An example of component protection is software assurance. An example of information protection is appropriate use of classification.

Ms. Baldwin pointed out how cyber security must be embedded in or baked-in all the government’s protection activities as part of our infrastructure. She also noted that each of the military services is moving forward with their own major initiatives to address cyber threats to acquisition. The three services need to develop and mature each of their approaches, while they also collaborate and move toward common approaches.


Number 17-3229-17

Regarding the weapon systems acquisition process, Ms. Baldwin offered some observations on its status and challenges of cybersecurity. These include:

• Weapon acquisition programs are seeking clear and specific cyber resilience guidance, and such guidance does not yet exist.

• Applying the Risk Management Framework (RMF) to weapon systems is a challenge; the existing baselines are intended for IT systems solely.

o The RMF is being implemented by network and IT professionals, but there is a lack of training at the Acquisition program level. IT professions do not necessarily have weapons domain experience. As a result, there is no understanding of which portions of NIST SP 800-53 or the RMF need to be applied, nor is there an understanding of which portions need to be tailored to the correct level for audit regimes.

o The RMF is often being applied as a compliance vehicle, which was never the intended purpose of the RMF.

o Cyber security controls assume enterprise and organizational control mechanisms are already in place; this assumption is much more problematic in a non-enterprise environment. In addition, some controls overlap with engineering functions, thus causing duplication and possible confusion.

• Inconsistencies: Services’ and Agencies’ PEO/programs and industry partners are each working to determine cyber resiliency solutions – there appears to be no common implementation of rules or principles. As a result, some solutions are beginning to diverge.

• Baked in vs. Added on: From the test community, findings in legacy systems indicate that cyber security must be designed in. Other types of efforts may be inefficient if not incorporated from the beginning.

• Ms. Baldwin also discussed some of the actions DoD is taking to improve the situation

o She noted that DoD has issued a new enclosure in DoDI 5000.02 that describes cyber security as a requirement for all DoD programs. The scope of responsibility is more comprehensive by including personnel, networks, systems. supporting systems. and program information.

o DoD has set up capabilities to support the assessment of software and hardware, e.g., the Joint Federation Assurance Center.

• Ms. Baldwin noted that there are remaining challenges: o Work still needs to be done to address the life cycle cyber security needs of

embedded throughout acquisition lifecycle.; o There is a need to bring policies, tools, and expertise to enable cyber resiliency in

DoD weapon systems. There is a need to translate IT and network resiliency to weapon system resiliency.

o We still need to establish security as a fundamental discipline of systems engineering.


Number 17-3229-17

• Ms. Baldwin noted that there are opportunities for government, industry and academia to engage. These include:

o How can we leverage our software assurance and trusted microelectronics capabilities to ensure cyber resiliency?

o How can system security engineering activities be integrated with network and enterprise-level cybersecurity resiliency?

o How can modular open systems approaches create resiliency for our systems and networks?

o How can system cybersecurity risk inform the operational commander’s risk assessment, factoring in all elements of the mission?

Cyber Resiliency in Financial Community Panel The panel on cyber resiliency in the financial community consisted of four individuals from three major financial institutions. This included a VP of Cyber Resilience, a Cyber Resilience Leader, the Senior Director of Cybersecurity and Resilience Strategy, and a Senior VP and Director of Cybersecurity Innovation. Due to the sensitive nature of the conversations we are not permitted to provide any summary of the responses or identify the individuals or their institutions.


Number 17-3229-17

Working Groups

Track 1: Cyber Resiliency and Architecture

The track leads were Mitch Miller from the US Air Force and Mindy Rudell from the MITRE Corporation. Attendees at the track included individuals from: the Department of Defense, a representative of the financial community, Comcast, IBM, NSA, MITRE, the Institute of Defense Analysis, JHU-APL, DARPA, and Tresys Technology.

The goal of the track was to develop preliminary guidance and a path forward on how to incorporate cyber resiliency into architectures. In support of this goal the group considered:

• Nature and needs of different architectures, e.g., IT systems, cyber physical/embedded systems

• Cyber Resiliency design principles; and • Strength and limitations of different cyber resiliency techniques

Challenges The group discussed some of the challenges in incorporating cyber resiliency into architectures.

• There is no single architecture. Architectures vary in terms of real time considerations, connectivity, human interaction and other factors. As such there is no single cyber resiliency solution for architectures.

• Resiliency is still a new field. Due to the newness of the field there is not a consensus as to the meaning of the term or how to apply cyber resiliency. The relative newness of the field also meant that there is very limited experience in applying cyber resiliency to architectures; hence there is limited example to draw upon.

• Flexibility: The group recognized that given the evolving nature of cyber threats, that cyber resiliency in architectures must provide adaptability and flexibility (the latter reinforcing the already noted point that one size does not fit all).

• Focus. The group agreed that the focus for cyber resiliency architectures should be IT systems and embedded. After further discussion the group decided the focus should first be on the IT systems.

• Limitations of current work. The group examined the MITRE-provided cyber resiliency design principles and cyber resiliency techniques. The general feeling was that while they did provide utility they lacked an architectural orientation. The group also recognized that the work to date on cyber resiliency was focused on IT systems. Applying the cyber resiliency constructs to embedded and cyber physical systems will take time.


Number 17-3229-17

Discussions Mr. Miller explained to the group the weapon systems concerns. This included

• Weapon systems are diverse in nature, with customized interfaces. They are not enterprise IT systems, not even with tailoring.

• The missions (and associated mission threat analysis) should always drive investment decisions.

• Cyber resiliency in weapon systems is not something that can be added on at the last minute. The concept must be baked-in at the inception of the weapon program design.

• Achieving a truly resilient weapon system requires a well-trained cyber workforce • The need for agility and adaptability noted earlier for achieving resiliency applies to

weapon systems as well. These characteristics are essential if the weapon system is to achieve its eventual mission goal.

• It is important to develop a common security environment so that one organization can leverage work already done.

• It is essential to assess and protect fielded systems. What does an organization do when it discovers fielded systems that are vulnerable? It is not realistic to simply decommission such systems. Therefore, it is essential that new systems are designed with agility and adaptability in mind, thus allowing for changes in response to threats. once systems are deployed.

• Intelligence support is essential for ensuring that weapon systems may achieve their missions.

• Bottom line: the key to a successful weapon system is that it can support and achieve the mission need even in a cyber contested environment.

The group felt that the cyber resiliency work to date was lacking some architectural components such as a cyber resiliency oriented reference architecture or architectural patterns. But they did find that the design principles and techniques could be used to help guide discussions regarding the difficulty of applying technologies to systems to achieve cyber resiliency, and efficacy of such technologies. Toward that end the group performed an exercise of taking each of the 14 cyber resiliency techniques (the resiliency techniques were chosen as opposed to the design principles as they were more mature and more “accepted” by the community) and then attempted to identify how they could be implemented in an IT system. IT, as opposed to embedded, was chosen because it was easier to do; where appropriate, examples supporting embedded systems were included. For each technique the group identified possible implementations (technologies and processes) that could implement cyber resiliency, and categorize them as low, moderate or high. The categorization was intended to represent both degree of difficulty in implementation and the efficacy of the proposed solutions.

Outcomes 1. The group formulated five recommendations: There is a need to create products (e.g.,

CSfC) with diversity. This his may mean employing multiple vendor products from trusted sources.


Number 17-3229-17

2. Systems engineering, in particular the Systems Design Life Cycle (SDLC), needs to include cyber resiliency.

3. IT mandates, from internal enterprises to government-driven, need to be influenced by cyber resiliency needs for all types of systems.

4. The lifecycle of operations should be included when considering design principles. 5. While techniques make for a good checklist for systems architects, they are often

insufficient for implementing actual architecture.

Track 2: Measuring the Effectiveness of Cyber Resilience

The ability to have a means of assessing and measuring the effectiveness of cyber resiliency solutions, especially at the mission level, is of growing importance. Developing such assessments and measurements remain very challenging. This track examined the challenges in measuring the effectiveness of cyber resiliency as well as identifying various efforts to develop effective measurements. The track leads were Dayton Marchese, U.S. Army Engineer Research and Development Center, and Deb Bodeau, The MITRE Corporation.

General Principles The group identified general principles for practitioners, those tasked with defining, evaluating, and reporting metrics related to cyber resiliency.

1. Start by considering the worst case. A frequently-mentioned idealized metric is the probability of reaching a recoverable state, under stated threat conditions. If a recoverable state cannot be reached, this constitutes a “cut and run” scenario. Because exhaustive evaluation is frequently infeasible, practitioners should look at the worst case first: what is the probability of getting to the “cut and run” decision?

2. Share lessons-learned. Practitioners should seek to share case studies of cyber resilience, identifying what solutions improved cyber resilience, what metrics were used to evaluate improvement, and what cost-benefit or return-on-investment (ROI) metrics were used. Case studies can aid in promulgating metrics and in validating possible solutions.

3. Take organizational priorities into consideration. Organizational strategies for the use of technology guide what will be included in a cyber resiliency solution. Organizational priorities can be expressed in terms of information security objectives: confidentiality, integrity, and availability. Metrics for availability should tie back to organizational, business, or mission objectives. Availability is not a priori more important than confidentiality or integrity; it is simply easier to measure.


Number 17-3229-17

4. Tailor metrics to the audience who will use them. Start by posing questions to the intended audience about how the metrics will be used, in relationship to what types of decisions members will make based on them. For mission owners and risk managers, key questions include: how much do we focus on the current mission or set of business functions, as contrasted with future missions or business opportunities, how do we reflect future impacts? For executives, questions include: what is your appetite for risk vs. resilience, what aspects of resilience are you hungriest for? For both executives and engineers, questions include: what are the measures for business success or measures of effectiveness? How do we link technical or architectural measures to these measures?

5. Make assumptions explicit: A variety of assumptions underpin the definition of a metric and the evaluation of the metric can be sensitive to these assumptions. Examples include: the scope (system, organization, sector, ecosystem) and assumptions about risk governance; the threat model, whether for a specific mission or generalized across several; the end user for the metric; the relationship between what is actually measured and what the end user wants to know (underlying model); how completely the aspects of the environment relevant to the metric evaluation have been represented; and the assumed values of parameters describing the operational environment.

Challenges The group identified several challenges related to defining, evaluating, and using metrics for cyber resilience.

• Threat modeling. Metrics are evaluated under assumptions about the nature of the threat against which a system, mission or business function, or organization must be cyber resilient. A major challenge is how to avoid simply measuring against yesterday’s threats. One approach is to use a layered architectural view as the starting point for analysis, assuming a compromise at one layer, and evaluating under that assumption.

• Cost and benefit metrics. The true cost of a cyber resiliency solution includes not only its total lifecycle cost, but also its impacts on usability, performance, and architectural alternatives (e.g., constraints on future product insertion). The benefits of a cyber resiliency solution can include increased functional capability and increased understanding of adversaries. However, the definitions of cost and benefit metrics are often vague or grounded in assumptions that change over the course of a system’s lifecycle. The group noted that, “What’s the ROI,” may be the wrong question; the focus needs to be on the mission or business benefits and the avoidance of unacceptable losses.

• Gaming. The confidence end users of a metric can have in the value (or the trend in values) of the metric depends on whether the metric value can be represented in a gaming environment, either by an adversary or by individuals in the organization who want to make their systems look good (or in need of extra resources). Sensitivity analysis can help address this concern.


Number 17-3229-17

Metrics and Cyber Resiliency Goals The group discussed characteristics of metrics related to the four cyber resiliency goals of: Anticipate, Withstand, Recover, and Evolve (or Plan, Absorb, Recover, and Adapt). For each goal, metrics capture one or more dimensions (time, performance, confidence), and relate to difference concepts as identified in the table below.

Goal Dimensions Discussion Anticipate Time

Performance of cyber defenders or cyber defense mechanisms Confidence (accuracy of prediction)

What forms of preparation (e.g., threat information sharing, exercises) help to Anticipate effectively? Time to Learn: How much learning is needed to Anticipate effectively? Costs associated with penetration testing, threat modeling, risk assessment Trade-offs between false negatives and false positives

Withstand Time Performance with respect to mission / business requirements Confidence (in the metric value, in the system)

Borrow from risk assessment – threat and vulnerability play a large role Repurpose resilience / performance metrics, with analogies to physical system resilience “Just limp over the goal line” Relates to risk appetite / risk tolerance

Recover Same as Withstand System focus rather than threat focus Repurpose resilience / performance metrics, with analogies to physical system resilience Relates to risk appetite / risk tolerance

Evolve Time Performance / meeting of future mission requirements Confidence

Key question: Given what we’ve just recovered from, if it happens again, how have we improved? Can we Withstand better? Can we Recover more quickly? How much better can we resist future attacks? How have we limited exposure?

Track 3: Different Types of Resilience

The track was chaired by Craig Jackson of Indiana University with support from Rich Graubart of The MITRE Corp. The track had a small number of individuals (approximately 12). They included Mr. Jackson and individuals from the intelligence community, the financial community, the regulatory community and MITRE. The small number of


Number 17-3229-17

individuals lent itself to strong and intense discussions. The following information is provided from the first day’s discussion.

Background Underlying the track inception was the understanding that there are multiple different types of resiliency. While they share some broad concepts, they tend to have different concerns and foci. Often, they have different threat and environmental assumptions. The overarching goal of the track was to provide a provide a better understanding of differences (assumptions, threats, operational environment) for different types of resilience, and how cyber resiliency could be better informed from these other forms of resilience.

Goals With this background in mind, the track had three objectives:

To identify the conceptual and real-world linkages, commonalities, differences, and tensions among the various types of resilience/resiliency.

To enhance cyber resiliency and other forms of resilience via this activity.

To learn from each other, expand our thinking, and have a coherent report-out.

It was stated explicitly, that there was no desire to try to define resilience or resiliency as part of the activity.

Path Forward Mr. Jackson proposed that resilience is an operative concept in other domains and environments and those types of resilience have interdependencies. All of the interdependencies must be considered when working towards cyber resiliency.

Mr. Jackson offered two paths that the conversation could take:

• Path 1: Comparative Resilience – focus on successful resilience in human-created, adversarial, defensive settings

• Path 2: Cyber Resilience in the Real World – psychological and organizational

In examining the two paths, it was agreed that while Path 1 is fascinating and can inform Path 2, Path 2 is where the action is.

The main assumption of Path 2 is mission assurance along the survive-thrive continuum. There are multiple stakeholders and actors. The group discussed some of the challenges. While cyber is being used in many parts of an organization’s resiliency effort, it does not

automatically mean cyber resiliency is present or achieved. The environmental and threat needs of one type of resilience may direct organizations to

mitigation approaches rather than to other forms of resilience.


Number 17-3229-17

As an example of the latter point, it was noted that in cyber resiliency there is a benefit to heterogenous software to prevent the same malware from propagating laterally and compromising homogeneous copies of the software. But in regional resilience, say after a devastating event like Hurricane Sandy in NYC, there is a need to facilitate the ability of workers from distant locations to come to stricken locations and aid in quickly restoring services; that ability is predicated on assumption that the services and associated software are homogeneous to stricken locations. How does one accomplish this with heterogenous software and skill sets? There was some subsequent interesting discussion regarding the challenges of achieving resilience, especially against sophisticated adversaries. Mr. Jackson proposed the following six questions to facilitate discussion for the next day.

1. In real world environments, what are the most important interactions between cyber and other types of resilience?

2. What processes/actions can support making the interdependencies explicit? 3. Whose job is it to align actor and stakeholder interests to achieve coordinated action?

(Top-down vs. Bottom-Up?) 4. Do we have success stories to tell-- and can we tell them in an unclassified way)? 5. Likewise, do we have failure stories to tell? 6. When does cyber help? When does cyber get in the way?

Unfortunately, several of the group members were unable to return the second day due to other commitments. As a result, the track had an insufficient number of participants to continue, and so was not held on the second day. Therefore, there was no opportunity to follow up on the points raised in the first day or discuss the questions Mr. Jackson posed.

Track 4: Table Top Exercise Introduction The track was chaired by Nick Multari, Project Manager, Pacific Northwest National Lab (PNNL). The purpose was to use a tabletop exercise as a brainstorming tool for cyber-security decision making with an eye toward defense. The goals were:

• Provide an engaging experience.

• Provoke discussion between participants through collaborative decision-making.

• Tap into players’ shared knowledge and creativity.


Number 17-3229-17

Background The table top game evolved from the Asymmetric Resilient Cybersecurity Initiative at PNNL. It was originally developed as a research tool. The tool would allow expert players work through a scenario that includes non-traditional cyber security technologies/techniques. It would allow them to observe their decision-making processes. The tool subsequently morphed into a training tool, which is how it was employed at the Invitational. As a training tool it allowed novice players to work through attack scenarios. It provided insight to participants on the general shape/structure of cyber-attack processes and layers of defense.

Organization Track attendees were divided into five teams (tables) of about six participants each. Each table was balanced with a varied level of experienced people, novice to experts. There was a laptop at each table with the “facilitator/game master” serving as a red team, moving laterally through a kill chain. The remaining participants at each table represented the IT (defenders) of an organization. Dice were used to reflect the probabilistic aspects of attacks and defense. When individuals rolled the dice, higher numbers were better. Both defenders and red team rolled dice. Dice values were seeds to the events (attacks) in game.

Flow and Elements The game was conducted in rounds. One round equaled one month. There was a maximum of 10 rounds.

There were 25 tokens per table. The tokens were used to “buy” three types of resources: infrastructure, policy, staff and investments (below). Events occurred by roll of the dice. The facilitator/game master on the laptop (red team) stated what was happening or what effect transpired. Players made use of response cards, resources, investments resiliency techniques.

There were twelve response cards, as follows:

1. Force password reset 7. Change outbound firewall rules 2. Change host-based firewall rules 8. Burn and rebuild suspected

infected systems 3. Implement signature 9. Patch a vulnerability 4. Conduct forensic analysis 10. Perform log analysis 5. Conduct internal penetration test/audit

11. Turn off network. Conduct deep rebuild of system

6. Perform malware analysis 12. Change inbound firewall rules

There were 25 investment cards, falling into five categories:

1. Prevent the bad 2. Detect the bad 3. Respond to the bad 4. Recover from the bad 5. Resiliency technique


Number 17-3229-17

There were seven resiliency techniques available:

1. IP hopping 2. System deception and hiding 3. System recycling 4. O/S cycling 5. Server application code cycling 6. Dynamic reconfiguration 7. User account risk minimization

Outcomes Games were played at each table. No table totally succeeded in preventing damage. Two tables had minimal damage and were still “alive” at end of game. Initially games were localized to each table. Once the teams were comfortable with the rules and constructs, the game moved to a collaboration approach across the tables.

Experience of constraints. Costs vs. benefits were reviewed. How much should a team invest in their IT infrastructure early on? How does an IT team prevent from running out of tokens (capital) to actually run systems and stay in the game (operational)? Which investments and resources were the costliest and why? What was an optimal sequence of investments, given the events that occurred? How benign or devastating were attacks and events, given a team’s infrastructure and responses? There was of course no single correct answer, results were dictated by the views of the players and the role of dice.

Experience of play. Consensus was that entire experience was very realistic (for those who have experienced real attacks on their systems) and illuminating for those who had not.

At the end of the game, a room report-out generated strong discussion for at least 30 minutes on what happened, what responses worked, what did not work well, what did not work at all. The discussion often referenced the fast pace of attacks on a kill chain, with surprise and ambush qualities. Participants at each table reflected that there was very little time to detect and diagnose, due to the varied rolls of the dice. Nearly every participant left stating they wanted to experience this type of table top cyber-attack training again in the near future.


Number 17-3229-17

References

[1] The MITRE Corporation (ed.), “Second Secure and Resilient Cyber Architectures

Workshop: Final Report,” 2012. [Online]. Available: https://www.mitre.org/cyberworkshop [2] The MITRE Corporation (ed.), “Third Annual Secure and Resilient Cyber

Architectures Workshop,” December 2013. [Online]. Available: https://www.mitre.org/publications/technical-papers/third-annual-secure-and-resilient-cyber-architectures-workshop

[3] The MITRE Corporation (ed.), “Fourth Annual Secure and Resilient Cyber Architectures Invitational,” 2015. [Online]. Available: https://www.mitre.org/cyberworkshop

[4] The MITRE Corporation (ed.), “Fifth Annual Secure and Resilient Cyber Architectures Invitational,” 2016. [Online]. Available: https://www.mitre.org/cyberworkshop

[5] The MITRE Corporation (ed.), “Sixth Annual Secure and Resilient Cyber Architectures Invitational,” 2017. [Online]. Available: https://www.mitre.org/cyberworkshop

https://www.mitre.org/cyberworkshop

https://www.mitre.org/publications/technical-papers/third-annual-secure-and-resilient-cyber-architectures-workshop

https://www.mitre.org/publications/technical-papers/third-annual-secure-and-resilient-cyber-architectures-workshop




Documents

7th Annual Secure and Resilient Cyber Architectures ...acquisition. In addition, vendor booths and representatives displayed leading-edge cyber resiliency offerings. Recent Year: 2017