View
22
Download
3
Category
Preview:
Citation preview
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Cybersecurity Test and EvaluationProcess
June 2018
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Agenda
• Cybersecurity T&E Introduction
• Cybersecurity T&E Policy
• Cybersecurity T&E Process
• Cybersecurity T&E in the TEMP
• Cyber Ranges
• Cybersecurity T&E Guidebook
2
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Introduction
• Many DoD systems have not proven to be cyber secure– Year after year DOT&E assessments have shown that systems remain vulnerable
• Security controls programs, such as the Risk Management Framework (RMF), are necessary but not sufficient
– These compliance measures do not adequately address threat tactics and capabilities
– These controls are frequently considered late in development
• Mission risk and operational resilience have not been properly addressed in controls based security
• There is a need for a more robust cybersecurity process– Establishing thorough cybersecurity requirements– Engineering cybersecurity into the system as opposed to adding it late– Thoroughly testing and evaluating systems and providing feedback to the
development engineers for action
• This brief describes the Cybersecurity T&E process
3
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Cybersecurity T&E Process
Cybersecurity T&E is necessary and required by policy– Evaluates a system’s mission performance in the presence of cybersecurity threats – Informs acquisition decision makers regarding cybersecurity, resilience and
survivability
4
CDD Validation
Dev RFP Release Decision
IOT&EOTRR
IATT
Full RateProduction
Decision ReviewATO
Lower Fidelity Mission-Based Cyber Risk Assessments Higher Fidelity
MDD
PHASES
TechnologyMaturation & Risk
Reduction
Engineering & Manufacturing
Development Production & Deployment
Materiel SolutionAnalysis
MS CMS BMS A
Phase 2 Characterize
the Cyber Attack
Surface
Phase 4 Adversarial
Cybersecurity DT&E
Phase 5 Cooperative
Vulnerability and Penetration
Assessment
Phase 3 Cooperative
Vulnerability Identification
Phase 6 Adversarial Assessment
Phase 1 Understand
Cybersecurity Requirements
DoDI 5000.02, Enclosure 14 – planning and conducting cyber T&E
Cyber T&E analysis and planning Cyber T&E
DRAFTCDD CDD
PDR CDRCPD
TRR
Operations & Support
OTRR IOT&E
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Cybersecurity T&E Policy
5
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Policy Overview
• DoDI 5000.02, Operation of the Defense Acquisition System, August 10, 2017, incorporating Change 3 – Enclosure 14
• DoDI 5000.75, Business Systems Requirements And Acquisition, February 2, 2017
• “Procedures for Operational Test and Evaluation of Cybersecurity in Acquisition Programs,” DOT&E Memo, April 3, 2018
• DoDI 8500.01, Cybersecurity, March 14, 2014
• DoDI 8510.01, Risk Management Framework (RMF), July 28, 2017, with Change 2
• JROCM 009-17, “System Survivability KPP Update to ensure Joint Force Mission Assurance”
– Cyber Survivability Endorsement Implementation Guide (CSEIG), v1.01a
6
DoDI 5000.02 DoDI 8500.01 DoDI 8510.01Procedures for Operational T&E
JROCM 009-17DoDI 5000.75
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
DoDI 5000.02, Enclosure 14Requires Cybersecurity T&E Planning
• General T&E Planning [Paragraph 3.b.(13)]– Work closely with the Chief Developmental Tester (CDT) and T&E WIPT to plan,
resource and conduct cybersecurity T&E– Refer to the Cybersecurity T&E Guidebook and DOT&E “Procedures for
Operational Test and Evaluation of Cybersecurity in Acquisition Programs”– Document T&E activities in TEMP, including the T&E Strategy, evaluation frameworks
(DT&E and OT&E), and resource requirements
• Requirements, Key Elements, and Resources [Paragraph 5.b.(10)]– Develop a cybersecurity T&E methodology based on derived system requirements and
draft performance specifications– Test key system elements and interfaces identified through criticality and vulnerability
analysis – Identify the cybersecurity T&E resources, (e.g., cyber ranges) for each T&E activity and
document T&E planning in the TEMP
• Cyber-Attack Surface [Paragraph 5.c.(5)] – For T&E, understand the cyber-attack surfaces and refine the T&E planning and
activities for cybersecurity
7
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
DoDI 5000.02, Enclosure 14Requires Cybersecurity DT&E and OT&E
• Development Test & Evaluation [Enclosure 14, Paragraph 3.b.(13)(a)]– Cooperative Vulnerability Identification (CVI)
Conduct T&E activities to collect data needed to identify vulnerabilities
– Adversarial Cybersecurity Developmental Testing (ACD)Conduct a cybersecurity DT&E event using realistic threat exploitation techniques in representative operating environments
• Operational Test & Evaluation [Enclosure 14, Paragraph 3.b.(13)(b)]– Cooperative Vulnerability and Penetration Assessment (CVPA)
An overt examination of the system to identify all significant vulnerabilities and the risk of exploitation of those vulnerabilities
– Adversarial Assessment (AA)Assesses the ability of a unit equipped with a system to support its mission while withstanding cyber threat activity representative of an actual adversary
• Provide T&E feedback to engineering teams [Enclosure 14, Paragraph 3.b.(13)]– This will help avoid costly and difficult system modifications late in the
acquisition life cycle
8
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
DoDI 5000.75Requires Cybersecurity T&E
• Defines policy and procedures, including cybersecurity, for DBS – Describes the use of the Business Capability Acquisition Cycle (BCAC) for business
systems requirements and acquisition– Outlines responsibilities the PM must implement to safeguard DoD business
systems throughout the system life cycle
• Program Office Implementation Plan must include cybersecurity processes to reduce technical risk through T&E management procedures
• Appendix 4B.2.h(2) requires:– Developmental Evaluation Framework– Cooperative vulnerability identification and adversarial cybersecurity testing in both
developmental and operational tests– A Cyber Economic Vulnerability Analysis (CEVA) - required at the discretion of
DOT&E for DoD systems whose functions include financial or fiscal/business activities or the management of funds
– Direction to Milestone Decision Authorities (MDAs) to avoid tailoring cybersecurity T&E solely to meet Authorization to Operate (ATO) requirements
9
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
DOT&E April 3, 2018 Memo, “Procedures for Operational Test and Evaluation of Cybersecurity in
Acquisition Programs”
• Reiterates requirement for CVPA and AA• Requirement applies to all system acquisition programs under
DOT&E oversight• Operational Test Agencies may tailor procedures specifically to
support the evaluation of weapons, platforms, networks and other systems that handle or transfer data and consider:
– Operational context– System extent– System-unique attributes– Specialized components
• Testing of cybersecurity during OT&E must include representative users and an operationally representative environment
• Certification, Accreditation, and/or Authorization processes should inform OT&E, but are not substitutes for OT&E
10
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
DoDI 8500.01 / DoDI 8510.01Planning and Conducting Cybersecurity T&E
• Cybersecurity [DoDI 8500.01]
– DASD(DT&E) and DOT&E collaborate on procedures for cybersecurity T&E− The Cybersecurity T&E Six Phase Process
– DoD Component− Provides for cybersecurity testing capability− Conducts vulnerability assessments− Ensures cybersecurity T&E is conducted throughout the acquisition lifecycle
– Defines activities for the CDT, Lead DT&E Organizations, and T&E community− Integrate RMF into DT&E − Document Cybersecurity T&E in the TEMP− Integrate with interoperability and other functional testing
• Risk Management Framework (RMF) [DoDI 8510.01]
– The RMF process will inform the acquisition process for all DoD IT, including developmental and operational T&E
– Requires integration of DT&E activities into the RMF and provides the RMF Technical Assurance Group with input as needed
– Ensure T&E of the assigned information system and information technology system is planned, resourced, and documented in the program T&E Master Plan
11
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
DoDI 8500.01Operational Resilience is Assessed During
Cybersecurity T&E
• Acquisition programs must conduct an operational resilience evaluation during cybersecurity DT&E and OT&E– Perform cybersecurity DT&E and OT&E, including the ability to detect and
react to penetrations and exploitations and to protect and restore data and information, in order to inform acquisition and fielding decisions. [Enclosure 3, paragraph 3.b]
– Exercise system under realistic cyber conditions using test procedures and tactics to develop work-arounds and fall-backs in the face of hostility [Enclosure 3, paragraph 3.e]
12
RMF Process Does NOT Assess Operational Resilience
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Joint Staff Guidance - System Survivability KPP
• JROCM 009-17 Item 4– Supports incorporation of cyberspace as a critical component of the SS KPP in
requirements documents– Applies to Joint programs and joint concern programs
• Service/component programs may include SS KPPs as applicable to mission context
• As part of the SS KPP assessment, the TEMP should describe: – Cyber survivability attributes – Technical performance specifications for the attributes– Countermeasures to support cyber survivability
• Programs should leverage the Cyber Survivability Endorsement Implementation Guide (CSEIG) developed by the Joint Staff/J6 in collaboration with DoD CIO, the DIA, and the NSA
– CSEIG consists of guidance that helps acquisition programs ensure cyber survivability requirements are included in system designs as early as possible
– Ensures cybersecurity is part of the operational risk trade space for functional requirements
– Incorporates cybersecurity attributes to support system survivability and operational resiliency requirements
13
Assess Cyber Survivability During DT and OT
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Cybersecurity T&E Process
14
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Cybersecurity T&E Process
15
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Cyber T&E analysis and planning DoDI 5000.75 Cyber T&E
Business Capability Acquisition Cycle (BCAC) and Cybersecurity T&E
BCAC
Phase 2 Characterize
the Cyber Attack
Surface
Phase 4 Adversarial
Cybersecurity DT&E
Phase 5 Cooperative
Vulnerability and Penetration
Assessment
Phase 3 Cooperative
Vulnerability Identification
Phase 6 Adversarial Assessment
Phase 1 Understand
Cybersecurity Requirements
Capability Need Identification
Business Solution Analysis
Business System Functional Requirements & Acquisition Planning
Business System Acquisition, Testing & Deployment
LimitedDeployment
ATP(s)
Acquisition ATP
Functional Requirement
ATP
Solution Analysis
ATP
Contract Award
Lower Fidelity Higher FidelityMission-Based Cyber Risk Assessments
Capability Support
Capability Support
ATP
FullDeployment
ATP
Alignment with BCAC (DoDI 5000.75) addressed in Appendix C of the Cybersecurity T&E Guidebook v2.0
16
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
System Development and TestCybersecurity, Survivability, and Resilience (1 of 2)
• Cyber Requirements and System Design– Design for system survivability – SS KPP
− Use Cyber Survivability Endorsement Implementation Guide to design for system survivability, as required by JCIDS Manual
– Design for operational resiliency in the operational environment under expected cyber threat conditions, as required by DoDI 8500.01
– Incorporate and validate cybersecurity controls as required by DoDI 8510.01, RMF
• Cybersecurity DT&E verifies system requirements to find problems and fix them during development – T&E conducted across cybersecurity, system survivability, and operational
resiliency objectives
17
Goal: Field Systems That Withstand Cyber Threats
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
System Development and TestCybersecurity, Survivability, and Resilience (2 of 2)
• Cybersecurity T&E (DT/OT) encompasses:– Cybersecurity assessments for software assurance, vulnerability
identification, configuration compliance and cybersecurity functionality verification (Phase 3) throughout the life cycle− Includes security controls assessment later in the life cycle for the RMF Step 4
and informs RMF Step 5− Submission to the AO
– System survivability testing to address specific cyber survivability attributes− Does the cybersecurity and system design prevent, mitigate, and recover from,
cyber-attacks (Phase 3, 4, 5 and 6)– Operational resilience testing
− Evaluates the ability of a system to successfully perform its mission in a cyber-contested environment
− When a threat is actively attempting to cause mission failure (Phase 4, 6)− Integrate with functional testing when possible
18
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Phase 1Understanding Cybersecurity Requirements
Understand the program’s cybersecurity and resilience requirements and develop an initial approach and plan for conducting Cybersecurity T&E• Cyber Working Group (CyWG) established• Compile List of Cybersecurity and Resilience Requirements
– Measurability, testability, and achievability• Prepare for Cybersecurity T&E Events
– Develop initial Developmental Evaluation Framework (DEF)– Identify Supporting Cybersecurity T&E Resources (labs, ranges, tools and personnel)– Develop the Initial OT Evaluation Framework– Align RMF activities with the TEMP– Plan and Schedule an MBCRA
• Plan for Cybersecurity T&E– Develop Cybersecurity T&E Strategy
Informs Request for Proposal (RFP), Preliminary Design Review (PDR), Capability Development Document (CDD), Solution Analysis Authority to Proceed (ATP), Functional
Requirements ATP, Acquisition ATP
Documented in MS A TEMP, Updated in MS B TEMP19
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Phase 2Characterize the Cyber-Attack Surface
Characterize attack surface to identify opportunities an attacker may use to exploit the system• Identify the cyber-attack surface
– Examine System Architecture, Components and Data Flows– Analyze and Decompose System Mission– Map Mission Dependencies
• Analyze the cyber-attack surface– Characterize the Cyber Threat– Select a Cyber Kill Chain– Examine Cyber Effects on the System and Mission– Perform (or Update) Mission-Based Cyber Risk Assessment
• Document Results and Update Test Planning and Artifacts• Prepare test strategy for Phase 3 and Phase 4 Cybersecurity DT&E Events
Informs RFP, PDR, Capability Development Document (CDD) validation, Critical Design review (CDR), MS B, Functional Requirements ATP, Acquisition ATP
Update the TEMP for MS B20
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Phase 3Cooperative Vulnerability Identification
Identify the existence of any known cyber vulnerabilities in hardware, software and architecture and verify cyber survivability and resilience capabilities• Early and ongoing feedback starting at MS B• CVI is NOT a single event
– Contractor T&E– Government T&E– Continuum of vulnerability assessment activities tailored to the program
• Integrates RMF assessments• Develop test objectives, plan events and infrastructure during Phases 1 and 2 • Vulnerability testing while planning for threat testing in Phase 4
Informs CDR and Test Readiness Review, Functional Requirements ATP, Acquisition ATP, Limited Deployment ATP
CVI Events are Risk Reduction Activities21
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Phase 4Adversarial Cybersecurity DT&E
Evaluate the system’s cybersecurity in a mission context, using realistic threat exploitation techniques while in a representative operating environment• Program must plan to replicate the system in a representative test infrastructure• Updated threat assessment• Cyber table top (CTT) exercises and service-specific MBCRAs help to inform
Phase 4 test objectives and test plans/vignettes• Execute prior to MS C and prior to the Authorization to Operate (ATO)• Cyber risk assessment describing operational mission impacts from tested cyber
attacks
Informs MS C production decision, ATO and Operational Test Readiness Review, Limited Deployment ATP
Certified Red Team NOT required22
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Phase 5Cooperative Vulnerability and Penetration
AssessmentProvide a comprehensive characterization of the cybersecurity and resilience status of a system in a fully operational context and provide reconnaissance of the system to support adversarial testing• Uses data taken from cooperative cybersecurity test events• Early engagement with the OTA is essential for planning
– Plan and coordinate with cybersecurity vulnerability assessment team (“blue team”)• Can be integrated with CVI activities, a standalone event, a series of test events,
or an operational component of an integrated test
System vulnerability data supports adversarial testing in Phase 6, MS C, LRIP, Full Rate Production (FRP), Full Deployment Decision (FDD) and ATP
decisions
Aligned to 2018 DOT&E Memorandum
23
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Phase 6Adversarial Assessment
Characterizes the operational effects to critical missions caused by threat-representative cyber activity against a unit trained and equipped with a system, as well as the effectiveness of defensive capabilities• Evaluate the ability of the system, tiered defenses, and defenders to
protect critical mission functions; detect and respond to cyber-attacks; and assess system resilience to survive and recover from attacks, and complete critical missions and tasks
• OTA plans testing• National Security Agency Certified Red Team performs testing
Results inform the operational effectiveness, suitability, and (in some cases) survivability of the system(s) under test due to cybersecurity vulnerabilities and the
resulting mission effects. Also informs FRP or FDD and Full Deployment ATP
Aligned to 2018 DOT&E Memorandum
24
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Mission-Based Cyber Risk Assessment (MBCRA)
• A process of identifying, estimating, assessing and prioritizing risks based on impacts to DoD operational missions resulting from cyber effects on the system(s) being employed
• Informs RMF Steps 1-5 AND informs Cybersecurity T&E planning– Activities begin in Phase 1
• Identifies mission-impacting risks to test and mitigate– Assists in focusing and prioritizing the Cybersecurity T&E effort– Several common methodologies, including Cyber Table Tops
• Best practices described in Cybersecurity T&E Guidebook v2.0 Appendix X3 (FOUO Appendix)
25
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
The Risk Management Framework is Necessary
• Required by policy – DoDI 8500.01 3.a and 3.h requires cybersecurity risk management– DoDI 8510.01 Risk Management Framework (RMF) implements DoD’s Risk
Management Policy • RMF provides a structured, tailorable, and repeatable process
that integrates security and risk management activities into the system development life cycle
– Considers effectiveness, efficiency, and constraints due to applicable laws, directives, Executive Orders, policies, standards, or regulations
• RMF helps ensure the appropriate “cyber hygiene” controls and security configurations are designed into the system
– Protections to help meet the goals of risk-managed Confidentiality, Integrity and Availability
– Adds continuous monitoring to system life cycle management to ensure ongoing awareness of and risk managed responses to changing threats and environments
26
RMF Does Not Replace Cybersecurity T&E
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
RMF Alignment with T&E Process
27
PHASES
TechnologyMaturation & Risk
Reduction
Engineering & Manufacturing
Development Production & Deployment
Materiel SolutionAnalysis
MS CMS BMS A
Phase 2 Characterize
the Cyber Attack
Surface
Phase 4 Adversarial
Cybersecurity DT&E
Phase 5 Cooperative
Vulnerability and Penetration
Assessment
Phase 3 Cooperative
Vulnerability Identification
Phase 6 Adversarial Assessment
CDD Validation
Dev RFP Release Decision
IOT&EOTRR
IATT
Full RateProduction
Decision ReviewATO
Phase 1 Understand
Cybersecurity Requirements
Lower Fidelity Mission-Based Cyber Risk Assessments Higher Fidelity
DRAFTCDD CDD
PDR CDR TRRCPD
Operations & Support
MDD
RMF Step 1 Categorize
RMF Step 2Select controls
RMF Step 3 Implement Controls
RMF Step 4 Assess Controls
RMF Step 5 Authorize
New ATO ~3 years
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Cybersecurity T&E in the TEMP
28
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Cybersecurity in the TEMP (1of 2)(Starting at MS A)
• T&E strategy identifies Cybersecurity DT&E activities (contractor and govt) to:– Understand cybersecurity requirements - compile and analyze for
measurability, testability, and achievability (Phase 1)– Expose system's reachable and exploitable vulnerabilities to
characterize the attack surface (Phase 2)– Assess sub-components and components against potential
vulnerabilities (Phase 3)– Assess system against potential vulnerabilities (Phase 3)– Assess system resiliency against an adversarial cybersecurity
threat (Phase 4)
29
High Level Descriptions of Who, What, Where, When, Why, How
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Cybersecurity in the TEMP (2 of 2)(Starting at MS A)
• DT&E Methodology includes cybersecurity and outlines the essential information needed to support programmatic, technical, and acquisition decisions
– If possible at MS A, include a Developmental Evaluation Framework (DEF) that identifies cybersecurity data for assessing progress toward cybersecurity requirements
– DEF is required at MS B• Identify when integrated cybersecurity testing (DT – OT) will
occur• Include RMF categorization and integrate RMF data needs into
DT&E activities• Include the SS KPP Cybersecurity Risk Categorization and the SS
KPP Cyber Survivability Attributes– Attributes should be available at MS B
• Define roles, responsibilities, and resources for detailed planning and execution of Cybersecurity DT&E activities, e.g., use of cyber ranges
30
TEMPs Should Not be a “Copy and Paste” of Guidance or Policy
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Cyber Test Planning in the TEMP
31
MS A TEMP MS B TEMP MS C TEMP
What to Test
Approach to conducting Phase 1 & 2 to inform cyber test events. Discussion of iterations of Phase 1 & 2 to support requirements and design changes
Test Events Planned (Who, When, Where, limitations, etc.)• Based on functions critical to
mission success that are potentially vulnerable
• Informed by mission-based cyber risk assessments
Testing conducted and testingyet to be done
How to TestDiscussion of plans for dedicatedcyber test events, integrated DT/OT events, and SCA
Test events fully documented: tools, contractor development labs, cyber ranges, etc.
Testing completed and described with references to test results
DEF - Cyber Initial DEF showing continuum of cyber test activities expected
Updated DEF showing planned cyber test activities; DEF is required at MS B
Updated DEF for future testing if needed
DataData needed to inform decision makers and system design process
Plans to collect required data during cyber testing
Test data used for mission-based risk assessment informing ACD events and TRR
ScheduleApproach to scheduling cyber test events including security controls assessment, integrated DT/OT, and functional T&E integration
Schedule for cyber tests and their estimated duration; test results inform CDR and TRR
Tests remaining to be conducted in schedule
ResourcesDiscussion of required test resources: people, test environment, processes, tools, etc.
Resources allocated to cyber testers (org that will supply testers) and test articles (HWIL, SIL, virtual systems, etc.)
Updated resources allocated to future testing
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Decision #3
DSQ #1 DSQ #2 DSQ #3 DSQ #4 DSQ #5 DSQ #6 DSQ #7 DSQ #8
Functional evaluation areas System capability categories
Technical Reqmts
Document Reference
Description
Performance3.x.x.5 Technical Measure #1 DT #1 M&S #2 DT #4 M&S #2
3.x.x.6 Technical Measure #2 M&S #1 DT #3 DT #4 M&S #2
3.x.x.7 Technical Measure #3 DT #3 IT #1
3.x.x.8 Technical Measure #4 M&S #4 IT #1
InteroperabilityInteroperability Capability #1 3.x.x.1 Technical Measure #1 DT #3 DT #4
3.x.x.2 Technical Measure #2 IT #2 M&S #4 DT #4 M&S #2
3.x.x.3 Technical Measure #3 IT #2 IT #1 DT #3
Cybersecurity
5.x.x.1 Technical Measure #1 CTT #1 CVI #2 ACDT CVI #3
5.x.x.2 Technical Measure #2 CTT #1 SCA ACDT CVI #3
5.x.x.3 Technical Measure #3 CVI #1 SCA ACDT ACDT
5.x.x.4 Technical Measure #4 CTT #1 CVI #1 ACDT CVI #3
Reliability4.x.x.1 Technical Measure #1 M-demo
#1 IT #54.x.x.2 Technical Measure #2 M-demo
#1 IT #2 IT #54.x.x.3 Technical Measure #3 M-demo
#2 IT #24.x.x.4 Technical Measure #4 M-demo #2 IT #2
Reliability Capability #1
Reliability Capability #2
Data Sources (Test, M&S events)
System Requirements / Measures
Developmental Evaluation Objectives
Decision #1 Decision #2 Decision #4
Performance Capability #1
Performance Capability #2
Security Capability #1
Interoperability Capability #2
Security Capability #2
Cybersecurity in the Developmental Evaluation Framework
32
Decisions
Evaluation
Test / M&S
Resources
Schedule
Def
ine Inform
Def
ine Data
Def
ine Execute
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Example Completed Cyber Portion of a DEF (Notional)
33
Developmental Evaluation Objectives
System Requirements/ Technical Measures EMD RFP Release
MS B / Contract AwardEMD Long Lead Items for A/C (A1, A2, A3) & Radars (for A/C and SIL)
Approval to Enter Gov't Led IDT&E
LRIP Long Lead Items Approval to Enter IOT&E
System Capabilities
SRD Rqmt's (Potential CTPs*)
Technical Measures
DSQ1: Did at least two Contractors provide technical
designs and information for
successful PDRs?
DSQ2: Have at least two
Contractors demonstrated
sufficient subsystem maturity?
DSQ3: Can the Aircraft meet
Requirements?
DSQ4: Can the Radar and SUT
subsystem integration meet Performance and
Processing Requirements?
DSQ5: Has the KTR demonstrated a fully integrated, functional and
stable, Radar/Comm/ C2 capability in the
SIL?
DSQ6: Has the KTR demonstrated a fully integrated, functional and
stable, Aircraft/ Radar/Comm/C2
system?
DSQ7: Do any system deficiencies
preclude an LRIP purchase?
DSQ8: Does the performance and reliability support
all required mission profiles?
DSQ9: Are cybersecurity vulnerabilities identified and
acceptable mitigations in place?
CyberSecurity Data Sources (Analysis, Test, M&S Events)
Protect; Data Security - System
Data at rest, Data in transmission
Architectural Vulnerability Analysis (AVA)
Mission Cyber Dependency
Analysis - Cyber Table Top Exercise
CVI-Data Security testing
CVI-STIG compliance verification
Security Controls Assessment (SCA)
ACD
RMF Risk Assessment/ATO
Software Assurance
Architectural Vulnerability
Analysis (AVA)Contractor T&E
CVI - Software Development Verification
CVI - Software Development Verification
PPP Supply Chain Risk Management
SCRM TAC Assessment
SCRM TAC Assessment
SCRM TAC Assessment
SCRM TAC Assessment
Hardware Assurance
Architectural Vulnerability
Analysis (AVA)Contractor T&E
CVI - Hardware Development Verification
CVI - Hardware Development Verification
Protect; Data Security –Interfaces
Critical Data Exchanges
Architectural Vulnerability Analysis (AVA)
Interoperability -Cybersecurity IT
CTT Verification Exercise
System Resilience and Survivability
SS KPP CSA
Detecting attacks (how long to
detect, how many detected versus
attempted, mission impacts)
Mission Cyber Dependency Analysis - Cyber Table Top ExerciseCVI-Cyber
Functionality Verification
CTT Verification Exercise ACD
SS KPP CSAResponding to
attacks (how long to respond)
Mission Cyber Dependency Analysis - Cyber Table Top ExerciseCVI-Cyber
Functionality Verification
CVI - Incident Response
AssessmentACD ACD
SS KPP CSA
Recovering from attacks (how long
does recovery take? Does that
impact success of the mission?)
Mission Cyber Dependency Analysis - Cyber Table Top Exercise CVI - COOP assessment ACD
ACDCVI - COOP assessment
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Cyber Ranges
34
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Cyber Range Overview
• Adequate DT&E, OT&E, and assessments may require testing on Cyber Ranges for one or more of the following reasons:
– Testing cannot occur on open operational networks– Representations of advanced cyber adversarial TTPs are not suitable for
operational networks– Scaling requirements (e.g., number of users, hosts, or interconnected systems;
amount of network traffic) cannot be otherwise achieved– Operational complexity and associated mission risk are such that impact to
operational networks should be avoided
• Planning for the use of a cyber range should begin as early as possible in the acquisition lifecycle and be reflected in the TEMP
• For more information about test range planning for Cyber T&E, refer to the Cybersecurity T&E Guidebook 2.0 Appendix X4
35
Cybersecurity T&E Performed Only During OT&E is Too Late
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
When to Use a Cyber Range
Test Range EventLarge-scale Simulation to Train Cyber Mission
Forces and Evaluate Cyber Defensive and Offensive Operations
Pre MS A/BRequirements and Systems Security
Engineering Analysis
Test Range EventMission Thread
Testing with Blue Team
Test Range EventMission Thread Testing with Red
Team in a Realistic Threat Environment
SE/DT&EEvaluate Software and
Systems Security Architecture
Training & ExercisesEvaluate TTPs in a
Contested Environment
Test Range EventCybersecurity
Verification and Validation
RMF/DT&EVerify Baseline Cybersecurity
Requirements and Vulnerability Assessment
DT&E/OT&EEvaluate Mission Capabilities
and Interoperability in a Contested Environment
Test Range EventCybersecurity Architecture Evaluation
36
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Cybersecurity T&E Guidebook
37
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
DoD Cybersecurity T&E Guidebook
• Version 2.0 published April 2018• Describes each phase, inputs, outputs, tasks• Addresses RMF integration• Includes new appendices - FOUO appendices published
separately (June 30, 2018)• Publicly accessible links to the Guidebook:
– https://www.acq.osd.mil/dte-trmc/docs/CSTE%20Guidebook%202.0_FINAL%20(25APR2018).pdf
– https://www.dau.mil/cop/test/DAU%20Sponsored%20Documents/CSTE%20Guidebook%202.0_FINAL%20(25APR2018).pdf?Web=1
38
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Cybersecurity T&E Guidebook Outline
• Introduction– Sections: Purpose, Organization, Audience– Shortened from v1.0
• Cybersecurity in the Defense Acquisition System– Overview of policy basis for Cyber T&E– Sections: DoDI 5000.02, DoDI 5000.75 Defense Business Systems, DoDI 8500.01
Cybersecurity, DoDI 8510.01 Risk management Framework (RMF), Joint Requirements Guidance, DOT&E Cybersecurity Procedures Memoranda
• Cybersecurity T&E - Phases Overview– Cyber Working Group, cyber threat assessments, DT&E and SE collaboration, early
tester involvement, MBCRA, role of Cybersecurity DT&E, DT&E and OT&E collaboration
• Phase 1 Through 6– Sections are uniform through each phase
− Purpose and schedule− Inputs – from Guidebook v1.0− Tasks – tailored for each phase, includes methods and best practices− Outputs – TEMP updates, acquisition decision informed
– RMF and MBCRA/Cyber Table Top (CTT) Exercise integration throughout each phase
39
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Cybersecurity T&E Guidebook Unclassified Appendices
Appendix A (new)Phase 1-6 Quick Look– Single page for each phase showing inputs, tasks, and outputs
Appendix B (new)Incorporating Cybersecurity T&E into DoD Acquisition Contracts
Appendix C (new)Considerations for Tailoring the Cybersecurity T&E Phases
Appendix D (update)Key Program Artifacts for Cybersecurity T&E Analysis and Planning
Appendix E (update)Guidance on the Cybersecurity Portion of the Developmental Evaluation Framework
Appendix F (new)Considerations for Staffing Cybersecurity T&E Activities
Appendix G (new)Considerations for Software Assurance Testing
40
CLEARED for Open Publication, August 06, 2018, DoD Office of Prepublication and Security Review, Case #18-S-1977
Cybersecurity T&E Guidebook FOUO Appendices
For Official Use Only Appendices are accessible to government and authorized contractor personnel
– Contact DASD DT&E Cybersecurity Technical Director
Appendix X1 (significant revision)Considerations for Cybersecurity Requirements and Measures for DT&E– Cyber Survivability Endorsement Implementation for System Survivability Key Performance Parameter– STAT Metrics for Cybersecurity Test Objectives
Appendix X2 (new)Cyber Threat Assessment for Cybersecurity T&E– Integrated with phases– Supply Chain Risk Management Threat Assessment Center
Appendix X3 (new)Mission-Based Cyber Risk Assessments (MBCRAs)– Survey of MBCRA methods including Cyber Table Top exercises; how to select an MBCRA
Appendix X4 (updated)Cybersecurity Test Infrastructure and Environment Planning– Test environments from development to OT; considerations for planning cyber test infrastructure; services-specific
infrastructure; cyber range use during a MBCRA/CTT
Appendix X5 (new)Cybersecurity Test Considerations for Non-Internet Protocol (Non-IP) Systems – Introductory materials only; 1553 and 1439 (CAN) bus testing; controls systems testing
41
Recommended