Test, Evaluation, and Certification Competency

SPAWAR 5.9 Overview

for AFCEA SDAugust 2011

BrieferJohn Iaia, SSC-PACIFIC Code 59000

SPAWAR Test , Evaluation & Certification National Competency Lead

Dist Statement

2

SPAWAR 5.0 Tier II Competency Functions

5.1 Enterprise Systems Engineering

5.2 Net-centric Engineering & Integration

5.3

C2

5.4

Business Systems/

EIS

5.5

Comms and

Networks

5.6

ISR/IO

5.7

Space Systems

5.8 Information Assurance

5.9 System T&E and Certification

5.0 Engineering

3

TEAM SPAWAR 5.9 Organization Chart

SPAWAR HQ 5.0 CHENGRDML Rodman

SPAWAR HQ 5.0 CHENGRDML Rodman

5.9 CAO HQ InitiativesDeputy - Keven Nelson5.9 CAO HQ InitiativesDeputy - Keven Nelson

5.9 TE&C Competency National Competency Lead

John Iaia / SSC-PAC

5.9 TE&C Competency National Competency Lead

John Iaia / SSC-PAC5.0.9 TE&C DEPT.

5.9 CAO Local Competency LeadScott Cramer / SPAWAR HQ

5.0.9 TE&C DEPT.5.9 CAO Local Competency Lead

Scott Cramer / SPAWAR HQ

5.9 TE&C Competency Jeff King / SSC LANT

5.9 TE&C Competency Jeff King / SSC LANT

SPAWAR TE&C Process Leadsand Certification Authority

SPAWAR TE&C Process Leadsand Certification Authority

Hilton, Paul TBDKirksharian, AronRicketts, Rose Le, BeemDr. Wu, Kevin

CA - NAVY CA TPO- V&V TAE -TA/SETRCA - LINK16 Certs TAE - FRCB TAE -SPEG

SPAWAR HQ TE&C LiaisonsSPAWAR HQ TE&C Liaisons

Nelson, KevenKolacki, BobLt. Roland, Theresa

5.9 CAO DoN, StakeholdersMilitary

5.9 CAO SSC-LANT NC Deputy / LCLJeff King

5.9 CAO SSC-LANT NC Deputy / LCLJeff King

5.9 CAO SSC-PAC Local CompetencyLead – Dan Slack

5.9 CAO SSC-PAC Local CompetencyLead – Dan Slack

SSC-LANT 5.9 CAO TA/TE&C LeadsSSC-LANT 5.9 CAO TA/TE&C Leads

Kassel, Ken

Guardia-Baker, Marilene

Vacant

Rideout, Ann

5.9.1 Planning and Req. Engineering

5.9.2 TE&C Execution

5.9.3 TE&C Resources

5.9.4 TE&C Cert. Standards & Verification

SSC-PAC 5.9 CAO TA/TE&C LeadsSSC-PAC 5.9 CAO TA/TE&C Leads

Vacant

Dimitriu, Anastasia

Watne, Nate

Whitby, Pete

5.9.1 Planning and Req. Engineering

5.9.2 TE&C Execution

5.9.3 TE&C Resources

5.9.4 TE&C Cert. Standards & Verification

SSC-PAC 5.9 CAO Tier II & III OrganizationSSC-PAC 5.9 CAO Tier II & III Organization SSC-LANT 5.9 CAO Tier II & III OrganizationSSC-LANT 5.9 CAO Tier II & III Organization

SSC Pacific 5.9 Competency Primary: 82Secondary: 305

SSC Atlantic 5.9 Competency Primary: 136Secondary: 150

TEAM SPAWAR

4

TE&C Competency – Primary/Secondary 5.9 Test, Evaluation, and Certification Competency Organization

5

5.9 Tier III Competency Functions▼ 5.9/5.9A

▼ Test, Evaluation

▼ and

▼ Certification

▼ 5.9.1

▼ TE&C Planning

▼ and

▼ Req’ts Engineering

▼ Provides TE&C process

▼ improvement and oversight,

▼ test/cert alignment

▼ across boundaries

▼ 5.9.2

▼ TE&C Execution

▼ Provides TE&C support,

▼ designs and conducts

▼ testing of C4I components,

▼ SoS, FoS, and E2E.

▼ 5.9.3

▼ TE&C Resources

▼ Provides TE&C support

▼ capabilities, management

▼ and innovation

▼ 5.9.4

▼ Certification

▼ and Standards

▼ Verification

▼ Provides independent

▼ verification and

▼ certification of IT/IS,

▼ standards and T&E Facilities

▼ Navy CA

▼ FRCB TAEs/SMEs

▼ SOVT TAEs

▼ Interop Test SMEs

▼ Link 16 EMC CA

▼ E2E Test SMEs

▼ T&E Labs

▼ Networks

▼ C4I Ranges

▼ M&S/Stim

▼ TE&C Tools

▼ Distributed Test

▼ E2E

▼C

2, B

us IT

, IA

, Com

ms

&

▼ N

etw

orks

, ISR

/IO, S

pace▼ TE&C Process

▼ Improvement

▼ SETR Checklists

▼ Guidance/Policy

▼ TE&C Rqmnts

▼ Analysis & Oversight ▼ TE&C Acq

▼ Doc Review

▼ RMA &

▼ Human

▼ Factors

▼ SoS

▼ FoS

6

SPAWAR TE&C Focus Areas

▼ Processes & Policy ▼ People & Organization

▼ Tools

▼ Resources▼ Tied together

▼ with Knowledge

5.9 TE&C National Lead

5.0 National Lead(RDML Rodman)

IPTs IPTs IPTs IPTs

A

5.9.1

5.9 TE&C Local Lead

5.9.1 5.9.2 5.9.4

5.9.25.9.3

5.9.4

5.9.15.9.2

5.9.35.9.4

5.9.15.9.2

5.9.35.9.4

5.9.15.9.2

5.9.35.9.4

B C D

5.9.3

7

TEAM SPAWAR

TE&C Strategy• Refine and advance standardized processes JPAT to further standardization Tailoring /scaling guidance PAL/PML/OMR-accessibility focus Template baseline expansion Systems Engineering Guide Vol. II

• 5.9 TE&C Academy- expand to DON

• 5.9 CDM DOE Training IT core plus IDP Guidelines Focused 852 course offerings• Continue DAWIA Billet

Adjudication

• Continue personnel alignment Primary/Secondary requirements and training• Enable Facilities alignment and management

• Pilot /Refine C4ISR Certification Process

C4ISR Certification- TBD/Essex• Advance the C5ISR Certification

Concept Develop cross SYSCOM

Certification CONOP/Way-Ahead• E2C Construct

Proposed JMETC Pilots DOD T&E Reliance Needs process

• E2C CONOP Refinement in concert with SE & Acquisition• Focused on linkages with E2E Acquisition WIPTs• Emphasis on distributed test environment operations• Ensure Comprehensive C5ISR Certification and TA Support

• Risk Management tied to all phases of test

• Key enabler/tool for E2E test scoping

8

TE&C Processes & Tools▼ E2C Test Activities and

Processes▼ V&V of Labs▼ C4ISR Certification

C4ISR Certification Model Effort− Assessment by

Demonstration− Assessment by Analysis− NWSCP Article 13

▼ PEO-C4I T&E Management PMW-120 PMW-130/160 PMW 170 PMW 150 Maritime C2

▼ SOVT Process and Execution Guidance (SPEG) Integrated SOVT

▼ Process and Artifact Management NSERC- T&E Document Archive SEG and CMMI OMR

▼ T&E Process JPAT▼ T&E TA Support▼ TE&C Education Efforts

T&E Academy-DON DAWIA Engagement Section 852 Funds DOE Training

9

C4I Platform Baseline ECR Stage 1 / Risk Assessment

PMW DevelopmentTesting and Documentation Completed. PMW submits Package to PEO boards

PGB E2C Event

Decision Point

Assessment By Demonstration

(SoS T&E)

C4I Platform Baseline ECR Stage 2 / Risk Assessment

Test/Cert Artifacts:IA Cert

OTA CertJITC

LinkQual Tests

SOVTEtc High Risk

InterfacesMed/LowRisk Interfaces

Assessment Data to Engineering Risk Assessment

5.0 Technical Authority

Review Board, review results

Cert Criteria

Met?

NWSCP WSCA/D(A-8/E-1)

Re-Work

Yes

No

Baseline Definition

Capability Assessment

Certification Decision

C4I Certification Process

C4I Platform Baseline ECR Stage 3 / Risk Assessment

TD SignatureFor Authorization/

Certification Approval

Cross SYSCOM Coordination

Other Cert Rqmnt TBD

Core E2C Lab

Distributed Node

SSC Pacific(San Diego)

SSC Atlantic(Charleston)

SPAWAR HQ(San Diego - Old Town)

SSC Atlantic(New Orleans)

SIPR/NIPR/JWICS, SDREN/DREN,

DISN/DISN-LES, ADMS, NMCI,

HSGR

SSC Atlantic(St. Julien’s Creek)

NCTAMS LANT

NCTS San Diego

NUWC(Newport, RI)

Distributed Lab Environment

Assessment By Analysis

10

C4I Certification ISO NWSCP Status

▼ Team in place 5.0.9, APEO(E), APMEs, CSEs, 5.X SMEs, .. Weekly Drumbeat established

▼ Building off work already done C4I Cert DFLSS/Non-E2C Assess RIE PEO Goal 1.1/1.2 No “test/cert” left behind

▼ Key Tenants Must be Achievable: executable within fiscal reality Must be Repeatable: not a one trick show Must be Fungible: adaptive to more than just NWSCP Must be Tangible: concrete enough to build a Policy

Stra

tegy

Com

plet

e

(Dra

ft C

ON

OP

S)

Fina

lize

CO

NO

PS

FY11 FY12 FY13

Dra

ft P

roce

ss

Def

ined

Wor

ksho

ps

Sta

rt P

ilot

End

Pilo

t

Pol

icy

Sig

ned

Exe

cute

Pro

cess

11

System of Systems Test – E2C Concept of Operations

SPAWAR/PEO C4I/SSC processes, engineers, and laboratory environments supporting development of End-to-End integrated solutions Verify system-of-systems components work together in an operationally

representative environment before they are fielded Provides operationally representative environment for development and

experimentation for new capabilities Entry Point to SPAWAR C4ISR (distributed test environment) for purposes of C4ISR Certification/C5ISR Test and Certification Support

Development

Integration

Delivery

TEAM SPAWAR

12

▼ Questions?

13

14

SOVT Preparation and Execution Guide (SPEG)

▼ Defines and requires standardized System SOVT▼ Defines and requires Platform SOVT based on the

approved System SOVT.▼ SOVT Common Format and Content requirements

Test boundary Test stages - A building block approach, where each

subsequent test stage builds upon known test results from previous test stages− Stage 1: Inspection and Validation− Stage 2: Cold Checks− Stage 3: Equipment Tests− Stage 4: Intra-system Tests− Stage 5: Inter-system Tests− Stage 6: Operational Tests− Stage 7: At-Sea and Other Special Tests

Minimum testing content Microsoft Word template included

▼ SOVT Test Element Rigor defined Analysis, selection, and mapping to specific test stages

15

FRCB Policy and Process▼ NETWARCOMINST 12271.1 dtd 17 Oct 2006:

“… all C4ISR systems planned for installation at Naval shore activities receive FRCB approval prior to the Installing Activity's commencement of onsite work.”

▼ COMNAVNETWARCOM deems the FRCB process to be invaluable and is committed to its continued use as a means of ensuring that mature C4ISR products are fielded to support Fleet operations and training”