Embed Size (px)
Citation preview
Test and Evaluation in a ‘complex
distributed’ customer environment
Dr Dai Morris
UK Complex Weapons Senior Responsible
Owner
SOME BACKGROUND
Weapons, Evaluation and Capability Assurance
UK Defence Reform – The “Levene” review
Weapons, Evaluation and Capability Assurance
Defence Acquisition Reform
Being one part of wider Defence Change
Defence Equipment & Support
● ‘Bespoke Trading Entity’ (Arms length Body of MoD)
● Procurement, Support & Logistics
Formalization of ‘Customer Relationship’
● Between Commands and DE&S
● Drives a stable ‘ask’ thus reducing change friction
● Improves ‘Hold-to-Account’ and delivary
Weapons, Evaluation and Capability Assurance
The UK ‘distributed’ customer is not dispersed!
Land
DE&S
Navy
Air
Joint
HO / Strat P
AN EXTERNAL VIEW OF T&E
Weapons, Evaluation and Capability Assurance
Why is this important?
UK Delegated Model
● No DOT&E!
● Commanders have the responsibility and authority
-And a very difficult resource challenge
-They can (and DO) trade time & cost in all aspects of a project
-Safety & Contractual compliance test tends to be robust
● The trade space can be in Operational Test….
Not really dispersed, but certainly distributed
● Different management chains only separated by one
floor of a building really matter
● The Customer needs to ‘want’, not just hear “must”
Weapons, Evaluation and Capability Assurance
Some common ‘Assertions’ – or Myths
Testing:
● Delays the Programme
● Is always over-specified
● Aims to break things, not show they work
● Is always too expensive
● Can all be done in simulation
● Is the most important part of any programme
So what does this say about the external view
of the Evaluation community?
Weapons, Evaluation and Capability Assurance
The Evaluation Community Solution
Demonstrate Operational Excellence
● Plan evaluation early, plan well, plan agile
● Never buy before you know how to test
● Demonstrate the value of ITEAPs, …..
● Invest in the infrastructure for efficiency
“The output is essential and worth it!”
● But is it valued by the whole customer community?
T&E AS AN ORGANISATIONAL,
NOT JUST PROJECT, BENEFIT
Weapons, Evaluation and Capability Assurance
Evaluation Spend Profile – a risk indicator?
0
10
20
30
40
50
60
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
An
nu
al
Sp
en
d
Project A
Project B
Weapons, Evaluation and Capability Assurance
Evaluation activity as a Prog. management tool
Evaluation is Risk Management
● So how much do we focus on the ‘up-side’ discovery?
● And how do we build that into Operational Capability?
● Can we use ‘Measure & Declare’ more
How integrated are the Training and Evaluation
communities?
Evaluation of the requirement
● Do we spend enough resource ‘helping’ set the
requirement?
● The evaluation community can identify optimism bias
Weapons, Evaluation and Capability Assurance
Great idea, or invitation for disaster?
What the advocate would like you to see What the Evaluation Community sees
EVALUATION & MISSION
EFFECTIVENESS
Weapons, Evaluation and Capability Assurance
Up-side ‘Risk’ - A simple Pan-DLoD Evaluation
0
20
40
60
80
100
120
140
First
Contact
Report
Second
Contact
Report
Third
Contact
Report
Fourth
Contact
Report
Fifth
Contact
Report
BG Wng
Order
Sqn/Coy
Gp Wng
Order
Sqn/Coy
Gp Order
H Hr Attack
Complete
Event
Tim
e (
min
)
Recce Phase C2 Phase Assault
Phase
ND
D
Weapons, Evaluation and Capability Assurance
0
10
20
30
40
50
60
70
80
90
0 1 2 3 4
HH
r -
Wn
gO
(/m
inu
tes
)
Group
1 – “Baseline Control”
• no changes to any DLOD
• illustrates the potential of training to
vary and improve average
performance.
2 – “Capability Evaluation”
• Change to Equipment DLOD only
• Improves Capability in statistically
significant manner.
3 - “Emergent Condition”
• ‘Habitual test subjects’ changed
Doctrine & TTPs to suit the change
to their Equipment.
• Suggests that changes in non-
equipment DLoD can cause great
improvement in capability.
‘The DLoD experiment in more detail….
Weapons, Evaluation and Capability Assurance
‘Defence Lines of Development
DLODS can be thought of as of as Primary or Enabling
when used to explore / assess improved capability:
Doctrine & Concepts
Equipment
Organisation 1° DLOD are those which
characterise the
theoretical maximum
capability.
Personnel
Logistics
Information
Training
Infrastructure
Enabling DLOD
can increase
capability to the
level defined by 1°
DLOD.
This FD construct doesn’t imply 1° DLODs are more important. It simply
serves to facilitate a potential Evaluation Approach.
Weapons, Evaluation and Capability Assurance
How Good are we at Evaluating DLoDS?
0 2 4 6 8 10
Interoperability
Logistics
Infrastructure
Organisation
Concepts & Doctrine
Information
Personnel
Equipment
Training
Weapons, Evaluation and Capability Assurance
A personal conclusion
Testing for ‘mission effectiveness’ is
fundamentally different to testing equipment
performance
● It should be a discovery process, not (just) verification
● A real challenge for experimental design
Our current Evaluation Capability is not always
well matched to the task
● DT, OT, Training communities all have a part to play
● It’s a through-life problem, not just ‘Milestone C’
-Mission effectiveness is as much about red as blue…
Weapons, Evaluation and Capability Assurance
Some Final Thoughts
We do need to sharpen our tools
● Especially in the non-equipment area
Evaluation is about Risk Management
● So how do we get at the ‘upside’ not just the downside?
The community has much to offer a SRO
● But it needs to communicate in a different way
● And accommodate the culture difference
Peter Drucker – ‘Culture eats strategy for breakfast’
Weapons, Evaluation and Capability Assurance
Questions, Comments ?
Test and Evaluation in a ‘complex
distributed’ customer environment
Dr Dai Morris
UK Complex Weapons Senior Responsible
Owner
