1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic...

11

Measurements & ValidationMeasurements & Validation

R G StampR G Stamp

Head of Department of ATM ResearchHead of Department of ATM Research

National Air Traffic Services Ltd, UKNational Air Traffic Services Ltd, UK

22

What is Validation?What is Validation?

EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)

““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be

demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”

33

What is Validation?What is Validation?

EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)

““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be

demonstrated against a demonstrated against a pre-defined levelpre-defined level of of functionality, operability and performance”functionality, operability and performance”

44

What is Validation?What is Validation?

EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)

““The process through which a The process through which a desired level of desired level of confidenceconfidence in the ability of a deliverable to in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be

demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”

55

What is Validation?What is Validation?

EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)

““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a operate in a real-life environmentreal-life environment may be may be

demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”

66

What is Validation?What is Validation?

EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)

““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be

demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”

Validation builds confidence that the system is Validation builds confidence that the system is fit for purposefit for purpose

77

How to ValidateHow to Validate

Operations

Operationaltrial

Shadow-modetrial

Field test

Large-scalereal-time

simulation

Small-scalereal-time

simulation

Fast-timesimulation

Analyticmodelling

88

How to ValidateHow to Validate

Operations

Operationaltrial

Shadow-modetrial

Field test

Large-scalereal-time

simulation

Small-scalereal-time

simulation

Fast-timesimulation

Analyticmodelling

99

Where to ValidateWhere to Validate

Small-scalereal-time

simulation

1010

Where to ValidateWhere to Validate

Small-scalereal-time

simulation

NATSNATS

DLRDLR

CENACENA

NLRNLR

EECEEC

real-life environmentreal-life environment

1111

Validation MethodologyValidation Methodology

Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites

H ig h L eve l O b jec tive

L ow L eve l O b jec tive

N u ll H yp o th es is A lte rn a te H yp o th es is

L ow L eve l O b jec tive L ow L eve l O b jec tive

H ig h L eve l O b jec tive H ig h L eve l O b jec tive

TR IA L A IM

1212

Validation MethodologyValidation Methodology

Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites

H ig h L eve l O b jec tive

L ow L eve l O b jec tive

N u ll H yp o th es is A lte rn a te H yp o th es is

L ow L eve l O b jec tive L ow L eve l O b jec tive

H ig h L eve l O b jec tive H ig h L eve l O b jec tive

TR IA L A IM

1313

Validation MethodologyValidation Methodology

Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites

H ig h L eve l O b jec tive

L ow L eve l O b jec tive

N u ll H yp o th es is A lte rn a te H yp o th es is

L ow L eve l O b jec tive L ow L eve l O b jec tive

H ig h L eve l O b jec tive H ig h L eve l O b jec tive

TR IA L A IM

1414

Validation MethodologyValidation Methodology

Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites

H ig h L eve l O b jec tive

L ow L eve l O b jec tive

N u ll H yp o th es is A lte rn a te H yp o th es is

L ow L eve l O b jec tive L ow L eve l O b jec tive

H ig h L eve l O b jec tive H ig h L eve l O b jec tive

TR IA L A IM

1515

Validation MethodologyValidation Methodology

Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites

H ig h L eve l O b jec tive

L ow L eve l O b jec tive

N u ll H yp o th es is A lte rn a te H yp o th es is

L ow L eve l O b jec tive L ow L eve l O b jec tive

H ig h L eve l O b jec tive H ig h L eve l O b jec tive

TR IA L A IM

1616

Validation MethodologyValidation Methodology

Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites

H ig h L eve l O b jec tive

L ow L eve l O b jec tive

N u ll H yp o th es is A lte rn a te H yp o th es is

L ow L eve l O b jec tive L ow L eve l O b jec tive

H ig h L eve l O b jec tive H ig h L eve l O b jec tive

TR IA L A IM

pre-defined level of functionality, operability and performanceof functionality, operability and performance

1717

Measures of MeritMeasures of Merit

MoM’s Chosen to Test HypothesesMoM’s Chosen to Test Hypotheses Controller WorkloadController Workload Airspace CapacityAirspace Capacity Quality of ServiceQuality of Service UsabilityUsability SafetySafety

Measurements Taken:Measurements Taken: e.g. Workload: ISA, TLX, R/T, …..e.g. Workload: ISA, TLX, R/T, ….. Objective; Subjective; ObservedObjective; Subjective; Observed

1818

AnalysisAnalysis

Descriptive StatisticsDescriptive Statistics

ORG 0 ORG 1 ORG 21.5

1.6

1.7

1.8

1.9

2

ORG 0 ORG 1 ORG 2

PC

TC

1919

AnalysisAnalysis

Descriptive StatisticsDescriptive Statistics

Inferential StatisticsInferential Statistics significant at 5% confidence levelsignificant at 5% confidence level

ORG 0 ORG 1 ORG 21.5

1.6

1.7

1.8

1.9

2

ORG 0 ORG 1 ORG 2

PC

TC

2020

AnalysisAnalysis

Descriptive StatisticsDescriptive Statistics

Inferential StatisticsInferential Statistics significant at 5% confidence levelsignificant at 5% confidence level

Backed up by Controller Subjective CommentsBacked up by Controller Subjective Comments ““I can still handle the aircraft while doing the I can still handle the aircraft while doing the

crossword!”crossword!”

ORG 0 ORG 1 ORG 21.5

1.6

1.7

1.8

1.9

2

ORG 0 ORG 1 ORG 2

PC

TC

2121

AnalysisAnalysis

Descriptive StatisticsDescriptive Statistics

Inferential StatisticsInferential Statistics significant at 5% confidence levelsignificant at 5% confidence level

Backed up by Controller Subjective CommentsBacked up by Controller Subjective Comments ““I can still handle the aircraft while doing the crossword!”I can still handle the aircraft while doing the crossword!”

desired level of confidencedesired level of confidence

ORG 0 ORG 1 ORG 21.5

1.6

1.7

1.8

1.9

2

ORG 0 ORG 1 ORG 2

PC

TC

2222

Effectiveness of PHARE ValidationEffectiveness of PHARE Validation

Core Validation Group with Representation from all Core Validation Group with Representation from all Participating SitesParticipating Sites

Common Methodology CrucialCommon Methodology Crucial Methodology Improved with ExperienceMethodology Improved with Experience Validation Conducted by Individual SitesValidation Conducted by Individual Sites

Lessons Re-learnt &/or LostLessons Re-learnt &/or Lost Effectiveness DiminishedEffectiveness Diminished

Validation Should be Applied Throughout Project - Validation Should be Applied Throughout Project - Not Just at the EndNot Just at the End

2323

SummarySummary

Validation Methodology Validation Methodology Major Contributor to Major Contributor to Credibility of PHARECredibility of PHARE

Forms the Backbone for Forms the Backbone for much of the Validation much of the Validation Activities within Activities within European ATM R&DEuropean ATM R&D

2424

SummarySummary

Validation Methodology Validation Methodology Major Contributor to Major Contributor to Credibility of PHARECredibility of PHARE

Forms the Backbone for Forms the Backbone for much of the Validation much of the Validation Activities within Activities within European ATM R&DEuropean ATM R&D