Designing a DTC Verification System Jennifer Mahoney NOAA/ESRL 21 Feb 2007

Designing a DTC Verification System

Jennifer MahoneyNOAA/ESRL21 Feb 2007

Talk Overview

• Define verification system

• Present the key components that will form the DTC verification system framework

Allow flexibility

Serve a variety of users

• Discuss the complexities of developing the DTC verification system

Acknowledgments

• ESRL/GSD Verification System Engineering TeamSean MadineNick MathesonMissy PettyDan Schaffer

What is a Verification System?

What is a DTC Verification System?

Verification Analysesas a Function of Software Characteristics

DTC Verification Analysis Activities

Software Characteristics

Non-Functional Requirements

• Important Operational Considerations Stability Reliability Security

• Important Development Considerations Support for collaborative interaction Adaptability to other environments Extensibility to other scientific problems Maintainability Portability to other organizations

of the ‘System’

Verification Mechanics

Aggregate/Combine to

Produce SpecificMetrics

Pre-processForecasts

Pre-processObservations

Create Displays

Verification System Components- Historical Perspective

• Good solution for a small set well defined questions

• System structure remains consistent, but can expand forecast dataset

• Solution needs to change when want to add observation datasets, new verification mechanics and assessment techniques, integrate non-meteorological information, and inter-compare a variety of forecast systems

Verification System Components:Data Management Structuring System

Data Warehouse

Finest resolutionForecast/observation pairs+ forecast/obs attributes

Verification System Components:Data Management Structuring System

Data Warehouse

Finest resolutionForecast/observation pairs+ forecast/obs attributes

Data Marts

Verification System Components:Data Management Structuring System

Data Warehouse

Finest resolutionForecast/observation pairs+ forecast/obs attributes

Data Marts

IntercompareForecastSystemsProduceStatisticalInformation

Aggregation driven by specific data relationships

Verification Mechanics

Pre-processForecasts

Pre-processObservations

Displays

DTC Verification System Components

Store FinestGranularityVerificationElements

Apply storageOptimization Strategy Database info

Query interrogationto compute score

Verification System Complexities

• Management of complex interdependent relationships between datasetsEvent-driven component initiation

• Eliminate unnecessary time delays between data execution steps• Handle abnormal delays in data access, processing, and user

access• Account for differences between operational deployments

Multi workflow requests• Incorporate relationships with the addition of new datasets,

verification methodologies, and forecast intercomparisons• Provide only meaningful workflow comparisons (i.e., par down list

of all possible choices to meet user analysis questions)

Verification System Complexities

• Effective access to the resultsProvide tiered optimizations

• Fast turn around of pre-defined set of analyses• Flexibility to define varying set of analyses• Support wide-range of user needs

Allow for wide variety of graphical and standard displays

• Traditional statistical displays• Displays that allow the integration of verification information

Summary

• Verification system will mainly support the forecast intercomparison evaluation (e.g., core tests) function of the DTC

• Scientific conceptualism developed for the toolkit will feed into the DTC verification system

• DTC requirements for the verification system require complex data management and interrogation strategies

• Tiered approaches for data access are required to provide maximum system flexibility to end users

Future Efforts

• Gather verification system requirements from the workshop

• Develop a Functional Requirements document• Begin preliminary development of the system

concepts by the end of the year

Contributors to the DTC System Development

• NCAR – scientific verification concepts and toolkit capabilities

• NCEP – bridge from the current NCEP verification system functionality to the DTC verification system framework

• GSD – verification system framework by extending database management and web access strategies

• DTC users – functional requirements for the verification system

Questions?