Upload
zandra
View
32
Download
0
Embed Size (px)
DESCRIPTION
AUTOMATED VALIDATION FOR SUMMARY OF THE DAY TEMPERATURE DATA. Authors: William E. Angel, Michael L. Urzen, Stephen A. Del Greco, and Matthew W. Bodosky Special Thanks to Arthur G. DeCotiis. COMMON ACROYMNS. ASOS – Automated Surface Observing System AWOS – Automated Weather Observing System - PowerPoint PPT Presentation
Citation preview
September 23, 2003 National Climatic Data Center
AUTOMATED VALIDATION FOR
SUMMARY OF THE DAY TEMPERATURE DATA
Authors: William E. Angel, Michael L. Urzen,
Stephen A. Del Greco, and Matthew W. Bodosky
Special Thanks to Arthur G. DeCotiis
September 23, 2003 National Climatic Data Center
COMMON ACROYMNS
ASOS – Automated Surface Observing SystemAWOS – Automated Weather Observing SystemCOOP – Cooperative Observers NetworkGEA – Graphical Edit AnalysisGMT – Generic Mapping ToolsNEWAE – NEW Areal EditTEMPVAL – Automated Temperature Validation
System
September 23, 2003 National Climatic Data Center
COOPERATIVE DATA – WHAT IS IT?
COOP OBSERVERAM, PM, MID READERS
September 23, 2003 National Climatic Data Center
~8500 Stations
September 23, 2003 National Climatic Data Center
NWS
NCDC
SIMPLIFIED COOPERATIVE DATA FLOW
COOP Forms
Data Entry
Monthly QC
Archive
Publication
Internet
COOP Observer
September 23, 2003 National Climatic Data Center
COOPERATIVE DATA PROCESSING (TEMPVAL Plug In)
RecordsTranscription
Auto Aerial Edit
(NEWAE)
InteractiveValidation
(GEA)Last Look
TEMPVAL
September 23, 2003 National Climatic Data Center
PRE-TEMPVAL OPERATIONAL OVERVIEW
Validation Based on Cooperative NeighborsApproximately 70% Automated, 30% ManualRules Based – AutomatedUnresolved Data – Manual Review by Validator
Compared Daily Departures From Station Monthly Mean Temperature3-Sigma Mean Value
September 23, 2003 National Climatic Data Center
PRE-TEMPVAL OPERATIONAL OVERVIEW
Looked For Shifter SignatureObservation Day Versus Calendar DayCommon With AM Readers
Detected FlatlinersRepeating Same Temperature 3 Or More Days In A Row
September 23, 2003 National Climatic Data Center
September 23, 2003 National Climatic Data Center
September 23, 2003 National Climatic Data Center
WHY TEMPVAL?Improve Validation Accuracy
Change in Methodology– Use of Grid Fields Instead of COOP Neighbors
High Quality of “Ground Truth” DataEliminate Validator Subjectivity
Make Edits Uniform and Consistent Reduce Manual Effort And Production Time
Reason For AutomationPrevious System In Use Since Early 1990’s
Time For An Update
September 23, 2003 National Climatic Data Center
“GROUND TRUTH” DATA
Use Of Approximately 900 ASOS SitesUse Of Approximately 450 AWOS SitesUse Of Approximately 300 Canadian ASOS SitesData Is Gridded On A ½ Degree Resolution Using
GMT SoftwareInverse Distance Squared Interpolation
Data Undergoes A “Bull’s Eye” Grid TestSlope Test Comparing A Grid Point Intersection Value
With Surrounding Grid Point Values
September 23, 2003 National Climatic Data Center
September 23, 2003 National Climatic Data Center
September 23, 2003 National Climatic Data Center
September 23, 2003 National Climatic Data Center
September 23, 2003 National Climatic Data Center
September 23, 2003 National Climatic Data Center
September 23, 2003 National Climatic Data Center
HOW DOES TEMPVAL WORK?
Computes Daily Max/Min Departure From Monthly Mean Max/Min (same as previous system)
Builds 1/2 Degree Grid Fields By Reader Time From Quality Controlled ASOS/AWOS Hourly And Summary Of Day Data
Compares Cooperative Daily Departure With ASOS/AWOS Grid Departures
Generates Edits By Adding Grid Departure To The Cooperative Station Monthly Mean Max/Min
September 23, 2003 National Climatic Data Center
TEMPVAL FLOWCHARTBuild ASOS Daily Max/Min
Departure Grid Analyses (AM,PM,Mid)
Test Cooperative Max/Min Departure Against Grid
Value
Validates/Not Missing?|OV - grid| < 7 + grad
Apply Grid Departure To Station Mean Max/Min
Detect and Remove Bull’s Eyes
Edit OVContinue
End
No
Yes
Note: stations on “Exception List” bypass this process
Note: OV represents the coop max/min departure value.
September 23, 2003 National Climatic Data Center
PARALLEL TESTING RESULTS
Tested for |COOP - GRID| >= 7.0 + Gradient TEMPVAL Generated 12,904 (8.63%) EditsTEMPVAL Generated 6060 (4.05%) Temperatures
For Missing Observed TemperaturesManually Validated 100 Outlier Cases (Operational
Aerial Edit versus TEMPVAL)TEMPVAL Results Are More Realistic Than
Operational Results In 90% Of Cases
September 23, 2003 National Climatic Data Center
GEA vs ASOS VALIDATION6 Nov 2001 R2 = 0.9665y = 0.986x + 0.4098
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
GEA Values
ASOS Values
September 23, 2003 National Climatic Data Center
GEA vs ASOS VALIDATION15 DEC 2000
R2 = 0.9547y = 0.9696x + 1.0937
-20
-10
0
10
20
30
40
50
60
70
80
90
-20 -10 0 10 20 30 40 50 60 70 80 90
GEA Values
ASOS Values
September 23, 2003 National Climatic Data Center
TempVal Edit DistributionNov 2001
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
-10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10
Difference of the TempVal Value and the Observed Value
Skewness = 1.279
September 23, 2003 National Climatic Data Center
SHORTFALLS OF TEMPVALLimited In Capability To Effectively Interpret Particular
Types Of Weather, Such AsWarm Air Advection Episodes For High Elevation
StationsInversionsStrong Frontal Passages“Backdoor Cold Fronts”
Lack Of Sufficient ASOS/AWOS Data Does Not Allow TempVal To Be Performed In Puerto Rico, Virgin Islands, Or The Remaining Territories
September 23, 2003 National Climatic Data Center
September 23, 2003 National Climatic Data Center
September 23, 2003 National Climatic Data Center
BENEFITS OF TEMPVAL
Fully Automated “Plug-In” ModuleEdits Are Uniform and SystematicGenerate Daily Replacement Temperature ValuesRe-Analysis Is PossibleAdditional Algorithms Easily Added
Currently Testing A “Time Shifter” AlgorithmRetain Original Data
Database Contains Original And Edited Values– Same As Previous System
September 23, 2003 National Climatic Data Center
CONCLUSIONS
ASOS/AWOS Station Density And Data Quality Are Sufficient To Produce Departure Grid Fields
Parallel Testing Shows That TEMPVAL Outperforms The Previous System
Eliminates Manual Processing And Removes Associated Validator Subjectivity
TEMPVAL Been Operational Since The January 2002 Data MonthCurrently Processing November 2002 Data
September 23, 2003 National Climatic Data Center
QUESTIONS - COMMENTS
THANKS!