5280 5340 5400 5460

5520

5580

5640

5700

5760

5820

5820

Mean Geoptential for Cluster 4

Development of an object-oriented verification technique

for QPFMichael Baldwin1

Matthew Wandishin2, S. Lakshmivarahan3

1 Cooperative Institute for Mesoscale Meteorological Studies,University of Oklahoma

2 Institute for Atmospheric Physics, University of Arizona3 School of Computer Science, University of Oklahoma

5280 5340 5400 5460

5520

5580

5640

5700

5760

5820

5820

Mean Geoptential for Cluster 4

Traditional verificationCompare a collection of matching pairs of forecast and observed values at the same set of points in space/timeOne “score” might end up representing the accuracy of millions of points, thousands of cases, hundreds of meteorological eventsBoiling down that much information into a couple of numbers is not very meaningful

5280 5340 5400 5460

5520

5580

5640

5700

5760

5820

5820

Mean Geoptential for Cluster 4

Forecast #1: smooth

OBSERVED

FCST #1: smooth

FCST #2: detailed

OBSERVED

5280 5340 5400 5460

5520

5580

5640

5700

5760

5820

5820

Mean Geoptential for Cluster 4

“Measures-oriented” (Brooks and Doswell, 1996) approach to verifying these forecasts

Verification Measure Forecast #1 (smooth)

Forecast #2 (detailed)

Mean absolute error 0.157 0.159

RMS error 0.254 0.309

Bias 0.98 0.98

Threat score (>0.45) 0.214 0.161

Equitable threat score (>0.45)

0.170 0.102

n

kkk xf

nMAE

1

1

xfBIAS

HOFHTS

)( ChHOF

ChHETS

5280 5340 5400 5460

5520

5580

5640

5700

5760

5820

5820

Mean Geoptential for Cluster 4

Characterize the forecast and observed fields

Verify the forecast with a similar approach that a human forecaster would use to visualize the forecast/observed fieldsCharacterize features, phenomena, events, etc. found in forecast and observed fields by assigning attributes to each object

5280 5340 5400 5460

5520

5580

5640

5700

5760

5820

5820

Mean Geoptential for Cluster 4

Object-oriented approach to verification

Decompose fields into sets of objects that can be objectively identified and described by attributesUse image processing and data mining techniques to locate and classify eventsProduce scores based upon the similarity/dissimilarity between forecast and observed objectsAnalyze joint distribution of forecast and observed objectsSimilar to Neilley (1993)

5280 5340 5400 5460

5520

5580

5640

5700

5760

5820

5820

Mean Geoptential for Cluster 4

Possible scores produced by this approach

f = (af, bf, cf, …, xf, yf )o = (ao, bo, co, …, xo, yo)

score = function( f , o )d ( f , o ) = ( f - o )t A ( f - o )

Generalized Euclidean distance, measure of dissimilarityA is a weight matrix, different attributes would probably have

different weightsc ( f , o ) = cov ( f , o )

Covariance, measure of similarity

f

o

d

5280 5340 5400 5460

5520

5580

5640

5700

5760

5820

5820

Mean Geoptential for Cluster 4

Characterization: How?Locate an eventCould use image processing edge detection routines

Event #16

5280 5340 5400 5460

5520

5580

5640

5700

5760

5820

5820

Mean Geoptential for Cluster 4

Characterization: How?Assign attributesExamples: location, mean, variance, structure

Event #16x=37.3,y=87.8,=2.8

5280 5340 5400 5460

5520

5580

5640

5700

5760

5820

5820

Mean Geoptential for Cluster 4

Multiscale statistical properties (Harris et al 2001)Fourier power spectrumGeneralized structure function: spatial correlationMoment-scale analysis: intermittency of a field, sparseness of sharp intensitiesLooking for “power law”, much like in atmospheric turbulence (–5/3 slope)

FIG. 3. Isotropic spatial Fourier power spectral density (PSD) for forecast RLW (qr; dotted line) and radar-observed qr (solid line). Comparison of the spectra shows reasonable agreement at scales larger than 15 km. For scales smaller than 15 km, the forecast shows a rapid falloff in variability in comparison with the radar. The estimated spectral slope with fit uncertainty is = 3.0 ± 0.1

attribute?

5280 5340 5400 5460

5520

5580

5640

5700

5760

5820

5820

Mean Geoptential for Cluster 4

Fourier power spectraCompare 3h accumulated QPF to radar/gage analysesForecasts were linearly interpolated to same 4km grid as “Stage IV” analysisErrico (1985) Fourier analysis code used. 2-d Fourier transform converted to 1-d by annular average Fixed grid used for analysis located away from complex terrain of Western U.S.Want to focus on features generated by model physics and dynamics, free from influence of orographically forced circulations

5280 5340 5400 5460

5520

5580

5640

5700

5760

5820

5820

Mean Geoptential for Cluster 4

Example

log[

E(k)

]

log[wavenumber]

Obs_4 Eta_12 Eta_8

WRF_22 WRF_10 KF_22

5280 5340 5400 5460

5520

5580

5640

5700

5760

5820

5820

Mean Geoptential for Cluster 4

June 2002 00z runs 12, 24, 36, 48h fcsts

5280 5340 5400 5460

5520

5580

5640

5700

5760

5820

5820

Mean Geoptential for Cluster 4

SummaryDeveloping an “object-oriented” verification approach by characterizing forecasts and observationsExamining use of spatial structure and variability as potential attributesProvides information on realism of forecasts that traditional QPF verification measures do notWorking with forecasters/users to determine useful attributes for characterizing events