Standardized Anomaly Model Output Statistics Over Complex ...May 21, 2018 · Ensemble Postprocessing-10 0 10 20 30-10 0 10 20 30 ensemble mean (x) observations (y) IBK T 12 +5d l

Standardized Anomaly Model OutputStatistics Over Complex Terrain

[email protected]

[email protected]

Outline

• statistical ensemble postprocessing

• introduction to SAMOS

• new snow amount forecasts in Tyrol

• sub-seasonal anomaly prediction over the U.S.

NOAA/PSD Seminar Talk – May 21, 2018 1

Ensemble Postprocessing

Numerical Weather Prediction

1 analysis: → current state

2 forecast: → future state

Error Sources

• observations

• simplified model world

• numerical approximation

• “unknown” atmospheric processes

Ensemble Prediction Systems

• to quantify the uncertainty

• number of members restricted

• typically underdispersive



Forecast Error

• total error = noise + systematic errors

• noise: unsystematic error

• systematic errors: correction possible


• correct bias

• correct uncertainty

• discrete→ full distribution

• probabilities, quantiles, extremes



−10 0 10 20 30

−10

010

2030

ensemble mean (x)

obse

rvat

ions

(y)

IBK T12 +5d

●

Non-homogeneous Gaussian Regression (NGR, EMOS)

y ∼ N(µ, σ

)µ = β0 + β1 · x̄

log(σ) = γ0 + γ1 · log(sx)



−10 0 10 20 30

−10

010

2030

ensemble mean (x)

obse

rvat

ions

(y)

IBK T12 +5d


y ∼ N(µ, σ

)µ = β0 + β1 · x̄




−10 0 10 20 30

−10

010

2030

ensemble mean (x)

obse

rvat

ions

(y)

IBK T12 +5d


y ∼ N(µ, σ

)µ = β0 + β1 · x̄



Spatial Postprocessing

●

●

●

●

●

●

● ●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

● ●

●●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●●

●

●●

●

●

●

●●

● ●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●●

●●

●

●

●

●

●●

??

??

??

?

? ?

??

?

?

??

−4 0 2 4 6 8

dens

ity

−4 0 2 4 6 8

dens

ity

NGR

• allows station-wise corrections

• not suitable for spatial predictions

Alternative approach required.


SAMOS

What is SAMOS?


SAMOS

What is SAMOS?

“A probabilistic spatio-temporal ensemblepostprocessing method using climatological backgroundinformation to remove site specific characteristics, whichallows to estimate one simple regression model for allstations and forecast lead times at once.”


SAMOS

Reminder: NGR

y ∼ N(µ, σ

)µ = β0 + β1 · x̄


NGR → SAMOS

Transform all quantities (y, x) into standardized anomalies:

y∗ =y− µ̃yσ̃y

, x∗ =x− µ̃xσ̃x

y∗, x∗: standardized anomalies

µ̃•, σ̃•: climatological properties of y, x


SAMOS

Gaussian SAMOS

y∗ ∼ N(µ∗, σ∗

)µ∗ = β0 + β1 · x̄∗

log(σ∗) = γ0 + γ1 · log(s∗x)

NGR → SAMOS







SAMOS

Gaussian SAMOS

y∗ ∼ N(µ∗, σ∗

)µ∗ = β0 + β1 · x̄∗

log(σ∗) = γ0 + γ1 · log(s∗x)

NGR → SAMOS







SAMOS

−10

010

2030

Station A (mean = 3.9°C)

tem

pera

ture

[°C

]

climatological meanclimatological 90% interval

Jan

Feb

Mar

Apr

May Jun

Jul

Aug

Sep Oct

Nov

Dec

Station B (mean = 13.4°C)

Jan

Feb

Mar

Apr

May Jun

Jul

Aug

Sep Oct

Nov

Dec

2040

6080

tem

pera

ture

[°F

]

Convert y to y∗ using y∗ = y−µ̃yσ̃y

.


SAMOS

−2

−1

01

2

Station A

stan

dard

ized

ano

mal

ies

[°C

]

Jan

Feb

Mar

Apr

May Jun

Jul

Aug

Sep Oct

Nov

Dec

Station B

Jan

Feb

Mar

Apr

May Jun

Jul

Aug

Sep Oct

Nov

Dec

Convert y to y∗ using y∗ = y−µ̃yσ̃y

.


SAMOS Summary

Spatio-Temporal Climatologies

• Account for ...

• seasonal and diurnal patterns,

• spatial differences (longitude, latitude, altitude),

• and possible interactions.

Standardized Anomalies

• location or station independent

• independent from season and time


SAMOS Summary

●

●

●

●●

●

● ●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●● ● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

● ●●●

●

●

●

●

● ●

●

●

●

●

●●

●

●

●●●

●

● ●

●

●

●●

●● ●

●

●

●

●●

●

●

●

●●

●

●

●

●

● ●

●●

●

●

●

●●

● ●●

●

●

●

●●

●

●

●

●

●

●

● ●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

● ●

●●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●●

●

●●

●

●

●

●●

● ●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●●

●●

●

●

●

●

●●

−4 0 2 4 6 8

dens

ity

−4 0 2 4 6 8

dens

ity

On the Anomaly Scale

• combine data from all stations and lead times

• estimate one “simple” model for the area of interest

• correct current ensemble forecast→ µ∗, σ∗

• de-standardize:

µ = µ∗ · σ̃y + µ̃y

σ = σ∗ · σ̃y


Hourly Probabilistic Snow ForecastsOver Complex TerrainA Hybrid Ensemble Postprocessing Approach

R Stauffer, GJ Mayr, JW Messner, A Zeileis

Snow Forecasts

8 9 10 11 12 13 14

46.5

47.0

47.5

48.0

Topography, Tyrol and Surrounding

1000

2000

3000

4000

●●

ATCH

DE

IT

Problem: lack of reliable fresh snow observations.


Snow Forecasts

Idea: Hybrid Approach

• forecast temperature & precipitation instead

• SAMOS• hourly temperature forecasts• daily precipitation forecasts

• ensemble copula coupling• restore spatio-temporal structure• convert temperature & precipitation into snow


Snow Forecasts

10 11 12 13

46.6

47.0

47.4

47.8

Station Network

●

●●

●

●

●

●●

●

●

●●

●●

●●

●

●

●

●

●

●

●

●

●

● ● ●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

● ●

●●

●

●

●

● TAWESEHYDAirport

Height Distr.

0 10 20 30

300−600

600−900

900−1200

1200−1500

1500−1800

1800−2100

2100−2400

2400−2700

2700−3000

3000−3300

3300−3600 TAWESEHYD

Observation Data

• hourly 2 m air temperature: 90 stations, up to 10+ years

• daily precipitation sums: 110 stations, up to 40+ years


Snow Forecasts

Numerical Weather Forecast Data

ECMWF hindcast

• 10 + 1 member ensemble

• 6-hourly output

• initialized twice a week (0000 UTC, 20 years)

ECMWF ensemble

• 50 + 1 member ensemble

• hourly output

• initialized 0000 UTC


Snow Forecasts

SAMOS Model AssumptionsTemperature

T∗ ∼ N (µ∗, σ∗)

µ∗ = β0 + β1 · x̄∗

log(σ∗) = γ0 + γ1 · log(x∗)

x∗: std. anomalies of the 2m temperature.

Precipitation

precipp∗ ∼ Lcens(µ∗, σ∗)µ∗ = β0 + β1 · x̄∗ · (1− z) + β2 · z

log(σ∗) = γ0 + γ1 · log(s∗x) · (1− z)

x∗: std. anomalies of power-transformed total precipitation.

z: split-variable (binary) to handle unanimous predictions.


Snow Forecasts

(1) Observations:

standardized anomalies

station observations

spatio-temporal climatology

(2) Numerical Weather Forecasts:

standardized anomalies

latest 8 ECMWF hindcast runs

model climatology

(3) Estimate SAMOS models:

estimate SAMOS regression coefficients

training data set (standardized anomalies)

(4) Prediction:

spatial probabilistic forecasts

standardized anomalies of latest EPS run

compute corrected SAMOS predictions


Snow Forecasts

Temperature Forecasts

−3

−2

−1

0

1

2

3C

LIM

raw

EP

S

EM

OS

SA

MO

S_h

om

xSA

MO

S_h

et

BIA

S [

° C]

2

4

6

8

10

12

CLI

M

raw

EP

S

EM

OS

SA

MO

S_h

om

xSA

MO

S_h

et

80%

PI [

° C

]

−100

−50

0

50

CLI

M

raw

EP

S

EM

OS

SA

MO

S_h

om

xSA

MO

S_h

et

CR

PS

S [%

]

better

worse

Verification: Dec 2016 - mid April 2017.


Snow Forecasts

Temperature Forecasts

●

●

CLI

M

raw

EP

S

EM

OS

SA

MO

S_h

om

xSA

MO

S_h

et

−3

−2

−1

0

1

2

3● ●

●

BIA

S [

° C]

●

●

●

●

●

●●

CLI

M

raw

EP

S

EM

OS

SA

MO

S_h

om

xSA

MO

S_h

et

2

4

6

8

10

12●

80%

PI [

° C

]

CLI

M

raw

EP

S

EM

OS

SA

MO

S_h

om

xSA

MO

S_h

et

−100

−50

0

50

● ●

CR

PS

S [%

]

better

worse

SAMOS

• Slightly outperformed by EMOS, ...

• but allows for spatial predictions.

• Ensemble spread: barely any additional information.


Snow Forecasts

24 h Precipitation Forecasts

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

CLI

M

raw

EP

S

SA

MO

S_h

om

SA

MO

S_h

et

−4

−2

0

2

4

BIA

S [m

m]

●●

●

●

●●

CLI

M

raw

EP

S

SA

MO

S_h

om

SA

MO

S_h

et

1

2

3

4

CR

PS

[mm

]●

●

●

●●

●●

CLI

M

raw

EP

S

SA

MO

S_h

om

SA

MO

S_h

et

0.1

0.2

0.3

0.4

0.5

0.6

BS

for

0 m

m

SAMOS

• Outperforms raw EPS, less skillful than for temperature.

• Ensemble variance barely any additional information.


Snow Forecasts

How to Get Hourly Snow Predictions?

• combine temperature/precipitation using ECC

• temperature:• draw 51 member ensemble from corrected hourly N• restore rank order structure

• 24 h precipitation:• draw 51 member ensemble from corrected L0

• restore rank order structure

• hourly precipitation:• re-weight raw hourly ensemble forecasts using:

ωms =t̂pcopula,ms

tpEPS,ms


Snow Forecasts

−10

05

1015

Index

2m te

mpe

ratu

re [°

C]

Forecast March 08, 2017 00UTCStation HOLZGAU (11315)

1.2

01

23

4

prec

ipita

tion

[mm

h−1

] raw ENSobservation

6 12 18 24 30 36 42 48 54 60 66 72 78forecast step [h]


Snow Forecasts

−10

05

1015

Index

2m te

mpe

ratu

re [°

C]


1.2

01

23

4

prec

ipita

tion

[mm

h−1

] raw ENSobservationSAMOS/ECC

6 12 18 24 30 36 42 48 54 60 66 72 78forecast step [h]


Snow Forecasts

Hourly Snow Forecasts

PIms =

“dry” if:

precipitationms ≤ 0.05 mmh

“rain” if:

precipitationms > 0.05 mmh ∧ T2m,ms > 1.2◦ C

“snow” if:

precipitationms > 0.05 mmh ∧ T2m,ms ≤ 1.2◦ C

PI: precipitation type indicator {dry, rain, snow}

m/s: copula member and forecast step


Snow Forecasts

Index

prob

abili

ty [%

]

2040

6080

precipitationrainsnow


020

4060

fres

h sn

ow [c

m]

prec

ipita

tion

[mm

]

32.837.947.2

2.57.814.7

precipitation (88.5%)precipitation (50.0%)precipitation medianfresh snow (88.5%)fresh snow (50.0%)fresh snow median

6 12 18 24 30 36 42 48 54 60 66 72 78forecast step [h]


Snow Forecasts

10.5°E 11°E 11.5°E 12°E 12.5°E

46.8

°N47

°N47

.2°N

47.4

°N47

.6°N

Expectation for March 10, 2017 00:00 UTC (+48h)ex

pect

atio

n

0mm 0.5mm 1mm 1.5mm 2mm 2.5mm 3mm 3.5mm 4mm


Snow Forecasts

predicted probability

obse

rved

freq

uenc

y

0.2

0.4

0.6

0.8

obse

rved

freq

uenc

y

0.2 0.4 0.6 0.8

forecasted probability

Precipitation (solid+liquid)

●

●●

●

●●

●

●

●

●

●

● ●

●

●

●

● ●

●

●

00.

20.

40.

60.

8 102000400060008000

raw EPS0

0.2

0.4

0.6

0.8 10

2000400060008000

ECC−Q

Brier Scoreraw EPS: 0.109ECC−Q: 0.077


obse

rved

freq

uenc

y

0.2 0.4 0.6 0.8


Rain (liquid)

●

●●

●

●

●

●

●

●●

●

●●

●

●

●

● ●●

●

00.

20.

40.

60.

8 102000400060008000

raw EPS

00.

20.

40.

60.

8 102000400060008000

ECC−Q



obse

rved

freq

uenc

y

0.2

0.4

0.6

0.8

obse

rved

freq

uenc

y

0.2 0.4 0.6 0.8


Snow (solid)

● ●● ●

● ● ●

● ●

●

●

●

●●

●

● ●

●●

●

00.

20.

40.

60.

8 102000400060008000

raw EPS

00.

20.

40.

60.

8 102000400060008000

ECC−Q


Empirical frequencies:precipitation 15.8 %, rain 9.8 %, snow 7.5 %.


Snow Forecasts Summary

• ensemble spread: barely any information

• large spread for postprocessed temperature

• improvements: more pronounced for temperature

• combine different data sources

• use data sets with different temporal resolution

• reliable hourly probabilistic spatial snow forecasts


Sub-Seasonal Climate Forecast RodeoImprove Existing Sub-Seasonal Forecasts, Bureau of Reclamation


The Rodeo

The Challenge

• predict anomalies

• mean temperature and accumulated precipitation

• week 3-4 and 5-6

Area of Interest (1°x1° grid)

N = 514


The Rodeo

The Price

• 800 000 USD in total

• iff. anomaly forecasts are significantly better than:

a damped persistence

the CFSv2 itself

Area of Interest (1°x1° grid)

N = 514


The Rodeo

The Truth

• Climate Prediction Center’s gridded data set

• gridded gauge data set

• gridded temperature data set

• climatology: 2-week mean 1981–2010

The Forecasts

• 4 member CFSv2

• mean over 8 runs (8× 4 = 32 member mean)

The Measure

ACC =

∑(f ′ · o′

)√∑o′2 ·

∑f ′2


The Rodeo

The Idea

• CPC: provides observation climatology (µ̃y, σ̃y)

• CVSv2 reforecasts: provide ensemble climatology (µ̃x, σ̃x)

• apply “complex” homoscedastic AMOS/SAMOS

Model Assumption

y∗ ∼ D(µ∗, σ∗

)Where µ∗ may include:

• forecasted anomalies (2 m temperature, dewpoint, . . . )

• spatial effects f(long, lat)

• teleconnection indices (NAO, NA, PNA; CPC)

• snow cover data (NSIDC)


The Rodeo

ECMWF Based Approach

• temperature (week 3-4)

• Gaussian AMOS

• 70 covariates

• optimization:

based on R package bamlss

likelihood-based gradient boosting

variable selection & parameter estimation


The Rodeo

x

Observed

xy

RAW EPS (0.06)

x

y

Corrected (0.56)

−4 −2 0 2 4

2m mean temperature anomaly (week 3−4) in degrees Celsius

x

Observed

x

y

RAW EPS (0.46)

x

y

Corrected (0.72)

−4 −2 0 2 4

2m mean temperature anomaly (week 3−4) in degrees Celsius


The Rodeo Leader Board


The Rodeo

●●●●

●

●●

●

●●●●●

●●●●●●●●●

●

●●●●●●●●●●●

●●●●

in−sample out−of−sample

−1.

0−

0.5

0.0

0.5

1.0

Sub−seasonal RodeoMean Temperature Week 3−4

AC

C

mean: 0.77 mean: 0.38


References I

Gneiting, T, AE Raftery, AH Westveld, and T Goldman, 2005: Calibrated

Probabilistic Forecasting Using Ensemble Model Output Statistics and

Minimum CRPS Estimation. Monthly Weather Review, 133, 1098–1118,

doi:10.1175/MWR2904.1.

Dabernig, M, GJ Mayr, JW Messner, and A Zeileis, 2017: Spatial Ensemble

Post-Processing with Standardized Anomalies. Quarterly Journal of the

Royal Meteorological Society, 143, 909–916, doi:10.1002/qj.2975.

Stauffer, R, N Umlauf, JW Messner, GJ Mayr, and A Zeileis, 2017: Ensemble

Postprocessing of Daily Precipitation Sums over Complex Terrain Using

Censored High-Resolution Standardized Anomalies. Monthly Weather

Review, 145, 955–969, 10.1175/MWR-D-16-0260.1.

Dabernig, M, GJ Mayr, JW Messner, and A Zeileis, 2017: Simultaneous

Ensemble Postprocessing for Multiple Lead Times with Standardized

Anomalies. Monthly Weather Review, 145, 2523–2531,

10.1175/MWR-D-16-0413.1.


References II

Gebetsberger, M, JW Messner, GJ Mayr, and A Zeileis, 2016: Tricks for

Improving Non-Homogeneous Regression for Probabilistic Precipitation

Forecasts: Perfect Predictions, Heavy Tails, and Link Functions. Working

Papers, Faculty of Economics and Statistics, University of Innsbruck.

Stauffer R, GJ Mayr, JW Messner, and A Zeileis, 2018: Hourly Probabilistic

Snow Forecasts over Complex Terrain: A Hybrid Ensemble Postprocessing

Approach. Working papers, Faculty of Economics and Statistics, University

of Innsbruck.


Thank you for your attentionand special thanks to Tom for the invitation!

Appendix

SAMOS

Climatological Location

µ~y

●●

−2.0

µ~x

●●

−9.6

−14 −12 −10 −8 −6 −4 −2

2m air temperature, January 1, 0000 UTC (+24h forecast). Unit: °Celsius

Climatological Scale

σ~y

●●

4.6

σ~x

●●

5.2

4.5 5 5.5 6


Snow Forecast: Hourly Calibration

Hourly 2 Meter Temperature, raw EPS

Den

sity

0.00

0.01

0.02

0.03

0.04

0.05

1 7 13 19 25 31 37 43 49rank

(a)Hourly 2 Meter Temperature, ECC−Q

Den

sity

0.00

0.01

0.02

0.03

0.04

0.05

1 7 13 19 25 31 37 43 49rank

(b)

Hourly Precipitation Sums, raw EPS

Den

sity

0.00

0.02

0.04

0.06

0.08

0.10

1 7 13 19 25 31 37 43 49rank

(c)Hourly Precipitation Sums, ECC−Q

Den

sity

0.00

0.02

0.04

0.06

0.08

0.10

1 7 13 19 25 31 37 43 49rank

(d)


Snow Forecast: Hourly Verification

●

●● ●

●● ●

●

●●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●●● ●

●

●● ● ●●

●●

●●

●●

●●●

●

●

●●

●

●●●

●

●●●

●

●●●

●

●●

● ●

●●●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●●

●

●

●●● ●

●

●●●

●● ●●●●

●●

●●●●

●●●●

●

●●●

●

●●

●●

●●

●

●

●

●●

●

●

●

●●

●

●

●

●●●

●●●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●●● ●

●

●● ●

●● ●●● ●●●●

●●●●

●●●● ●●

●

● ●

●●

●

●

●●●

●

●

●●●

●

●

●●●●

−50

0

50

CR

PS

S [%

] for

hour

ly te

mpe

ratu

reStationwise Mean CRPS Skill Scores for Hourly Copula (raw EPS as Reference)

(a)

●

●

●

●

● ●

●

●

●

●

● ●●

●

●

●

●

●

●

●●●

●●●●

●●

●

●

●

●●●●●●●●

●

●

●

●●●●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●●

●

●●●

●

●

●●

●●●

●●

●

●

●

●

●

●●

●●

●●

●

●

●●

●

●

●

●

●

●●

●

●

●●●●●

●

●

●

●

●

●

●

●

●●●

●

●

●●

●●

●

●

●

●

●●● ●● ●

●

●●

●●●●

●

●

●

●

●

●

●

●

●

●●●

●

●●

●

●●●

●

●●

●

●

●

●●

●

07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78

−100

−50

0

50

forecast step or lead time (hours)

CR

PS

S [%

] for

hour

ly p

reci

pita

tion

sum

s

(b)


Documents

Standardized Anomaly Model Output Statistics Over Complex ...May 21, 2018 · Ensemble Postprocessing-10 0 10 20 30-10 0 10 20 30 ensemble mean (x) observations (y) IBK T 12 +5d l