Verification Continued… Holly C. Hartmann

Verification Continued…

Holly C. Hartmann

Department of Hydrology and Water ResourcesUniversity of Arizona

hollyoregon@juno.com

RFC Verification Workshop, 08/14/2007

1. Introduction to Verification- Applications, Rationale, Basic Concepts- Data Visualization and Exploration- Deterministic Scalar measures

2. Categorical measures – KEVIN WERNER- Deterministic Forecasts- Ensemble Forecasts

3. Diagnostic Verification- Reliability- Discrimination- Conditioning/Structuring Analyses

4. Lab Session/Group Exercise- Developing Verification Strategies- Connecting to Forecast Operations and Users

Agenda

Probabilistic Ensemble Forecasts

From: California-Nevada River Forecast Center

From: A. Hamlet, University of Washington

• Identifies systematic flaws of an ensemble prediction system.• Shows effectiveness of ensemble distribution in sampling the observations. • Does not indicate that the ensemble will be of practical use.

Talagrand Diagram – Also Called Ranked Histogram

With only one ensemble member ( | ) all (100%) observations ( ) will fall “outside”

With two ensemble members two out ofthree observations ( 2/3=67%) should fall outside

With three ensemble members two out of four observations ( 2/4=50%) should fall outside

| | | For any number of ensemble members, 2/#members should fall outside the ensemble

• Identifies systematic flaws of an ensemble prediction system.• Shows effectiveness of ensemble distribution in sampling the observations. • Does not indicate that the ensemble will be of practical use.

Principle Behind Talagrand Diagram

Talagrand Diagram – Also Called Ranked Histogram

Adapted from A. Persson, 2006

Talagrand Diagram Computation Example

YEAR E1 E2 E3 E4

1981 42 74 82 90

1982 65 143 223 227

1983 82 192 295 300

1984 211 397 514 544

1985 142 291 349 356

1986 114 277 351 356

1987 98 170 204 205

1988 69 169 229 236

1989 94 219 267 270

1990 59 175 244 250

1991 108 189 227 228

1992 94 135 156 158

Four sample ensemble members (E1 – E4) for daily flow forecasts (produced from reforecasts using carryover each year)

Step 1: Rank lowest to highest for each year. Four members results in 5 bins.

Step 2: Determine which bin the corresponding observation falls into.

Step 3: Tally how many observations fall in each bin.

Step 4: Plot frequency of observations for ranked bin.

Bin1 Bin2 Bin3 Bin4 Bin5

Bin # Tally

1 2 3 4 5

YEAR E1 E2 E3 E4

1981 42 74 82 90

1982 65 143 223 227

1983 82 192 295 300

1984 211 397 514 544

1985 142 291 349 356

1986 114 277 351 356

1987 98 170 204 205

1988 69 169 229 236

1989 94 219 267 270

1990 59 175 244 250

1991 108 189 227 228

1992 94 135 156 158

Four sample ensemble members (E1 – E4) for daily flow forecasts (produced from reforecasts using carryover each year)

Step 1: Rank lowest to highest for each year. Four members results in 5 bins.

Step 2: Determine which bin the corresponding observation falls into.

Step 3: Tally how many observations fall in each bin.

Step 4: Plot frequency of observations for ranked bin.

Bin # Tally

1 1 2 1 3 3 4 3 5 4

Talagrand Diagram

1 2 3 4 5

Categor y Defi ned by Or der ed E nsembl e Member s

YEAR E1 E2 E3 E4

1981 42 74 82 90

1982 65 143 223 227

1983 82 192 295 300

1984 211 397 514 544

1985 142 291 349 356

1986 114 277 351 356

1987 98 170 204 205

1988 69 169 229 236

1989 94 219 267 270

1990 59 175 244 250

1991 108 189 227 228

1992 94 135 156 158

Four sample ensemble members (E1 – E4) ranked lowest to highest for daily flow (produced from reforecasts using carryover in each year)

Bin # Tally

1 1 2 1 3 3 4 3 5 4

Talagrand Diagram: 25 traces/ensemble, 375 observations

Example: “U-Shaped”Observations too often falling outside ensembleIndicates ensemble spread too small

Example: “L-Shaped”Observations too often larger (smaller) than ensembleIndicates under- (over-) forecasting bias

Example: “N-Shaped” (domed shaped)Observations too rarely falling outside ensembleIndicates ensemble spread is too big

Example: “Flat-Shaped”Observations falling uniformly across ensembleIndicates appropriately sized ensemble distribution

1 3 5 7 9

Category(defined by ordered ensemble members)

1 3 5 7 9

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

Talagrand Diagram Example: Interpretation?

YEAR E1 E2 E3 E4

1981 42 74 82 90

1982 65 143 223 227

1983 82 192 295 300

1984 211 397 514 544

1985 142 291 349 356

1986 114 277 351 356

1987 98 170 204 205

1988 69 169 229 236

1989 94 219 267 270

1990 59 175 244 250

1991 108 189 227 228

1992 94 135 156 158

Bin # Tally

1 1 2 1 3 3 4 3 5 4

Four sample ensemble members (E1 – E4) ranked lowest to highest for daily flow (produced from reforecasts using carryover in each year)

Talagrand Diagram

1 2 3 4 5

Categor y Defi ned by Or der ed E nsembl e Member s

Distributions-orientedForecast Evaluation

leads toDiagnostic Verification

It’s all about conditional and marginal distributions!

P(O|F), P(F|O), P(F), P(O)

Reliability, Discrimination, Sharpness, Uncertainty

Forecast Reliability -- P(O|F)

For a specified forecast condition, what does the distribution of observations look like?

Forecasted Probability

User perspective: “When you say 20% chance of flood flows, how often do flood flows actually happen?”

User perspective: “When you say 80% chance of flood flows, how often do flood flows actually happen?”

Good reliability – close to diagonal

Sharpness diagram (p(f)) –histogram of forecasts in each probability bin shows shows marginal distribution of forecasts

The reliability diagram is conditioned on the forecasts. That is, given that X was predicted, what was the outcome?

Reliability (Attributes) Diagram – Reliability, Sharpness

Reliability Diagram Example Computation

YEAR E1 E2 E3 E4

1981 42 74 82 90

1982 65 143 223 227

1983 82 192 295 300

1984 211 397 514 544

1985 142 291 349 356

1986 114 277 351 356

1987 98 170 204 205

1988 69 169 229 236

1989 94 219 267 270

1990 59 175 244 250

1991 108 189 227 228

1992 94 135 156 158

Step 1: Choose threshold value to base probability forecasts on. For simplicity we’ll choose the mean forecast over all years and all ensembles (= 208).

YEAR E1 E2 E3 E4

1981 42 74 82 90

1982 65 143 223 227

1983 82 192 295 300

1984 211 397 514 544

1985 142 291 349 356

1986 114 277 351 356

1987 98 170 204 205

1988 69 169 229 236

1989 94 219 267 270

1990 59 175 244 250

1991 108 189 227 228

1992 94 135 156 158

Step 2: Choose how many forecast probability categories to use (5 here: 0,.25,.5,.75,1)

Step 3: For each forecast, calculate the forecast probability below the threshold value.

P(peakfor < 208)

YEAR E1 E2 E3 E4

1981 42 74 82 90

1982 65 143 223 227

1983 82 192 295 300

1984 211 397 514 544

1985 142 291 349 356

1986 114 277 351 356

1987 98 170 204 205

1988 69 169 229 236

1989 94 219 267 270

1990 59 175 244 250

1991 108 189 227 228

1992 94 135 156 158

Step 2: Choose how many forecast probability categories to use (5 here: 0,.25,.5,.75,1)

Step 3: For each forecast, calculate the forecast probability below the threshold value.

P(peakfor < 208)

Step 4: Group the observations into groups of equal forecast probability (or, more generally, into forecast probability categories).

P(peakfor < 208)

P(peak < 208) = 0.0

516P(peak < 208) = 0.25

348, 98, 233P(peak < 208) = 0.5

206, 301, 245, 248, 227P(peak < 208) = 0.75

P(peak < 208) = 0.0

516P(peak < 208) = 0.25

348, 98, 233P(peak < 208) = 0.5

206, 301, 245, 248, 227

P(peak < 208) = 1.0

Step 4: Group the observations into groups of equal forecast probability (or, more generally, into forecast probability categories).

P(peakfor < 208)

P(peak < 208) = 0.0

516P(peak < 208) = 0.25

348, 98, 233P(peak < 208) = 0.5

206, 301, 245, 248, 227P(peak < 208) = 0.75

P(peak < 208) = 0.0

516P(peak < 208) = 0.25

348, 98, 233P(peak < 208) = 0.5

206, 301, 245, 248, 227

P(peak < 208) = 1.0

112, 156, 167

Step 5: For each group, calculate the frequency of observations above the threshold value, 208 cfs.

P(peak < 208) = 0.0

516P(peak < 208) = 0.25

348, 98, 233P(peak < 208) = 0.5

206, 301, 245, 248, 227P(peak < 208) = 0.75

N/AP(peak < 208) = 1.0

112, 156, 167

P(obs peak < 208 given [P(peakfor < 208) = 0.0]) = 0/1 = 0.0

P(obs peak < 208 given [P(peakfor < 208) = 1.0]) =

P(obs peak < 208 given [P(peakfor < 208) = 0.75]) = 0/0 = NA

Step 5: For each group, calculate the frequency of observations above the threshold value, 208 cfs.

P(peak < 208) = 0.0

516P(peak < 208) = 0.25

348, 98, 233P(peak < 208) = 0.5

206, 301, 245, 248, 227P(peak < 208) = 0.75

N/AP(peak < 208) = 1.0

112, 156, 167

P(obs peak < 208 given [P(peakfor < 208) = 1.0]) = 3/3 = 1

P(obs peak < 208 given [P(peakfor < 208) = 0.75]) = 0/0 = NA

Step 6: Plot centroid of the forecast category (just points in our case) on the x-axis against the observed frequency within each forecast category on the y-axis. Include the 45 degree diagonal for reference.

Step 7: Include sharpness plot showing the number of observation/forecast pairs in each category.

Good reliability – close to diagonal

Sharpness diagram (p(f)) –histogram of forecasts in each probability bin shows marginal distribution of forecasts

Good resolution –wide range of frequency of observations corresponding to forecast probabilities

Skill – related to Brier Skill Score, in reference to sample climatology (not historical climatology)

The reliability diagram is conditioned on the forecasts. That is, given that X was predicted, what was the outcome?

Reliability Diagram – Reliability, Sharpness – P(O|F)

Overall relative frequency of observations

(sample climatology)

Points closer to perfect-reliability line than to no-resolution line: subsamples of probabilistic forecast contribute positively to overall skill (as defined by BSS) in reference to sample climatology

No-skill line : halfway between perfect-reliability line and no-resolution line, with sample climatology as a reference

Attributes Diagram – Reliability, Resolution, Skill/No-skill

Climatology Minimal RESolution Underforecasting

Good RES, at expense of REL

Reliable forecasts of rare event

Small sample size

Source: Wilks (1995)

Interpretation of Reliability DiagramsInterpretation of Reliability Diagrams

Reliability

P[O|F]

Does the frequency of occurrence match your probability statement?

Identifies conditional biasR

Forecasted probability

No resolution

EVS Reliability Diagram Examples

25th Percentile Observed Flows (low flows)

Sharp forecasts, but low resolution

Arkansas-Red Basin, 24-hr flows, lead time 1-14 days

85th Percentile Observed Flows (high flows)

Good reliability at shorter lead times, long-leads miss high events

From: J. Brown, EVS Manual

Historical seasonal water supply outlooks

Colorado River Basin

Morrill, Hartmann, and Bales, 2007

Forecast probability

tive F

requency

2) These months show best reliability; low resolution limiting reliability

1) Few high prob. fcasts, good reliability between 10-70% probability; reliability improves.

Reliability: Colorado Basin ESP Seasonal Supply OutlooksReliability: Colorado Basin ESP Seasonal Supply Outlooks

LC JM (5 mo. lead)

LC MM (3 mo. lead)

LC AM (2 mo. lead)

UC JJy (7 mo. lead)

UC AJy (4 mo. lead)

UC JnJy (2 mo. lead)

3) Reliability decreases for later forecasts as resolution increases; UC good at extremes.

high 30% mid 40% low 30%

Franz, Hartmann, and Sorooshian, 2003

For a specified observation category, what do the forecast distributions look like?

Discrimination – P(F|O)

“When dry conditions happen… What do the forecasts usually look like?

You sure hope that forecasts look different when there’s a drought, compared to when there’s a flood!

Discrimination – P(F|O)

You sure hope that forecasts look different when there’s a drought, compared to when there’s a flood!

Example: NWS CPC Seasonal climate outlooks, sorted into DRY cases (lowest tercile), 1995-2001, all forecasts, all lead-times

Good discrimination! Not much discrimination!

Climatology

0.00 0.33 1.00

Probability of dry

Probability of wet

Climatology

0.00 0.33 1.00

Probability of dry

Probability of wet

tive F

requency

There is some discrimination…

Early forecasts warned “High flows less likely”

Jan-May

When unusually low flows happened…

P(F|Low flows). Low < 30th percentile

Franz, Hartmann, and Sorooshian (2003)

Discrimination: Lower Colorado ESP Supply Outlooks

tive F

requency

Good Discrimination…

Forecasts were saying:

1) high and mid- flows less likely.

2) Low flows more likely

Jan-May

Apr-May

There is some discrimination…

Early forecasts warned “High flows less likely”

Discrimination: Lower Colorado ESP Supply Outlooks

When unusually low flows happened…

P(F|Low flows). Low < 30th percentile

tive F

requency

high 30%

mid 40%

low 30%

1)High flows less likely.

2) No discrimination between mid and low flows.3) Both UC and LC show good discrimination for low flows at 2-month lead time.

Lower Colorado Basin Jan-May (5 mo. lead)

April-May (2 mo. lead)

Upper Colorado Basin Jan-July (7 mo. lead)

June-July (2 mo. lead)

For observed flows in lowest 30% of historic distribution

Discrimination: Colorado Basin ESP Supply Outlooks

Historical seasonal water supply outlooks

Colorado River Basin

All observation CDF is plotted and color coded by tercile.

Forecast ensemble members are sorted into 3 groups according to which tercile its associated observation falls into.

The CDF for each group is plotted in the appropriate color. i.e. high is blue.

Discrimination: CDF Perspective

Credit: K. Werner

In this case, there is relatively good discrimination since the three conditional forecast CDFs separate themselves.

Discrimination

Credit: K. Werner

Discrimination Example Computation

YEAR E1 E2 E3 E4

1981 42 74 82 90

1982 65 143 223 227

1983 82 192 295 300

1984 211 397 514 544

1985 142 291 349 356

1986 114 277 351 356

1987 98 170 204 205

1988 69 169 229 236

1989 94 219 267 270

1990 59 175 244 250

1991 108 189 227 228

1992 94 135 156 158

Step 1: Order observations and divide ordered list into categories. Here we will use terciles (≤ 167, 206 ≤ ≤ 245, ≥ 248) .

OBS Tercile

Middle

LowCredit: K. Werner

YEAR E1 E2 E3 E4

1981 42 74 82 90

1982 65 143 223 227

1983 82 192 295 300

1984 211 397 514 544

1985 142 291 349 356

1986 114 277 351 356

1987 98 170 204 205

1988 69 169 229 236

1989 94 219 267 270

1990 59 175 244 250

1991 108 189 227 228

1992 94 135 156 158

Step 2: Group forecast ensemble members according to OBS tercile.

Low OBS Forecasts:42, 74, 82, 90, 114, 277, 351, 356,98, 170, 204, 205, 94,135, 156, 158

OBS Tercile

Middle

YEAR E1 E2 E3 E4

1981 42 74 82 90

1982 65 143 223 227

1983 82 192 295 300

1984 211 397 514 544

1985 142 291 349 356

1986 114 277 351 356

1987 98 170 204 205

1988 69 169 229 236

1989 94 219 267 270

1990 59 175 244 250

1991 108 189 227 228

1992 94 135 156 158

Mid OBS Forecasts:65, 143, 223, 227, 69, 169, 229, 236, 94, 219, 267, 270,

OBS Tercile

Middle

YEAR E1 E2 E3 E4

1981 42 74 82 90

1982 65 143 223 227

1983 82 192 295 300

1984 211 397 514 544

1985 142 291 349 356

1986 114 277 351 356

1987 98 170 204 205

1988 69 169 229 236

1989 94 219 267 270

1990 59 175 244 250

1991 108 189 227 228

1992 94 135 156 158

Mid OBS Forecasts:65, 143, 223, 227, 69, 169, 229, 236, 94, 219, 267, 270, 108, 189, 227, 228

OBS Tercile

Middle

YEAR E1 E2 E3 E4

1981 42 74 82 90

1982 65 143 223 227

1983 82 192 295 300

1984 211 397 514 544

1985 142 291 349 356

1986 114 277 351 356

1987 98 170 204 205

1988 69 169 229 236

1989 94 219 267 270

1990 59 175 244 250

1991 108 189 227 228

1992 94 135 156 158

Hi OBS Forecasts:82, 192, 295, 300,

142, 291, 349, 356, 59, 175, 244, 250

OBS Tercile

Middle

YEAR E1 E2 E3 E4

1981 42 74 82 90

1982 65 143 223 227

1983 82 192 295 300

1984 211 397 514 544

1985 142 291 349 356

1986 114 277 351 356

1987 98 170 204 205

1988 69 169 229 236

1989 94 219 267 270

1990 59 175 244 250

1991 108 189 227 228

1992 94 135 156 158

Hi OBS Forecasts:82, 192, 295, 300, 211, 397, 514, 544142, 291, 349, 356, 59, 175, 244, 250

OBS Tercile

Middle

Step 3: Plot all-observation CDF color coded by tercile (≤ 167, 206 ≤ ≤ 245, ≥ 248).

Credit: K. Werner

OBS Tercile

Middle

Step 4: Add forecasts conditioned on observed terciles CDFs to plot.

Low OBS Forecasts:42, 74, 82, 90, 114, 277, 351, 356,98, 170, 204, 205, 94, 135, 156, 158Mid OBS Forecasts:65, 143, 223, 227,69, 169, 229, 236,94, 219, 267, 270,108, 189, 227, 228Hi OBS Forecasts:82, 192, 295, 300,211, 397, 514, 544,142, 291, 349, 356,59, 175, 244, 250

Credit: K. Werner

Step 5: Discrimination is shown by the degree to which the conditional forecast CDFs are separated from each other.

In this case, high forecasts discriminate better than mid and low forecasts.

Credit: K. Werner

How well do April – July volume forecasts discriminate when they are made in Jan, Mar, and May?

Poor discrimination in Jan between forecasting high and medium flows. Best discrimination in May.

Discrimination

Credit: K. Werner

Another way to look at discrimination using PDF’s in lieu of CDF’s.

The more separation between the PDF’s the better the discrimination.

Discrimination

Credit: K. Werner

Deterministic forecasts

• traditional in hydrology

• sub-optimal for decision making

Common perspective

“Deterministic model simulations and probabilistic forecasts … are two entirely different types of products. Direct comparison of probabilistic forecasts with deterministic single valued forecasts is extremely difficult”

Comparing Deterministic & Probabilistic ForecastsComparing Deterministic & Probabilistic Forecasts

- Anonymous

How can we compare deterministic and probabilistic forecasts?

Deterministic

Probabilistic

Source: XEFS Design Team, 2007

Option: Use ensemble median with standard metrics – No! x

55From: A. Hamlet, University of Washington

The ensemble mean minimizes error, but doesn’t represent the overall behavior.

“Pretend Determinism”

What’s wrong with using ‘deterministic’ metrics?

Metrics using only central tendency of each forecast pdf fail to distinguish between forecasts 1-3, but will identify 4 as inferior.Metrics that reward accuracy but punish spread will rank the forecast skill from 1 to 4.

0 0.5 1 1.5 2 2.5

Obs Value

How can we compare deterministic and probabilistic forecasts?

Deterministic

Probabilistic

Source: XEFS Design Team, 2007

Option: Use ensemble median with standard metrics – No! x

Climatologydistribution

Forecast distribution

Tercile boundaries(equal probability) Deterministic

forecast

Jack-knife calibration error =PDF of error distribution

can determine any quantiles

Deterministic vs. Probabilistic Forecasts

Observation

Flow, QApproach used by Morrill, Hartmann, Bales 2007

Lab Session -- Group Exercise

Choose a set of forecasts.

Develop strategies for verifying these forecasts from two perspectives:

- Users

- Forecasters during operations

Report back to group.

Repeat for second set of forecasts, if time permits.

Verification Continued… Holly C. Hartmann

Documents

4628 Hartmann - apwa.net

Package Contents - Hartmann

110719 Judgement Hartmann

2014.09.11 - NAEC seminar_MaRs_Philipp Hartmann

Carl Hartmann, Temando

Verification Introduction Holly C. Hartmann

Hartmann Secret-symbols Rosecrucians

012 hartmann comm_lpl_1962

ICTY Hartmann Judgment

Hartmann Final

HARTMANN AND KESTER’S PLANT PROPAGATION stuff/ph... · HARTMANN AND KESTER’S PLANT PROPAGATION. PRINCIPLES AND PRACTICES. EIGHTH EDITION. Hudson T. Hartmann, PhD. University of

Bootstrapping by Boye Hartmann

Hartmann - Model Basedworking

The Compliance- Heroes HARTMANN

Hartmann et al

Structure - Hartmann & Bender

Decision Support Services, Climate Services Holly C. Hartmann, University of Arizona What are decision support services? Climate change decision support

Hartmann Farbprospekt BO

Write. Click. Scrapbook. - have a holly Christmas have a holly … · 2009-12-08 · Christmas have a holly JOLLY have a holly Christmas have a holly JOLLY have a holly Christmas

1 Verification Introduction Holly C. Hartmann Department of Hydrology and Water Resources University of Arizona hollyoregon@juno.com RFC Verification Workshop,