Predictive Skill, Predictive Capability and Predictability in Ocean Forecasting

Allan R. Robinson

Patrick J. Haley, Jr.

Pierre F.J. Lermusiaux

Wayne G. Leslie

Division of Engineering and Applied Sciences

Department of Earth and Planetary Sciences

Harvard University

29 October 2002

Ocean Prediction

• Ocean prediction for science and operational applications has been initiated on basin and regional scales. Evaluation is now essential.

• Predictability limit is the time for two slightly different ocean states to evolve into realistic but entirely different states

• Predictive capability is ultimately limited by predictability but errors in data, models and methodology now limit prediction capability to shorter times

Predictive Skill

• Qualitative and quantitative evaluation of ocean forecasts by generic and regional-specific skill metrics is essential

• Phase errors/structural errors, initial/BC/model errors and their sources need to be identified

• Simple metrics from meteorology (root-mean square error, anomaly pattern correlation coefficient) used now but more sophisticated statistical metrics and quantitative measures associated with underlying dynamical processes are required.

CPSE/REA

Coastal Predictive Skill Experiment (CPSE)

• measures the ability of a forecast system to combine model results and observations in coastal domains or regimes and to accurately define the present state and predict the future state– oversampling is required for rigorous quantitative verification – provides the basis for optimal, efficient sampling for required accuracies

Rapid Environmental Assessment (REA)

• defined in the NATO naval environment as "the acquisition, compilation and release of tactically relevant environmental information in a tactically relevant time frame"

• Assessment of Skill for Coastal Ocean Transients– 6-26 June 2001– Massachusetts Bay and Gulf of Maine– http://www.deas.harvard.edu/~leslie/ASCOT01/sci_plan.html

• Predictive Skill Experiment– quantitative skill evaluation– forecast system development– real-time at-sea forecasting– real-time adaptive sampling

• Coupled Physical-Biological Experiment– initialization surveys of Mass. Bay and Gulf of Maine– wind-induced events, e.g. upwelling and buoyancy circulations– Gulf of Maine inflow to Mass. Bay– Mass. Bay outflow to Gulf of Maine– MWRA diffuser dispersion– verification survey of Mass. May

• Multiple vessels– NRV Alliance – SACLANTCEN – La Spezia,Italy– RV Gulf Challenger – UNH – Portsmouth, NH– RV Lucky Lady – UMass-Dartmouth – New Bedford, MA– RV Neritic – UMass-Boston – Boston, MA

ASCOT-01

ASCOT-01 Data and Modeling Domains6-26 June 2001

Forecast Skill MetricsSkill of the operational forecasts is measured using the metrics, Root-Mean-Square Error (RMSE) and Pattern Correlation Coefficient (PCC). These numbers are computed model level by model level (1 to 16), and as a volume average. Perfect values of the RMSE and PCC are, respectively, zero and one.  The metrics RSME and PCC are respectively defined by: 

  where denotes the true ocean, its forecast, a background field vector (e.g. large-scale field, climatological field, etc.), and || . ||2 the vector l2 norm.

A classic measure of skill is to compare the RMS and PCC of the forecast with that of the initial conditions (IC) (persistence). If the RMSE of the forecast is smaller than that of the IC, the forecast has RMS-skill or beats persistence. Similarly, if the PCC of the forecast is larger than that of the IC, the forecast has PCC-skill or has better patterns than persistence.

2ˆ

2

)ˆ()( PCC

bTTbTfT

bTTTbTfT

2

ˆ )ˆ()ˆ( RMSE TTTTTT ffTf

T̂ fT bT

Observation Errors – ASCOT-0119 June 2001

ASCOT-01 Skill Metrics

RMS (Temperature - Left; Salinity - Right)

PCC (Temperature - Left; Salinity - Right)

• Assessment of Skill for Coastal Ocean Transients– 7-17 May 2002– Tyhrrenian Sea, Ligurian Sea, Corsican Channel, Elba– http://www.deas.harvard.edu/~leslie/ASCOT02/sci_plan.html

• Predictive Skill Experiment– quantitative skill evaluation– forecast system development– real-time at-sea forecasting– real-time adaptive sampling– rigorous test of distributed ocean prediction system– AUV exercise support

• Physics Experiment– initialization survey of Corsican Channel and Elba island region– flow between Corsica and Elba– anticyclone north of Elba– flow between Elba and the coast of Italy– reduce multi-variate forecast errors

• Multiple vessels– NRV Alliance – SACLANTCEN – La Spezia,Italy– AUVs – SACLANTCEN – La Spezia,Italy– AUVs – MIT – Cambridge, MA

ASCOT-02

ASCOT-02 Data and Modeling Domains7-17 May 2002

Observation Errors – ASCOT-02

15 May 16 May

ASCOT-02 Skill Metrics

RMS (Temperature - Left; Salinity - Right)

PCC (Temperature - Left; Salinity - Right)

General AdaptiveSampling Objectives

• Go to dynamical “hotspots”

• Reduce error variance Reduce errors for tomorrow Maintain accurate forecast

• Maintain accurate synoptic picture

• Optimal sampling issues Automate all 3 above quantitatively Nonlinear and interdisciplinary impacts on the sampling Optimal sampling can be highly dependent on objectives and metrics Reducing error in analysis differs from reducing error in forecast Minimal final time error differs from minimal time-averaged error Minimize cost function containing 3 terms based on:

forecasted model errors (ESSE), forecasted significant dynamical events (MS-EVA, pattern

recognition) maximum length of time an area can be left without updating

Motivations for Adaptive Sampling Tracks – ASCOT-

02/GOATS• Sample in regions not yet covered to locate local structures

• Sample in regions not recently covered to understand evolution of structures

• Determine strength and structure of anticyclone north of Elba

• Determine general nature of flow in vicinity of Procchio Bay (e.g.. is it from north or result of flow through Corsican Channel from Tyrrhenian turning around island?)

• Evaluate structure and evolution of flow between Corsica and Elba

• Determine impact of flow between Elba and coast of Italy

Adaptive sampling tracksdesigned on a real-time basis.

AUV - Procchio BayNRV Alliance - Channel Domain

(Top left) Surface temperature after 4 days of model run. Overlaid on the temperature field are the 50, 200 and 500m isobaths. (Right) Satellite sea

surface temperature. (Bottom Left) Surface current from the ocean model. All fields are from 3 October.

Currents measured by NRV Alliance with the ship-borne ADCP during the

first update surveys. Data of 4 consecutive nights are merged. The SE current in the south-eastern corner is

due to high winds.

Currents measured by NRV Alliance with the ship-borne ADCP during

update surveys in early October. The anti-cyclonic eddy has shifted towards

the north.

HOPS ASCOT-01 Simulations (5 meters),SeaWiFS Composite Imagery and in situ data

13 June

20 June

10-17 June

18-25 June

Conclusions

• It is critically important to interpret and evaluate regional forecasts in order to establish usefulness to scientific and applied communities

• Results from ASCOT highlight the dual use of data for assimilation and skill evaluation and demonstrate quantitative forecast skill

• Real-time forecast experiments can lead to discoveries of regional features

• Multi-scale adaptive sampling is a fundamental component of forecast systems

Issues in MultiscaleAdaptive Sampling

• Uniformly sampled observations for initialization and assimilation as forecasts advance in time

sampled uniformly over a predetermined space-time grid, adequate to resolve scales of interest only a small subset of observations have significant impact on the accuracy of the forecasts impact subset is related to intermittent energetic synoptic dynamical events

• Adaptively sampled observations for initialization and assimilation as forecasts advance in time

sampling scheme tailored to the ocean state to be observed knowledge of ocean state from ongoing observations, nowcasts and forecasts adaptive sampling targets observations of greatest impact efficient adaptive sampling reduce observational requirements by one or two orders of magnitude

• Subjective adaptive sampling and objective adaptive sampling sampling can be based on environmental forecasts or error forecasts forecast information combined with a priori experience to intuitively choose future sampling objectively, forecast serves as input to a quantitative sampling criterion whose optimization predicts

the adapted sampling automated objective adaptive sampling