The Developmental Testbed Center:The Developmental Testbed Center:Historical Perspective and Historical Perspective and

Benefits to NOAABenefits to NOAA

Steve KochSteve KochDTC Deputy DirectorDTC Deputy Director

Director, NOAA/ESRL/Global Systems DivisionDirector, NOAA/ESRL/Global Systems Division

Fundamental Purpose of DTC

ResearchCommunity

OperationalCommunity

New Science and Technology

Operational Codes

DTC

Operational Codes

DTC

To serve as a bridge between research and operations to facilitate the activities of both halves of the NWP Community in pursuit of their own objectives:

Research community gets a functionally equivalent operational environment to test and evaluate new NWP methods in retrospective extended period tests using advanced tools

Operational community benefits from DTC testing and evaluation of strengths and weaknesses of new NWP advances prior to consideration for operational implementation

NOAA will use the DTC as the primary gateway through which promising well-tested NWP science and technology originating in the research community will be selected for further development and evaluation by NCEP/EMC

EMC resources will be used much more efficiently since DTC will filter out less promising technologies and greatly reduce the time needed for the better ones to become ready for operational implementation

DTC activity links with Weather and Water Goal’s STI program, with future support to be provided primarily through the Environmental Modeling Program

Agency priorities for DTC

NOAA: Top priority of DTC is “to manage the WRF community code and support the WRF modeling system to the research community… the DTC is the critical link to reaping the benefits of the WRF Program.”

April 28, 2006 WRF ExOB

AFWA: Top priority of DTC is “to manage the WRF baseline. DTC is the key component/enabler for establishing, maintaining, and supporting the ‘Official’ (ARW- and NMM-WRF cores) baseline of the community’s end-to-end WRF ‘Reference-grade’ code.”

June 5, 2006 SOW

Operational Community Provide guidance for next generation Operational Configurations

Research Community Provide baselines for evaluating impacts of new techniques Aid in selecting configurations for research projects

DTC demonstrated the capabilities of the candidate dynamical cores to qualify them for a 6-member WRF IOC ensemble system to run daily in High Resolution Window (HRW) domains:

◦ NMM + ARW dynamical WRF cores with NCEP + NCAR physics

◦ NMM and ARW cores with switched physics packages

◦ Model variants using bred Initial / Boundary Conditions

Ensemble improves accuracy over a single deterministic forecast

and provides for a measure of uncertainty.

WRF Test Plan: Getting to the NCEP IOC

DTC testing and evaluation demonstrated to NCEP that the 6 WRF members tested were qualified to run as an ensemble system

21 Sept 2004: IOC implemented with only 2 members (the two dynamical cores - ARW and NMM - without physics swapping)

6 Dec 2005: Six WRF members added to NCEP Short-Range Ensemble Forecast (SREF) included physics swapping between the cores and 2 members from NCEP breeding system

New WRF-based SREF outperformed the current one in all ensemble aspects including mean, spread and probability distributions

Impact of the WRF Test Plan on NCEP

Goal: Provide real-time model guidance for winter weather over a large domain using 2 WRF configurations with explicit convection

Objectives NWS: Expose forecasters to WRF at very high (5-km) resolution

prior to the first scheduled operational implementation at NCEP.

NCEP: Evaluate WRF in HRW at 8-10 km resolution relative to WRF run at 4-5 km over CONUS with explicit microphysics only.

NCAR: Determine whether encouraging results from warm season 4-km WRF runs carry over to winter events out to 48h.

Real-time displays:◦ NCAR: DTC website◦ GSD: FX-Net◦ GSD: 2D AWIPS fields-NMM only

Objective & subjective evaluations by DTC and NWS, respectively

Provided valuable hands-on experience to forecasters with high-resolution WRF model output over CONUS

NCEP added simulated reflectivity to diagnostic fields available to WFOs

Results helped NCEP decide to upgrade resolution of HRW domains (NMM 8 to 5.1 km / ARW 10 to 5.8 km)

DTC demonstrated importance of statistical tests of significance when comparing forecast skill from multiple models

NMM 5 km real-time runs were continued into Spring at request of forecast offices (NMM5-CONUS)

DWFE Impact on NWS

0600 UTC Radar Mosaic

ARW 3-h forecast Composite Reflectivity

Lake-effect snowbands

Lake-effect snowbands

Narrow Cold-Frontal Rainband

Narrow Cold-Frontal Rainband

Observed Composite Reflectivity at 03Z

WRF-ARW 00Z 22Jan05 F27

988

Verifying 5-km Surface Mesoanalysis for 0300 UTC

995

Stationary Mesoscale Wave-Precip Systems

Eta-12

WRF-NMM

WRF-ARW

DWFE Verification Activities

• Key questions still remain: • What are the implications of choosing a single dynamic core for WRF?• Do the potential cost savings overcome the loss of model diversity for

ensemble predictions?

• Previous DTC experiments provided some idea of how the two WRF cores compare in terms of forecast skill, but no tests have been totally “clean” so far – this requires each core to be run with the same physics package, initial and boundary conditions, horizontal and vertical resolution, & run over the same domain

• This was done with the WRF-Rapid Refresh “core test” • WRF-RR is the first operational WRF model at NCEP for which the DTC can

perform such tests and evaluations prior to implementation• Never before has such a clean dynamic core test been performed• 2 tests completed: using NMM physics in both cores and RUC-like physics• 13-km WRF run to 24h even though RUC only extends to 12h

WRF-Rapid Refresh “Core Testing”

Goals Conduct controlled experiments carefully

comparing the 2 WRF dynamic cores.  Provide datasets to support GSD’s dynamic core

recommendation to NCEP for WRF-RR

Impacts 1st “clean” WRF dynamic core comparison ARW selected for initial implementation of WRF-RR

Domain: CONUS Dt: ARW 72 s, NMM 30 s

ICs: RUC13

BCs: NAM

Forecast length: 24 hForecast cycles: 00 & 12 UTC

Upper BC: NMM – default ARW – damping layer (5 km)

Physics frequency: Radiation - 30 min Other - ARW 72 s, NMM 60 s

Dx: 13 km

Vertical levels: 50

Summer 15 Jul - 15 Aug 05

Fall 1 - 30 Nov 05

Winter 15 Jan - 15 Feb 06

Spring 25 Mar - 25 Apr 06

Physics Phase 1 Phase 2

Microphysics Ferrier Thompson

Radiation GFDL GFDL

PBL MYJ MYJ

LSM Noah-99 RUC

Cumulus BMJ Grell-Devenyi

Red = Physics 1 = NMM set

Blue = Physics 2 = RUC-like set

2-sigma = 95% CI

Red = Physics 1 = NMM set

Blue = Physics 2 = RUC-like set

2-sigma = 95% CI

Objective Assess whether the medium range (60h) skill of the two

dynamic cores is similar enough to allow the direct transfer of research results obtained with one core to the other.

Approach Use Rapid Refresh Core Test as basic template for test

design Use current code for all components of end-to-end system

◦ Relevance to community!

◦ Maximize support from developers

WRF Core Test - Extended

•Wind errors are maximum at 250 hPa•NMM has best forecast at 850 and 700 hPa

60-h bias-corrected RMSE 60-h bias-corrected RMSE (NMM in red; ARW in blue; ARW-NMM in green)

•Temperature errors show max at 850 and 200 hPa•Core differences are not statistically significant

•RH errors increase with height to 500 hPa•Core differences are not statistically significant

• Most variables and levels, such as surface temperature, do not show an increase in ARW – NMM differences in time out to 60h.• However, core differences do grow in time for a few variables and levels, such as temperature at 100 hPa.

Evolution in TimeEvolution in Time

2-m temperature bias 100 hPa temperature bias

Extended Core Test

ARW and NMM results are not substantially different Inter-core differences do not grow in time Potential for transferring research done with one core to the other

T G164 P500 bcrmse (linux followed by ibm)

0 10 20 30 40 50 60

-0.15

-0.05

0.05

0.15

T G164 P500 bcrmse (linux followed by ibm)

0 10 20 30 40 50 60

-0.15

-0.05

0.05

0.15

difference in bcrmse (linux - ibm)

forecast time

0 10 20 30 40 50 60

-0.010

0.0

0.010

W W

W

W

W WV V

VV

V V

S

S

S

S

S

S

AA

A

A

A A

IBM vs Linux Comparison

WRF Test Plan◦ NMM ported to WRF Framework (increased number of options

available to community) DWFE

◦ Problem related to initialization of small lakes during winter season brought to attention of developers

WRF-RR Core Test◦ Significant increase in physics options available for NMM

Extended Core Test◦ More consistency between pre-processing options available for 2

dynamic cores Pre-release and system testing

◦ Bug fixes for both dynamic cores◦ Improved interoperability (more microphysics & PBL schemes

available for NMM)

Impact of DTC T & EWRF Community Code