ASSESSING THE IMPACT OF COLLABORATIVE

RESEARCH PROJECTS ON NWS PERFORMANCE

Jeff Waldstreicher

Scientific Services Division – Eastern Region

Northeast Regional Operational Workshop (NROW)

November 4-5, 2003

Albany, NY

ASSESSING THE IMPACTS OF COLLABORATIVE PROJECTS

• IMPACT ON VERIFICATION SCORES– Performance Metrics

• CASE STUDY ANALYSIS– Event Verification– Subjective Evaluation of Impact on Forecast

Process• AFDs• Event Reviews and Impact Reports

COMPLICATIONS

• Many Factors Influence Performance Metrics– Factors Not Independent– Difficult to tie performance changes to a specific factor

• Cannot Analyze “Null Case” – If forecasters have “knowledge” or “data,” cannot

directly answer “What if they did not have the knowledge?”

FACTORS THAT INFLUENCE WARNING VERIFICATION SCORES

• Infusion of New Technologies – Hardware (New systems or Processors) – Software (New algorithms or Models)

 • Applied Research and Development

– National (Research Laboratory)– Local/Regional– Independent– Collaborative  

• Changes to Operational Procedures– Implementation of Best Practices

 

FACTORS THAT INFLUENCE WARNING VERIFICATION SCORES

• Climate Variability– Variations in frequency and type of events

• External Outreach and Education– Including development of spotter networks

• Personnel/Staffing Issues– Including forecaster experience levels

• Training– All other factors are also tied to training issues

HOW DO THESE FACTORS IMPACT PERFORMANCE METRICS?

• Can be global (national), regional, or local– Technology usually national

– Climate Variability typically regional

– Staffing usually local • Impact can be long or short term

– Technology infusion typically has long term impact

– Climate Variability is usually short term

• Factors are often inter-related– New technology or staffing changes require training

– Applied R&D often yields/suggests changes to operations

COMET PROJECTS IN EASTERN REGION

• Different types of projects– Cooperative (2-3 year ~$35K/yr)

• 19 ER Cooperative Projects funded since 1991

– Partners (1 year ~$9K/yr)• 39 ER Partners Projects funded since 1991

• 20 ER WFOs and 3 RFC have participated– 90 Offices Nationwide

• 21 Universities have participated in ER projects– 70+ Universities Nationwide

EVALUATING COMET PROJECTS

• Examined COMET Cooperative and Partners Projects in Eastern Region– Projects completed between 1995 and mid-2001– Projects specifically addressing warning programs

• Tornado• Severe Thunderstorms• Flash Flooding• Winter Storms

• Study was designed to minimize as much as possible the impact of the factors previously discussed.

METHODOLOGY• 3-year running verification scores used

– Minimize impact of short-term factors such as variability of events

• Compared 3 years before project to 3 years following project– 1996-mid 2001 period helped ensure 3-years of post-88D data in “before” scores

and a full 3 years of “after” scores

• 3-year “expected” improvements calculated based on long term trend of ER-wide scores– Compared rate of improvement for WFO involved in collaborative project to ER-

wide improvement– Use of ER scores as a baseline minimizes the impact of national/region-wide

factors such as AWIPS and radar system improvements

• Very difficult to evaluate the impact of project results beyond the primary WFO

INDIVIDUAL PROJECT COMPARISON

• Compare 3-year performance change of WFOs involved in COMET projects to the region-wide improvements for the same time period

PROBABILITY OF DETECTION (POD) 

# of COMET Projects With

WFO Greater Improvement

# of COMET Projects With ER Greater

Improvement

# of COMET Projects With No

Difference in Improvement

Tornado Warnings 3 2 1

Severe Tstm Warnings 6 1 0

Flash Flood Warnings 5 1 0

Winter Storm Warnings 4 3 0

FALSE ALARM RATIO (FAR) 

# of COMET Projects With

WFO Greater

Improvement

# of COMET Projects With ER Greater

Improvement

# of COMET Projects With No

Difference in Improvement

Tornado Warnings 2 4 0

Severe Tstm Warnings 3 4 0

Flash Flood Warnings 3 3 0

Winter Storm Warnings 3 3 1

LEAD TIME 

# of COMET Projects With

WFO Greater Improvement

# of COMET Projects With ER Greater

Improvement

# of COMET Projects With No

Difference in Improvement

Tornado Warnings 5 1 0

Severe Tstm Warnings 5 1 1

Flash Flood Warnings 4 1 1

Winter Storm Warnings 4 3 0

IMPACT OF LONG TERM COLLABORATIVE ACTIVITIES

• WFO RAH and NCSU– Continuous collaborative projects since January 1991– COMET Projects 1991-2000

• 3 Cooperative Projects• 3 Partners Projects• 1 Graduate Fellowship

– CSTAR Projects• 2000-2003• 2003-2006

• Unique “laboratory” to examine the impact of long-term collaborative activities

IMPACT OF LONG TERM COLLABORATIVE ACTIVITIES

• WFO ALY and U-Albany– Continuous since February 1995– Numerous COMET Projects

• 3 Cooperative Projects

• 7 Partners Projects

• Several additional projects between UA and NCEP

– CSTAR PROJECT 2001-2004

SUMMARY

• Verification scores for ER WFOs involved in COMET collaborative research projects appear to improve at a greater rate than overall ER performance

• There are indications that certain

performance metrics (e.g., lead time) are more responsive to improved scientific understanding, while others (POD) are more dependent on technology upgrades.