Transcript
Page 1: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

ASSESSING THE IMPACT OF COLLABORATIVE

RESEARCH PROJECTS ON NWS PERFORMANCE

Jeff Waldstreicher

Scientific Services Division – Eastern Region

Northeast Regional Operational Workshop (NROW)

November 4-5, 2003

Albany, NY

Page 2: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

ASSESSING THE IMPACTS OF COLLABORATIVE PROJECTS

• IMPACT ON VERIFICATION SCORES– Performance Metrics

• CASE STUDY ANALYSIS– Event Verification– Subjective Evaluation of Impact on Forecast

Process• AFDs• Event Reviews and Impact Reports

Page 3: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

COMPLICATIONS

• Many Factors Influence Performance Metrics– Factors Not Independent– Difficult to tie performance changes to a specific factor

• Cannot Analyze “Null Case” – If forecasters have “knowledge” or “data,” cannot

directly answer “What if they did not have the knowledge?”

Page 4: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

FACTORS THAT INFLUENCE WARNING VERIFICATION SCORES

• Infusion of New Technologies – Hardware (New systems or Processors) – Software (New algorithms or Models)

 • Applied Research and Development

– National (Research Laboratory)– Local/Regional– Independent– Collaborative  

• Changes to Operational Procedures– Implementation of Best Practices

 

Page 5: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

FACTORS THAT INFLUENCE WARNING VERIFICATION SCORES

• Climate Variability– Variations in frequency and type of events

• External Outreach and Education– Including development of spotter networks

• Personnel/Staffing Issues– Including forecaster experience levels

• Training– All other factors are also tied to training issues

Page 6: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

HOW DO THESE FACTORS IMPACT PERFORMANCE METRICS?

• Can be global (national), regional, or local– Technology usually national

– Climate Variability typically regional

– Staffing usually local • Impact can be long or short term

– Technology infusion typically has long term impact

– Climate Variability is usually short term

• Factors are often inter-related– New technology or staffing changes require training

– Applied R&D often yields/suggests changes to operations

Page 7: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

COMET PROJECTS IN EASTERN REGION

• Different types of projects– Cooperative (2-3 year ~$35K/yr)

• 19 ER Cooperative Projects funded since 1991

– Partners (1 year ~$9K/yr)• 39 ER Partners Projects funded since 1991

• 20 ER WFOs and 3 RFC have participated– 90 Offices Nationwide

• 21 Universities have participated in ER projects– 70+ Universities Nationwide

Page 8: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

EVALUATING COMET PROJECTS

• Examined COMET Cooperative and Partners Projects in Eastern Region– Projects completed between 1995 and mid-2001– Projects specifically addressing warning programs

• Tornado• Severe Thunderstorms• Flash Flooding• Winter Storms

• Study was designed to minimize as much as possible the impact of the factors previously discussed.

Page 9: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

METHODOLOGY• 3-year running verification scores used

– Minimize impact of short-term factors such as variability of events

• Compared 3 years before project to 3 years following project– 1996-mid 2001 period helped ensure 3-years of post-88D data in “before” scores

and a full 3 years of “after” scores

• 3-year “expected” improvements calculated based on long term trend of ER-wide scores– Compared rate of improvement for WFO involved in collaborative project to ER-

wide improvement– Use of ER scores as a baseline minimizes the impact of national/region-wide

factors such as AWIPS and radar system improvements

• Very difficult to evaluate the impact of project results beyond the primary WFO

Page 10: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
Page 11: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
Page 12: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
Page 13: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
Page 14: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
Page 15: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
Page 16: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

INDIVIDUAL PROJECT COMPARISON

• Compare 3-year performance change of WFOs involved in COMET projects to the region-wide improvements for the same time period

Page 17: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

PROBABILITY OF DETECTION (POD) 

# of COMET Projects With

WFO Greater Improvement

# of COMET Projects With ER Greater

Improvement

# of COMET Projects With No

Difference in Improvement

Tornado Warnings 3 2 1

Severe Tstm Warnings 6 1 0

Flash Flood Warnings 5 1 0

Winter Storm Warnings 4 3 0

Page 18: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

FALSE ALARM RATIO (FAR) 

# of COMET Projects With

WFO Greater

Improvement

# of COMET Projects With ER Greater

Improvement

# of COMET Projects With No

Difference in Improvement

Tornado Warnings 2 4 0

Severe Tstm Warnings 3 4 0

Flash Flood Warnings 3 3 0

Winter Storm Warnings 3 3 1

Page 19: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

LEAD TIME 

# of COMET Projects With

WFO Greater Improvement

# of COMET Projects With ER Greater

Improvement

# of COMET Projects With No

Difference in Improvement

Tornado Warnings 5 1 0

Severe Tstm Warnings 5 1 1

Flash Flood Warnings 4 1 1

Winter Storm Warnings 4 3 0

Page 20: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

IMPACT OF LONG TERM COLLABORATIVE ACTIVITIES

• WFO RAH and NCSU– Continuous collaborative projects since January 1991– COMET Projects 1991-2000

• 3 Cooperative Projects• 3 Partners Projects• 1 Graduate Fellowship

– CSTAR Projects• 2000-2003• 2003-2006

• Unique “laboratory” to examine the impact of long-term collaborative activities

Page 21: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
Page 22: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
Page 23: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
Page 24: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
Page 25: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
Page 26: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
Page 27: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

IMPACT OF LONG TERM COLLABORATIVE ACTIVITIES

• WFO ALY and U-Albany– Continuous since February 1995– Numerous COMET Projects

• 3 Cooperative Projects

• 7 Partners Projects

• Several additional projects between UA and NCEP

– CSTAR PROJECT 2001-2004

Page 28: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
Page 29: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
Page 30: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
Page 31: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
Page 32: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
Page 33: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE
Page 34: ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

SUMMARY

• Verification scores for ER WFOs involved in COMET collaborative research projects appear to improve at a greater rate than overall ER performance

• There are indications that certain

performance metrics (e.g., lead time) are more responsive to improved scientific understanding, while others (POD) are more dependent on technology upgrades.


Recommended