ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCEJeff WaldstreicherScientific Services Division Eastern Region
Northeast Regional Operational Workshop (NROW)November 4-5, 2003Albany, NY
ASSESSING THE IMPACTS OF COLLABORATIVE PROJECTSIMPACT ON VERIFICATION SCORESPerformance Metrics
CASE STUDY ANALYSISEvent VerificationSubjective Evaluation of Impact on Forecast ProcessAFDsEvent Reviews and Impact Reports
COMPLICATIONSMany Factors Influence Performance MetricsFactors Not IndependentDifficult to tie performance changes to a specific factor
Cannot Analyze Null Case If forecasters have knowledge or data, cannot directly answer What if they did not have the knowledge?
FACTORS THAT INFLUENCE WARNING VERIFICATION SCORES Infusion of New Technologies Hardware (New systems or Processors) Software (New algorithms or Models)Applied Research and DevelopmentNational (Research Laboratory)Local/RegionalIndependentCollaborativeChanges to Operational ProceduresImplementation of Best Practices
FACTORS THAT INFLUENCE WARNING VERIFICATION SCORESClimate VariabilityVariations in frequency and type of events
External Outreach and EducationIncluding development of spotter networks
Personnel/Staffing IssuesIncluding forecaster experience levels
TrainingAll other factors are also tied to training issues
HOW DO THESE FACTORS IMPACT PERFORMANCE METRICS?Can be global (national), regional, or localTechnology usually nationalClimate Variability typically regionalStaffing usually local Impact can be long or short termTechnology infusion typically has long term impactClimate Variability is usually short termFactors are often inter-relatedNew technology or staffing changes require training Applied R&D often yields/suggests changes to operations
COMET PROJECTS IN EASTERN REGIONDifferent types of projectsCooperative (2-3 year ~$35K/yr)19 ER Cooperative Projects funded since 1991 Partners (1 year ~$9K/yr)39 ER Partners Projects funded since 1991
20 ER WFOs and 3 RFC have participated90 Offices Nationwide21 Universities have participated in ER projects70+ Universities Nationwide
EVALUATING COMET PROJECTSExamined COMET Cooperative and Partners Projects in Eastern RegionProjects completed between 1995 and mid-2001Projects specifically addressing warning programsTornadoSevere ThunderstormsFlash FloodingWinter Storms
Study was designed to minimize as much as possible the impact of the factors previously discussed.
METHODOLOGY3-year running verification scores usedMinimize impact of short-term factors such as variability of events
Compared 3 years before project to 3 years following project1996-mid 2001 period helped ensure 3-years of post-88D data in before scores and a full 3 years of after scores
3-year expected improvements calculated based on long term trend of ER-wide scoresCompared rate of improvement for WFO involved in collaborative project to ER-wide improvementUse of ER scores as a baseline minimizes the impact of national/region-wide factors such as AWIPS and radar system improvements
Very difficult to evaluate the impact of project results beyond the primary WFO
INDIVIDUAL PROJECT COMPARISONCompare 3-year performance change of WFOs involved in COMET projects to the region-wide improvements for the same time period
IMPACT OF LONG TERM COLLABORATIVE ACTIVITIESWFO RAH and NCSUContinuous collaborative projects since January 1991COMET Projects 1991-20003 Cooperative Projects3 Partners Projects1 Graduate FellowshipCSTAR Projects2000-20032003-2006Unique laboratory to examine the impact of long-term collaborative activities
IMPACT OF LONG TERM COLLABORATIVE ACTIVITIESWFO ALY and U-AlbanyContinuous since February 1995Numerous COMET Projects3 Cooperative Projects7 Partners ProjectsSeveral additional projects between UA and NCEP CSTAR PROJECT 2001-2004
SUMMARYVerification scores for ER WFOs involved in COMET collaborative research projects appear to improve at a greater rate than overall ER performance There are indications that certain performance metrics (e.g., lead time) are more responsive to improved scientific understanding, while others (POD) are more dependent on technology upgrades.