ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE

  • Published on
    21-Jan-2016

  • View
    46

  • Download
    0

Embed Size (px)

DESCRIPTION

ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCE. Jeff Waldstreicher Scientific Services Division Eastern Region Northeast Regional Operational Workshop (NROW) November 4-5, 2003 Albany, NY. ASSESSING THE IMPACTS OF COLLABORATIVE PROJECTS. - PowerPoint PPT Presentation

Transcript

  • ASSESSING THE IMPACT OF COLLABORATIVE RESEARCH PROJECTS ON NWS PERFORMANCEJeff WaldstreicherScientific Services Division Eastern Region

    Northeast Regional Operational Workshop (NROW)November 4-5, 2003Albany, NY

  • ASSESSING THE IMPACTS OF COLLABORATIVE PROJECTSIMPACT ON VERIFICATION SCORESPerformance Metrics

    CASE STUDY ANALYSISEvent VerificationSubjective Evaluation of Impact on Forecast ProcessAFDsEvent Reviews and Impact Reports

  • COMPLICATIONSMany Factors Influence Performance MetricsFactors Not IndependentDifficult to tie performance changes to a specific factor

    Cannot Analyze Null Case If forecasters have knowledge or data, cannot directly answer What if they did not have the knowledge?

  • FACTORS THAT INFLUENCE WARNING VERIFICATION SCORES Infusion of New Technologies Hardware (New systems or Processors) Software (New algorithms or Models)Applied Research and DevelopmentNational (Research Laboratory)Local/RegionalIndependentCollaborativeChanges to Operational ProceduresImplementation of Best Practices

  • FACTORS THAT INFLUENCE WARNING VERIFICATION SCORESClimate VariabilityVariations in frequency and type of events

    External Outreach and EducationIncluding development of spotter networks

    Personnel/Staffing IssuesIncluding forecaster experience levels

    TrainingAll other factors are also tied to training issues

  • HOW DO THESE FACTORS IMPACT PERFORMANCE METRICS?Can be global (national), regional, or localTechnology usually nationalClimate Variability typically regionalStaffing usually local Impact can be long or short termTechnology infusion typically has long term impactClimate Variability is usually short termFactors are often inter-relatedNew technology or staffing changes require training Applied R&D often yields/suggests changes to operations

  • COMET PROJECTS IN EASTERN REGIONDifferent types of projectsCooperative (2-3 year ~$35K/yr)19 ER Cooperative Projects funded since 1991 Partners (1 year ~$9K/yr)39 ER Partners Projects funded since 1991

    20 ER WFOs and 3 RFC have participated90 Offices Nationwide21 Universities have participated in ER projects70+ Universities Nationwide

  • EVALUATING COMET PROJECTSExamined COMET Cooperative and Partners Projects in Eastern RegionProjects completed between 1995 and mid-2001Projects specifically addressing warning programsTornadoSevere ThunderstormsFlash FloodingWinter Storms

    Study was designed to minimize as much as possible the impact of the factors previously discussed.

  • METHODOLOGY3-year running verification scores usedMinimize impact of short-term factors such as variability of events

    Compared 3 years before project to 3 years following project1996-mid 2001 period helped ensure 3-years of post-88D data in before scores and a full 3 years of after scores

    3-year expected improvements calculated based on long term trend of ER-wide scoresCompared rate of improvement for WFO involved in collaborative project to ER-wide improvementUse of ER scores as a baseline minimizes the impact of national/region-wide factors such as AWIPS and radar system improvements

    Very difficult to evaluate the impact of project results beyond the primary WFO

  • INDIVIDUAL PROJECT COMPARISONCompare 3-year performance change of WFOs involved in COMET projects to the region-wide improvements for the same time period

  • IMPACT OF LONG TERM COLLABORATIVE ACTIVITIESWFO RAH and NCSUContinuous collaborative projects since January 1991COMET Projects 1991-20003 Cooperative Projects3 Partners Projects1 Graduate FellowshipCSTAR Projects2000-20032003-2006Unique laboratory to examine the impact of long-term collaborative activities

  • IMPACT OF LONG TERM COLLABORATIVE ACTIVITIESWFO ALY and U-AlbanyContinuous since February 1995Numerous COMET Projects3 Cooperative Projects7 Partners ProjectsSeveral additional projects between UA and NCEP CSTAR PROJECT 2001-2004

  • SUMMARYVerification scores for ER WFOs involved in COMET collaborative research projects appear to improve at a greater rate than overall ER performance There are indications that certain performance metrics (e.g., lead time) are more responsive to improved scientific understanding, while others (POD) are more dependent on technology upgrades.

Recommended

View more >