ESDSWG meeting – 10/21-23/2008

Metrics Planning and Reporting (MPAR) WGBreakout Summary

H. K. (Rama) RamapriyanNASA GSFCClyde BrownNASA LaRC

Co-Chairs, MPARWG

8th Earth Science Data Systems Working Group

Joint Working Group Meeting

Wilmington, DE

October 20-22, 2009

ESDSWG meeting – 10/21-23/2008

MPAR Working Group

Attendance – 23

Topics DiscussedDiscussions with Martha MaidenMetrics baseline reviewFY 09 ReportingCore and Community metrics collection

ESDSWG meeting – 10/21-23/2008

MPARWG Breakout Sessions

Item Who

Day 1 - Session 1: FY09 Activities and Progress  

Introduction Rama

HQ / Program view of Metrics Maiden Martha Maiden's Four Questions and MPARWG responses Hunolt

E-Books vs Metrics - What is Reported Via Each Rama

Action / Open Items from Oct 2008 workshop RamaSummary of FY09 Metrics Reporting Hunolt

Day 2 - Session 2: Metrics and MEaSUREs  

MEaSUREs Telecon Results RamaMetrics Baseline Review Hunolt / All

Day 2 - Session 3: Metrics Collection - Core and Community

 

Overview of Metrics Collection MCT & EMS Rama

Introduction for MEaSURES and Changes to MCT Randy Barth

Transition to "MCT2" and EMS for Metrics Reporting Kevin Murphy

Day 2 - Session 4: TBD, Action Items and Work Plan

 

Continued discussion of metrics baseline Hunolt/All

ESDSWG meeting – 10/21-23/2008

Discussions with Martha Maiden Martha emphasized importance of metrics for her reporting to her management and OMB Metrics from both Community systems and Core systems are important

User satisfaction metrics – American Consumer Satisfaction Index is an important metric watched by OMB

Aggregates and highlights from core and community metrics are useful

Impact metrics are very useful, but a better format is needed

Encouraged use of metrics for the benefit of the Projects themselves

ESDSWG meeting – 10/21-23/2008

Actions Planned in Response

Provide separate aggregates for community systems that include trends, so community systems are not lost in the totals.

Provide periodic reports in a form useful to MarthaProgram level summaries and trendsHighlights from individual community systems. These

could be based on the regular metrics, e.g. noting significant growth in users, distribution, addition of new projects or services, etc.

Projects could use supplemental comments to highlight / call attention to items.

The form in which this information is provided to Martha could be quad-chart or a short pdf ‘newsletter’.

ESDSWG meeting – 10/21-23/2008

Metrics Baseline Review

Products and Services Metrics -

Common Metrics:Metric 1 - Distinct Users – No changeMetric 2 – Distinct Users by Class – No changeMetric 3 – Products Distributed – Added option of breaking down by product types

Metric 4 – Product Types AvailableMetric 5 – Volume Distributed – after considerable discussion, left unchanged

Metric 6 – Volume Available – after considerable discussion, left unchanged

Metric 7 – no longer usedMetric 11 – Services ProvidedMetric 12 – Services Available – need explanatory examples for projects to use this better

ESDSWG meeting – 10/21-23/2008

Metrics Baseline Review

Project Defined (Project Specific) Metrics (up to 4, defined by each project, metrics 100, 101, 102, 103)

Programmatic Metrics:Metric 8 – Science Focus Areas SupportedMetric 9 – Applications Areas SupportedMetric 10 – Education categories Supported

Impact Metrics – decided to recommend use of quad chart format, but will develop strawman examples and validate with Martha

New Citation Metric Deborah Smith (RSS, Inc) presented an approach they have

been using to collect citations of their productsRecommend use of citation metric to be used optionally

ESDSWG meeting – 10/21-23/2008

FY 09 Reporting

22 REASoN; 15 ACCESS; 8 MEaSUREs [Many of the REASoN and ACCESS Projects completed during FY 09

Impact metrics – 3 from 2 REASoN and 1 MEaSUREs Projects

We expect 10 or more MEaSUREs projects to start reporting in FY 10Generally, projects report when they start distributing data to users

Last REASoN and ACCESS Projects are expected to finish in FY 10

Upcoming ACCESS reporting - TBD

ESDSWG meeting – 10/21-23/2008

Core and Community Metrics CollectionCore system metrics

EOSDIS Data Center metrics collection using EMS was demonstrated

Internal use of metrics in ESDIS for planning purposes

Network flows vs. requirements are analyzed using periodic “pinging” and measuring actual data flows

Community metrics Collected using MCTFY 09 improvements to MCT discussed (e.g., 508 compliance)

Transition to EMS-based collection of community metrics discussed – needs further examination

Recommended prototyping automated collection with two separate (diverse) projects using log files