RAINEX Data Management UCAR Joint Office for Science Support José Meitín Jim Moore Dick Dirks UCAR Joint Office for Science Support José Meitín Jim Moore

RAINEX Data ManagementRAINEX Data Management

UCAR Joint Office for Science Support

José MeitínJim MooreDick Dirks

UCAR Joint Office for Science Support

José MeitínJim MooreDick Dirks

RAINEX Planning Meeting, Miami, FLRAINEX Planning Meeting, Miami, FL

•Early involvement in project planning

•Involvement with PIs to develop data management strategy

•Consistent implementation of strategy for lifetime of project and beyond

•Reliable and efficient archive and distribution system

•Easy and efficient access to datasets by broader community

•Early involvement in project planning

•Involvement with PIs to develop data management strategy

•Consistent implementation of strategy for lifetime of project and beyond

•Reliable and efficient archive and distribution system

•Easy and efficient access to datasets by broader community

JOSS Data ManagementJOSS Data Management


UCAR Office Of ProgramsJoint Office for Science Support

Data Management ConsiderationsData Management Considerations

•Develop a consistent Data Policy

•Consider Data Types Formats and Documentation (metadata)

•Data Collection strategies (operations support, supplementary data)

•Real-time Requirements ( data calibration, intercomparison, and needs for operational decision making)

•Data Quality Control and Processing responsibilities

•Data Archive structure, functions and responsibilities

•Data Distribution protocols (user interface, metadata inventories, browse products)

•Collaboration with other programs and data centers (interoperability)

•Develop a consistent Data Policy

•Consider Data Types Formats and Documentation (metadata)

•Data Collection strategies (operations support, supplementary data)

•Real-time Requirements ( data calibration, intercomparison, and needs for operational decision making)

•Data Quality Control and Processing responsibilities

•Data Archive structure, functions and responsibilities

•Data Distribution protocols (user interface, metadata inventories, browse products)

•Collaboration with other programs and data centers (interoperability)



• Data Questionnaire• Data Management Plan• Project Web Pages• Real-time Operations and Data Ingest• Field Data Catalog• Quality control and dataset development• Interactive Data Archive and Distribution• Special media products (e.g. CD-Roms)

• Data Questionnaire• Data Management Plan• Project Web Pages• Real-time Operations and Data Ingest• Field Data Catalog• Quality control and dataset development• Interactive Data Archive and Distribution• Special media products (e.g. CD-Roms)

JOSS Data ServicesJOSS Data Services



JOSS Data ServicesData Questionnaire

JOSS Data ServicesData Questionnaire



• Project Web Pages• Project Web Pages



• Real-time Operations Support and Data Ingest

Satellite Ground Station

Forecast Products

Numerical Model Analyses

Operational Data Streams

• Real-time Operations Support and Data Ingest

Satellite Ground Station

Forecast Products

Numerical Model Analyses

Operational Data Streams





GTS METAR Observation Locations 6-10 November 2004

GTS METAR Observation Locations 6-10 November 2004


GTS Ship and Buoy Observation Locations 6-10 November 2004

GTS Ship and Buoy Observation Locations 6-10 November 2004


Coastal and Marine Observation Locations Coastal and Marine Observation Locations

* NDBC Buoy o TCOON (TX)* CMAN * TABS (TX)o NWLON + LUMCON (LA)o PORTS o WAVCIS (LA)

+ COMPS (FL)

Land Observation Locations Land Observation Locations

* ASOS o LAIS (LA)* AWOS + TX ET (TX)* AWOS modem o FAWN (FL)o GAEMN (GA) + TNRCC (TX)+ GFC (GA) o ALMESO (AL)


JOSS Data Services

• Field Data Catalog (WWW based)

Daily Mission Reports

Operations Summary

Status Reports

Data Analysis Products

Authoring Tools

JOSS Data Services

• Field Data Catalog (WWW based)

Daily Mission Reports

Operations Summary

Status Reports

Data Analysis Products

Authoring Tools



JOSS Data Services

• Quality control and dataset development

Processing of research-quality, high resolution data

Processing of standard operational data streams

Compatible formats

Composite datasets and data integration

JOSS Data Services

• Quality control and dataset development

Processing of research-quality, high resolution data

Processing of standard operational data streams

Compatible formats

Composite datasets and data integration



Composite Data Sets at UCAR/JOSS Composite Data Sets at UCAR/JOSS

A composite dataset is a collection (over some time period and region) of similar data (e.g. surface meteorological) from a variety of sources, put into a common format, and passed through a uniform quality control.

Why does UCAR/JOSS develop composites?

- Provides data in a uniform format with QC.

- Allows determination of network/site problems.

- Useful for model applications.

- Prevents duplication of effort.

A composite dataset is a collection (over some time period and region) of similar data (e.g. surface meteorological) from a variety of sources, put into a common format, and passed through a uniform quality control.

Why does UCAR/JOSS develop composites?

- Provides data in a uniform format with QC.

- Allows determination of network/site problems.

- Useful for model applications.

- Prevents duplication of effort.


ASOS

AWOS

ARM

Others (7)

OKMESO

ABLE

SCAN

LTER

HPCN

WTXMESO

NCAR (4)

MADIS

Ameriflux

KVII

GWMD

NMSU

CVT

HRLY

HRLY

HRLY

HRLY

HRLY

HRLY

HRLY

Visual

Visual

Visual

Visual

Visual

Visual

Visual

Visual

Visual

Visual

Visual

Visual

Visual

Visual

Visual

Visual

MERGE

Gross LimitChecks

HorizontalQC

Visual

Hourly Surface Composite

IHOP_2002 Hourly Surface Composite Development

ExamineStatistics

CVT

CVT

CVT

CVT

CVT

CVT

CVT

CVT

CVT

CVT

CVT

CVT

CVT

CVT

CVT



NCAR / RAF NetCDF

NOAA / AOC“standard” format

JOSS convert“type 5” toNetCDF

JOSS RAINEX Archive

Integrated RAINEX NetCDF

Aircraft Flight Level DatasetsAircraft Flight Level Datasets

JOSS DATA MANAGEMENT TOOLSJOSS DATA MANAGEMENT TOOLS

Field Catalog

In-field tool to ingest and display operational and preliminary research data and project documentation for

making real-time decisions and evaluating project progress

Features:

•Daily Mission Reports

•Operations Summary

•Facility Status Reports

•Data Analysis Products

•Authoring Tools

•Web-based access

Field Catalog

In-field tool to ingest and display operational and preliminary research data and project documentation for

making real-time decisions and evaluating project progress

Features:

•Daily Mission Reports

•Operations Summary

•Facility Status Reports

•Data Analysis Products

•Authoring Tools

•Web-based access

Interactive Data Archive/Access

System

Primary means for all project scientists and researchers to

browse and retrieve data from any JOSS-supported projects

Features:

• Long-term field project data archival and distribution

• Interactive data browsing, subsetting, and format conversion

• Remote Archive Access

• Value-added datasets

• Data documentation

Interactive Data Archive/Access

System

Primary means for all project scientists and researchers to

browse and retrieve data from any JOSS-supported projects

Features:

• Long-term field project data archival and distribution

• Interactive data browsing, subsetting, and format conversion

• Remote Archive Access

• Value-added datasets

• Data documentation



JOSS Data Services http://www.joss.ucar.edu/codiac/

• Interactive Data Archive and Distribution

Ease of Data Access (web based)

Data available via FTP, magnetic tape, CD-Rom

Metadata descriptions

Data Browse, Subsetting, Format Translation Capabilities

Quality Control

Distributed Data Archives

Customized data ordering and reformat

JOSS Data Services http://www.joss.ucar.edu/codiac/

• Interactive Data Archive and Distribution

Ease of Data Access (web based)

Data available via FTP, magnetic tape, CD-Rom

Metadata descriptions

Data Browse, Subsetting, Format Translation Capabilities

Quality Control

Distributed Data Archives

Customized data ordering and reformat



RAINEX Data Management Issues:

•Data Policy

•Data Management Plan

•Real Time Data Needs (model, satellite)

•Distributed Archive Responsibilities

•Special Data Processing (surface, upper-air, airborne radar aircraft flight-level)

RAINEX Data Management Issues:

•Data Policy

•Data Management Plan

•Real Time Data Needs (model, satellite)

•Distributed Archive Responsibilities

•Special Data Processing (surface, upper-air, airborne radar aircraft flight-level)


[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

http://www.joss.ucar.edu/http://www.joss.ucar.edu/


PORTS – Physical Oceanographic Real-Time System

NWLON – National Water Level Observation Network

TCOON – Texas Coastal Ocean Observation Network (TAMU – Corpus)

TABS – Texas Automated Buoy System (TAMU)

LUMCON – Louisiana Universities Marine Consortium

WAVCIS – Wave Current Surge Information System (LSU)

COMPS – Coastal Ocean Monitoring and Prediction System (USF)

GAEMN – Georgia Automated Environmental Monitoring Network (U of Georgia)

GFC – Georgia Forestry Commission

ALMESO – Alabama Mesonet (Auburn U)

LAIS – Louisiana Agriclimatic Information System (LSU)

TX ET – Texas Evapotranspiration Network (TAMU)

FAWN – Florida Automated Weather Network (U of Florida)

TNRCC – Texas Natural Resources Conservation Commission

Acronyms

Documents

RAINEX Data Management UCAR Joint Office for Science Support José Meitín Jim Moore Dick Dirks UCAR Joint Office for Science Support José Meitín Jim Moore