NOAA NWS CIO Report APSDEU-9 Tokyo, Feb 2009 Office of the Chief Information Officer NOAA’s National Weather Service Office of the Chief Information Officer

NOAA NWS CIO ReportAPSDEU-9

Tokyo, Feb 2009

NOAA NWS CIO ReportAPSDEU-9

Tokyo, Feb 2009

Office of the Chief Information Officer

NOAA’s National Weather Service

Office of the Chief Information Officer

NOAA’s National Weather Service

“Taking the pulse of the planet”“Taking the pulse of the planet”

NWSTG Functional OverviewNWSTG Functional Overview

The NWSTG is:

the central communication facility of the NWS;

the primary acquisition and distribution center for NWS data and products;

the primary acquisition and distribution center for international data and products to meet WMO, ICAO and bi-laterally agreed US requirements;

A major data exchange hub for NOAA and other agency data and products.

The NWSTG is:

the central communication facility of the NWS;

the primary acquisition and distribution center for NWS data and products;

the primary acquisition and distribution center for international data and products to meet WMO, ICAO and bi-laterally agreed US requirements;

A major data exchange hub for NOAA and other agency data and products.


NWSTG Functional OverviewNWSTG Functional Overview

The NWSTG includes

WMO Regional Telecommunication Hub (RTH) Washington

ICAO OPMET Databank

the ASOS Operations and Monitoring Center

• Operational oversight of U.S. federal automated surface observing systems

the AWIPS Network Control Facility

• AWIPS is the main NWS’ system which supports the NWS forecast and warning mission requirement

• The NCF is the central communications hub and technical support center for AWIPS

The NWSTG includes

WMO Regional Telecommunication Hub (RTH) Washington

ICAO OPMET Databank

the ASOS Operations and Monitoring Center

• Operational oversight of U.S. federal automated surface observing systems

the AWIPS Network Control Facility

• AWIPS is the main NWS’ system which supports the NWS forecast and warning mission requirement

• The NCF is the central communications hub and technical support center for AWIPS


Data Input to the NWSTGData Input to the NWSTG

MPLS

IP / Sockets

X.25

Asynchronous

FTP – http://weather.gov/tg/ftpingest.html

Email – http://weather.gov/tg/emailingest.html

Web - http://weather.gov/tg/bullguid.html

MPLS

IP / Sockets

X.25

Asynchronous

FTP – http://weather.gov/tg/ftpingest.html

Email – http://weather.gov/tg/emailingest.html

Web - http://weather.gov/tg/bullguid.html


Dissemination Systems Dissemination Systems

GTS

ISCS

EMWIN

Internet

GMDSS

NWS Telecom. Gateway

GTS

ISCS

EMWIN

Internet

GMDSS

NWS Telecom. Gateway

NWWS

NWR

LDAD

Family Of Services (FOS)

Interagency Connections

AWIPS SBN / NOAAPORT

NWWS

NWR

LDAD

Family Of Services (FOS)

Interagency Connections

AWIPS SBN / NOAAPORT

International National


Dissemination and Distribution

NWSTG

FOS

SBN/NOAAPORT• GOES •NCEP Product Suite

NWWS(10 sec)

WAN

NESDIS

Internet-BasedDissemination

EMWIN(> 1 min)

Telecommunications Operations Center

Satellite Legend:EMWIN

NOAAPORT

NWWS

Commercial Satellite

GOES

NWRWSR-88D

Field Offices

NCEP

DataServers

NCF

Media &Other

Customers

Public

SpecializedCustomers

Other

Agencies

LDAD

Designated Local

Customers

•Commercial Weather Services•Research Institutions•FAA, etc.

ISCS

GTS EMWIN

Private line Dissemination

• Observations

Imagery

WMO

ICAO



Geographically separatedbackup system

Replacement / BackupTG Description

Replacement / BackupTG Description

Worldwide

users• ICAO / WMO• Govt. agencies• Family of

services• Internet users• Foreign

countries• Emergency

mgrs

NWS users• Nat’l Centers for Environ. Prediction• NWS Regional Offices• Domestic/int’l observation & forecast offices• AWIPS

NWS Telecom Gateway• Located in Silver Spring• New message switchingsystem allows future upgrades• 2x upgradeability• Much improved response time • Redundancy ensuresuninterrupted service• Full configuration management

120GB/46600 GB

80GB/38200 GB

90GB/46375 GB

700GB/382000 GB

Daily Throughput/# of circuitsLegacy over Replacement


RTH WashingtonRTH Washington

Can route more than 50 routine messages per second with reliability for all dissemination to all of its users of 99.9 percent

Latency for high priority traffic of 10 seconds or less and routinely disseminates 1.2 terabytes of information per day

On Main Trunk Network (MTN) of the WMO Global Telecommunication System (GTS)

– GTS delivers tsunami data and warnings to connected MTN centers within two minutes

Can route more than 50 routine messages per second with reliability for all dissemination to all of its users of 99.9 percent

Latency for high priority traffic of 10 seconds or less and routinely disseminates 1.2 terabytes of information per day

On Main Trunk Network (MTN) of the WMO Global Telecommunication System (GTS)

– GTS delivers tsunami data and warnings to connected MTN centers within two minutes


Replacement NWSTGReplacement NWSTG

Full functional replacement of existing capabilities

Expanded capacity and capability

– Input data growth from 200 to 800 GB/day

– Output data growth from 800 to 2400 GB/day

Transition to new technology

– Middleware for internal transport

– Network-centric systems interconnectivity

– Central switching engine with relational database

– SAN and NAS storage solutions

Highly scaleable architecture

Hardware refresh

Full functional replacement of existing capabilities

Expanded capacity and capability

– Input data growth from 200 to 800 GB/day

– Output data growth from 800 to 2400 GB/day

Transition to new technology

– Middleware for internal transport

– Network-centric systems interconnectivity

– Central switching engine with relational database

– SAN and NAS storage solutions

Highly scaleable architecture

Hardware refresh


RTG/BTG PerformanceRTG/BTG Performance

Performance Metric Threshold System Availability (averaged monthly) 99.90% Warning Message Latency (averaged monthly) 10 seconds

Routine Message Latency (averaged monthly) 60 secondsDaily Traffic Volume (averaged monthly) 1.2TB

NWSTG Performance Results (Oct, 2008)

Performance Category Threshold ActualSystem Availability 99.90% 100%Warning Message Latency 10 seconds < 1 secondRoutine Message Latency 60 seconds < 1 secondDaily Traffic Volume 1.2TB 1.3TB

(plus .2TB in last year)

Performance Metric Threshold System Availability (averaged monthly) 99.90% Warning Message Latency (averaged monthly) 10 seconds

Routine Message Latency (averaged monthly) 60 secondsDaily Traffic Volume (averaged monthly) 1.2TB

NWSTG Performance Results (Oct, 2008)

Performance Category Threshold ActualSystem Availability 99.90% 100%Warning Message Latency 10 seconds < 1 secondRoutine Message Latency 60 seconds < 1 secondDaily Traffic Volume 1.2TB 1.3TB

(plus .2TB in last year)


Government Networking Requirements - NOAANetGovernment Networking Requirements - NOAANet

• IP-based networking solution

− Any-to-any connectivity

− High degree of bandwidth scalability

− Optimum redundancy and survivability

− IP convergence (i.e., voice, video and data over IP)

− High-end performance

• Network security remains paramount, particularly in light of today’s socio-political threats

• Segmentation from the public Internet

• Minimizes risk of security or privacy breaches


Logical DesignLogical Design


Primary / BackupNWSTG Access

Primary / BackupNWSTG Access


MS3

FEP #1Sockets

FEP #2X.25

FEP #8 MSD

FEP #3FTP

FEP #4Async

FEP #7Marta

FEP #6eMail

FEP #5Web

MS3 #1

#1

MS3

#2

#3

#4#8

#7

#6

#5

MS3 #2

MS3 #3

MS3 #4

MS3 #5

MS3 #6

MS3 #7

MS3 #8

Legend

Front EndProcessor (FEP)

MS3 Hub

Process FlowsWebSphere MQ Integrator

MS3

#1

MS3 inbound queue

MS3 customer queue

WebSphere MQ Queues

WeatherData

Source/Customers

Weather ProductPrefix

MS3 Message

Logical cluster queue

Weather Product

Weather ProductPrefix


Major Software ComponentsMajor Software Components

Core software components of the NWSTG replaced in 2006

Data switching software

– Core software rewritten and implemented in IBM WebSphere Message Broker

– Message Broker provides an application execution environment, threading, and fourth generation language support (ESQL), integrated with middleware services

Data transport infrastructure (middleware)

– Shared data stores replaced with IBM MQSeries Middleware

– Applications send data to each other without concern for target location

– Data delivery guaranteed even during failures

Data storage infrastructure (relational database)

– All data stores moved into Sybase relational database

Core software components of the NWSTG replaced in 2006

Data switching software

– Core software rewritten and implemented in IBM WebSphere Message Broker

– Message Broker provides an application execution environment, threading, and fourth generation language support (ESQL), integrated with middleware services

Data transport infrastructure (middleware)

– Shared data stores replaced with IBM MQSeries Middleware

– Applications send data to each other without concern for target location

– Data delivery guaranteed even during failures

Data storage infrastructure (relational database)

– All data stores moved into Sybase relational database


Internal RedundancyInternal Redundancy

Internal redundancy implemented for critical processes

Active – Warm Standby

– Standby system running at all times, system disks moved to standby system during failover

– EXAMPLES: Individual front-end processors, application servers

Active – Hot Standby

– Data replicated to Hot Standby system in realtime, automatic failover

– EXAMPLE: Sybase relational database servers

Active - Active

– Multiple systems active in parallel, data shared/routed between systems groups

– EXAMPLES – Switching system, HTTP/FTP server groups

Internal redundancy implemented for critical processes

Active – Warm Standby

– Standby system running at all times, system disks moved to standby system during failover

– EXAMPLES: Individual front-end processors, application servers

Active – Hot Standby

– Data replicated to Hot Standby system in realtime, automatic failover

– EXAMPLE: Sybase relational database servers

Active - Active

– Multiple systems active in parallel, data shared/routed between systems groups

– EXAMPLES – Switching system, HTTP/FTP server groups


Architecture OverviewArchitecture Overview


Architecture OverviewArchitecture Overview


System ExtensibilitySystem Extensibility

Virtualized server hardware platform selected for extensibility

Virtualization

– System resources (CPUs, Memory, Internal Storage, Network Interfaces, etc.) available within hardware frame

– Individual resources selected to create virtual servers

– Major storage implemented in Storage Area Network

– Virtual disks allocated to virtual servers

Extensibility

– Unused resources can be allocated to virtual servers, as needed

– Underused resources can be removed from a virtual server

– Virtual disks can be extended or reduced as storage requirements change


Virtualization

– System resources (CPUs, Memory, Internal Storage, Network Interfaces, etc.) available within hardware frame

– Individual resources selected to create virtual servers

– Major storage implemented in Storage Area Network

– Virtual disks allocated to virtual servers

Extensibility

– Unused resources can be allocated to virtual servers, as needed

– Underused resources can be removed from a virtual server

– Virtual disks can be extended or reduced as storage requirements change




Advanced Virtualization

– Network and SAN I/O interfaces can be shared through virtual I/O

– CPUs can be carved into fractional components down to 1/10th of a CPU

Advanced Extensibility

– Resources (CPU, Memory) included within system that were not purchased

– If additional resources are needed, resources are “turned on” via software key after purchase and are immediately available


Advanced Virtualization

– Network and SAN I/O interfaces can be shared through virtual I/O

– CPUs can be carved into fractional components down to 1/10th of a CPU

Advanced Extensibility

– Resources (CPU, Memory) included within system that were not purchased

– If additional resources are needed, resources are “turned on” via software key after purchase and are immediately available



(Currently migrating to P595 systems)


RTH Washington WIS approach


RTG/BTG upgrades made with knowledge of coming WIS/GISC requirements

– Production and Replicated core systems

• All functions not essential to mission critical operations (i.e. switching) execute against replicant

– SOA Approach to design

– Integral file, message and application switching

– RDBS core design

• ~ 2000 destinations defined

• ~ 400K defined products

• ~ 10M entries (rows) in switching table

– Many tables used for managing bulletin, file and report storage including dissemination and services metadata.

RTG/BTG upgrades made with knowledge of coming WIS/GISC requirements

– Production and Replicated core systems

• All functions not essential to mission critical operations (i.e. switching) execute against replicant

– SOA Approach to design

– Integral file, message and application switching

– RDBS core design

• ~ 2000 destinations defined

• ~ 400K defined products

• ~ 10M entries (rows) in switching table

– Many tables used for managing bulletin, file and report storage including dissemination and services metadata.




View Data Access & Retrieval DAR service as the only truly new service provision

– Plan to lay this service on to existed architecture

– Will break up DAR functions

• DAR catalog creation & update

• discovery & access servicing

– DAR will not execute on primary production system (at least in early stages of implementation

View Data Access & Retrieval DAR service as the only truly new service provision

– Plan to lay this service on to existed architecture

– Will break up DAR functions

• DAR catalog creation & update

• discovery & access servicing

– DAR will not execute on primary production system (at least in early stages of implementation




Develop metadata catalog first in conjunction with national & regional partners

– Regional WIGOS Development Project

• RA-IV Integrated Atmosphere Observing System

• With initial emphasis on RADAR observations

– Will leverage knowledge of partner expertise

– Will expand to all RTH (GISC) data holdings and holdings of appropriate Centers (RSMC/DCPC/NC) and willing partners.

– Work will be made available to all Members

– Will collaborate with WIS Project Team, WMO ETs and developers in other Regions

Develop metadata catalog first in conjunction with national & regional partners

– Regional WIGOS Development Project

• RA-IV Integrated Atmosphere Observing System

• With initial emphasis on RADAR observations

– Will leverage knowledge of partner expertise

– Will expand to all RTH (GISC) data holdings and holdings of appropriate Centers (RSMC/DCPC/NC) and willing partners.

– Work will be made available to all Members

– Will collaborate with WIS Project Team, WMO ETs and developers in other Regions


Meteorological Assimilation Data Ingest System (MADIS) Transition to Operations

SFC-LANDSFC-MARINE

U/A-IN SITU

U/A-REMOTESENSING

SATELLITE

GRIDSMETADATA

MADISCollection,

QC, andDistribution

InformationBases

(QCed Datasets)

Quality ControlInformation

(Data QC Flags)And Metadata

PM 12/12/05


MADIS Ingests and Performs QC on Meteorological Observational DataMADIS Ingests and Performs QC on Meteorological Observational Data

Meteorological Surface

– METAR

– Airways

– Maritime

– Modernized NWS Cooperative Observer

– UrbaNet

– Integrated Mesonet

• State DOT

• Mesonets

• AWS

RSAS Surface Grids

Meteorological Surface

– METAR

– Airways

– Maritime

– Modernized NWS Cooperative Observer

– UrbaNet

– Integrated Mesonet

• State DOT

• Mesonets

• AWS

RSAS Surface Grids

NOAA Profiler Network

Hydrological Surface

Automated Aircraft

Multi-Agency Profiler

Cooperative Agency Profiler

Radiosonde

Radiometer

Satellite Wind

– NOAA GOES Products

Satellite Radiances and Soundings

– NOAA POES

Snow

NOAA Profiler Network

Hydrological Surface

Automated Aircraft

Multi-Agency Profiler

Cooperative Agency Profiler

Radiosonde

Radiometer

Satellite Wind

– NOAA GOES Products

Satellite Radiances and Soundings

– NOAA POES

Snow

“Taking the pulse of the planet”“Taking the pulse of the planet”TOC Environment

Gateway

MADIS-T

FTP

Ex

tern

al I

SP

LDM

FTP

Storage(SAN)

Automated QC processing

Gateway Customers (NCDC, NCF, GTS etc)

Data Processing

LDM

NOAA Customers

Non-NOAAObservations

LDM

Integrated Datasets

Web Server

Internet

Web

Ser

ver

?

?

?Individual Data

Products

Tunnel


Weather Impacts on SocietyWeather Impacts on Society


Data, Products, and Services to:

Greatly reduce loss of life and injury

Enable communities to mitigate property loss well in advance of threatening conditions

Alert economic sectorsto environmental riskswith sufficient lead timeto limit or avoid impacts

Data, Products, and Services to:

Greatly reduce loss of life and injury

Enable communities to mitigate property loss well in advance of threatening conditions

Alert economic sectorsto environmental riskswith sufficient lead timeto limit or avoid impacts

Where We’re HeadedWhere We’re Headed


Integrated Surface Observing System (ISOS)

– ASOS

– Cooperative Observers (COOP)

– COOP-Modernization

– National Mesonet

Integrated Surface Observing System (ISOS)

– ASOS

– Cooperative Observers (COOP)

– COOP-Modernization

– National Mesonet



Climate Services

Enhance products and services with climate change information

Local outlooks for temperature, precipitation, El Niño/La Niña

Seamless Climate, Water, and Weather products and services

Climate Services

Enhance products and services with climate change information

Local outlooks for temperature, precipitation, El Niño/La Niña

Seamless Climate, Water, and Weather products and services

• Reduce losses from wildfires

• Save lives and reduce property damages caused by major climate anomalies including drought

• Meet new needs with end-to-end suite of climate products and services

• Reduce losses from wildfires

• Save lives and reduce property damages caused by major climate anomalies including drought

• Meet new needs with end-to-end suite of climate products and services



Water Resource Services

Provide high-resolution water quantity, quality, and soil moisture forecasts

– Emergency and resource managers mitigate losses for conditions ranging from droughts to floods

Water Resource Services

Provide high-resolution water quantity, quality, and soil moisture forecasts

– Emergency and resource managers mitigate losses for conditions ranging from droughts to floods

• Enables NOAA to meet our Nation's growing needs for water forecasts • Provides forecasts for consumption resources• Provides important resource protection capabilities

• Enables NOAA to meet our Nation's growing needs for water forecasts • Provides forecasts for consumption resources• Provides important resource protection capabilities



Ecosystem Impact Information

Provide forecasts of weather, water, and climate impacts for management decisions

– Management decisions reflect relationships among humans, nonhuman species, and the environments in which they live

Ecosystem Impact Information

Provide forecasts of weather, water, and climate impacts for management decisions

– Management decisions reflect relationships among humans, nonhuman species, and the environments in which they live

• Benefits through innovative approaches to spill preparedness, response, damage assessments and restoration

• NOAA contributes approximately $75 million annual to the U.S. economy.

• Benefits through innovative approaches to spill preparedness, response, damage assessments and restoration

• NOAA contributes approximately $75 million annual to the U.S. economy.



Forecast Uncertainty Information

Integral and essential part of all forecasts

Enterprise-wide partnership to generate and communicate forecast uncertainty to decision makers and public

Expressed in terms of probabilities

Forecast Uncertainty Information

Integral and essential part of all forecasts

Enterprise-wide partnership to generate and communicate forecast uncertainty to decision makers and public

Expressed in terms of probabilities

• Users decide whether to take action and appropriate level of response

• Thresholds unique to decision maker – based on mission risk

• Users decide whether to take action and appropriate level of response

• Thresholds unique to decision maker – based on mission risk



Future Activities – RTH/GISC Washington

Future Activities – RTH/GISC Washington

Continuing tech refresh

Differences between TG & BTG

– Not affecting functional backup

– Is a maintenance issue

Next round of Continuity of Operations Planning (COOP)

– Backup strategy – hot, warm. cold?

Continuing tech refresh

Differences between TG & BTG

– Not affecting functional backup

– Is a maintenance issue

Next round of Continuity of Operations Planning (COOP)

– Backup strategy – hot, warm. cold?


Future ActivitiesFuture Activities GTS – WIS evolution

– MTN & GISC core network?

– HAZNET VPN

GISC planning and Implementation

WIGOS integral with WIS

Our national strategy is integral with the regional strategy

WDP - proof of concept prior to wider architecture implementation

GTS – WIS evolution

– MTN & GISC core network?

– HAZNET VPN

GISC planning and Implementation

WIGOS integral with WIS

Our national strategy is integral with the regional strategy

WDP - proof of concept prior to wider architecture implementation

A flexible services oriented framework based on a regular requirements review process responding to and feeding into the strategic planning process, at national, regional and WMO levels


IGDDSGTS

Data pull

Data push

WIS

WIS DATA-COMMUNICATIONS FUNCTIONS AND SERVICES

WIS/GTS: for time and operation-critical data & products WIS/IGDDS: for space-based data & productsWIS/DAR: data discovery, access and retrieval

DAR

HAZ-NETVPN

How do we serve the People?


Food for thoughtFood for thought

Services Delivery

Data Processing

Data Exchange

Observations

Danger of fracturing or stove piping GEO, WMO & other data sharing initiatives

Services Delivery

Data Processing

Data Exchange

Observations

Danger of fracturing or stove piping GEO, WMO & other data sharing initiatives


ChallengesChallenges

To peer into the future and have a vision of future needs

The volume of observations from satellites, RADARs, mesonets, and non-traditional domains continues to grow well beyond levels at which we are able to keep up with.

NWP presents a two fold problem to the data exchange community.

Insatiable Hunger - Profuse Output

What demands will climate services and multi-domain models bring?

To peer into the future and have a vision of future needs

The volume of observations from satellites, RADARs, mesonets, and non-traditional domains continues to grow well beyond levels at which we are able to keep up with.

NWP presents a two fold problem to the data exchange community.

Insatiable Hunger - Profuse Output

What demands will climate services and multi-domain models bring?


ChallengesChallenges

I want to scare you, at least a little bit

Our community is struggling to handle these demands

We are often so busy planning for today’s needs that future requirements are a best a casual afterthought.

As a result we stay behind curve , always playing the catch-up game.

We need to embrace new technologies and develop adaptive, agile exchange architectures

Services Orientation

SCALABILITY EXPANDABILITY EXTENSIBILITY

I want to scare you, at least a little bit

Our community is struggling to handle these demands

We are often so busy planning for today’s needs that future requirements are a best a casual afterthought.

As a result we stay behind curve , always playing the catch-up game.

We need to embrace new technologies and develop adaptive, agile exchange architectures

Services Orientation

SCALABILITY EXPANDABILITY EXTENSIBILITY

Documents

NOAA NWS CIO Report APSDEU-9 Tokyo, Feb 2009 Office of the Chief Information Officer NOAA’s National Weather Service Office of the Chief Information Officer