Kelvin K. Droegemeier University of Oklahoma Internet2 Spring Meeting 20 April 2004

The The Collaborative Collaborative

Radar Acquisition Field Radar Acquisition Field Test (CRAFT): A Unique Public-Test (CRAFT): A Unique Public-Private Partnership in Mission-Private Partnership in Mission-

Critical Data DistributionCritical Data DistributionKelvin K. DroegemeierKelvin K. DroegemeierUniversity of OklahomaUniversity of Oklahoma

Internet2 Spring MeetingInternet2 Spring Meeting20 April 200420 April 2004

NCDC

http://www.nsf.gov/start.htm

http://www.roc.noaa.gov/

http://my.unidata.ucar.edu/

http://abilene.internet2.edu/

NEXRAD Doppler Radar NetworkNEXRAD Doppler Radar Network

The Moore, Oklahoma TornadoThe Moore, Oklahoma Tornadoof 3 May 1999of 3 May 1999

Storm A

Storm B

Storm D

Storm E

Storm C

Storm H

Storm G

100 kmrange ring

Damage Paths Ground Truth - 53 Tornadoes

Courtesy D. Zittel

Trimmed Detections and Ground Truth Damage Paths

Hits (142)

Misses (25)

FAs (21) Courtesy D. Zittel

USA Today

Doppler Weather Radar: Doppler Weather Radar: Measuring the Wind SpeedMeasuring the Wind Speed

A Unique Tool for A Unique Tool for Sampling the 3D Sampling the 3D

Atmosphere in DetailAtmosphere in Detail The radar scans a 3-D conical volume of the The radar scans a 3-D conical volume of the

atmosphere every 5-6 minutes through a series atmosphere every 5-6 minutes through a series of 360 degree sweeps…of 360 degree sweeps…

… … at several elevation anglesat several elevation angles

A Unique Tool for A Unique Tool for Sampling the 3D Sampling the 3D

Atmosphere in DetailAtmosphere in Detail

NEXRAD Facts and NEXRAD Facts and FiguresFigures

159 radars (142 in the Continental US)159 radars (142 in the Continental US)– 121 National Weather Service radars121 National Weather Service radars– 26 Department of Defense radars26 Department of Defense radars– 12 Federal Aviation Administration radars12 Federal Aviation Administration radars

NEXRAD Data TypesNEXRAD Data Types Archive Level IArchive Level I

– analog, time domain outputanalog, time domain output Level II dataLevel II data

– digital base data at full precision and all elevation anglesdigital base data at full precision and all elevation angles Archive Level IIIArchive Level III

– digital products + lowest four elevation anglesdigital products + lowest four elevation angles– reduced precision: data are binnedreduced precision: data are binned

Archive Level IVArchive Level IV

Archive Level IArchive Level I– analog, time domain outputanalog, time domain output

Level II dataLevel II data– digital base data at full precision and all elevation anglesdigital base data at full precision and all elevation angles

Archive Level IIIArchive Level III– digital products + lowest four elevation anglesdigital products + lowest four elevation angles– reduced precision: data are binnedreduced precision: data are binned

Archive Level IVArchive Level IV

NEXRAD Data TypesNEXRAD Data Types

The Value – Hazard DetectionThe Value – Hazard Detection

The Value – Real Time 3D The Value – Real Time 3D AnalysesAnalyses

http://caps.ou.edu

Carbone et al. (2002)

The Value – Fundamental PredictabilityThe Value – Fundamental Predictability

The Value – Storm ClimatologiesThe Value – Storm Climatologies

The Value – Computer The Value – Computer Prediction ModelsPrediction Models

6 pm 7 pm 8 pmR

adar

Hourly Radar Observations(Fort Worth Shown by the Pink Star)

6 pm 7 pm 8 pmR

adar

Com

pu

ter

For

ecas

t

2 hr 3 hr 4 hr

Fcs

t w

/o R

adar 2 hr 3 hr 4 hr

6 pm 7 pm 8 pmR

adar

Space Shuttle Columbia Debris TrailSpace Shuttle Columbia Debris Trail

Bird Migration StudiesBird Migration Studies

NEXRAD data are *THE* mostvaluable data for mesoscale meteorology…

but they were not available in real timeto the academic and corporate communities

The Collaborative Radar The Collaborative Radar Acquisition Field Test Acquisition Field Test

(CRAFT)(CRAFT) Begun in fall, 1998 as a Begun in fall, 1998 as a collaborative R&D projectcollaborative R&D project

to demonstrate the real time compression and to demonstrate the real time compression and Internet-based transmission of WSR-88D base Internet-based transmission of WSR-88D base (Level II) data(Level II) data

PhilosophyPhilosophy: Leverage existing infrastructures : Leverage existing infrastructures (Internet), public-domain software (Unidata LDM), (Internet), public-domain software (Unidata LDM), and secure, low-cost services (56K phone lines)and secure, low-cost services (56K phone lines)

Short-term goalsShort-term goals: Storm-scale research and : Storm-scale research and improvement of NCDC archive capabilityimprovement of NCDC archive capability

Long-term goalLong-term goal: National real-time capability for : National real-time capability for operations, research, and private industryoperations, research, and private industry

Initiated and led by the OU Center for Initiated and led by the OU Center for Analysis and Prediction of Storms (CAPS) Analysis and Prediction of Storms (CAPS)

National Severe Storms Laboratory (NSSL)National Severe Storms Laboratory (NSSL) NOAA WSR-88D Radar Operations Center NOAA WSR-88D Radar Operations Center

(ROC)(ROC) UCAR Unidata ProgramUCAR Unidata Program National Climatic Data Center (NCDC)National Climatic Data Center (NCDC)

University of WashingtonUniversity of Washington

TREMENDOUS support from the US WeatherTREMENDOUS support from the US WeatherResearch ProgramResearch Program

Coordinating Coordinating OrganizationsOrganizations

Foundation #1: NSSL’s Foundation #1: NSSL’s RIDDSRIDDS

Courtesy NSSL

Foundation #2: Unidata Local Foundation #2: Unidata Local Data Manager (LDM) Data Manager (LDM)

Source

LDM

Source

Source

LDM LDM

LDMLDM

LDM LDM

LDM

LDM

Internet

Courtesy D. Fulker, Unidata

Foundation #3: Foundation #3: Data Compression – BZIP2Data Compression – BZIP2

The Internet

Adaptation to LDM by Harry Edmon, U. of Washington

Foundation #4: The Foundation #4: The InternetInternet

TheInternet

Invention courtesy A. Gore

CommodityNGI

Internet2Abilene

Foundation #5Foundation #5 Incredible cooperation, collegiality, and Incredible cooperation, collegiality, and

agency supportagency support

NCDC

http://www.nsf.gov/start.htm

http://www.roc.noaa.gov/

Original Technical Original Technical StrategyStrategy

RIDDS

WSR-88D

At the radar site


RIDDS Linux PC + LDM + Compression

($1000)

WSR-88D

At the radar site



($1000)

Router ($2000)

WSR-88D

At the radar site



($1000)

Dedicated 56K line, T1, ($2000 - $6000/year)

or NWS LANRouter ($2000)

WSR-88D

At the radar site

LDM Server (OU)



($1000)

LDM Server (OU)

Dedicated 56K line, T1, ($2000 - $6000/year)

or NWS LANRouter ($2000)

WSR-88D

At the radar site

Users (LDM)

Internet or Abilene

Slow But Sure GrowthSlow But Sure Growth

February, 2000

BZIP2 off-the-shelf compression algorithmBZIP2 off-the-shelf compression algorithm– maximum 16:1 in clear air; 8:1 in the worst case; maximum 16:1 in clear air; 8:1 in the worst case;

average of about 12:1average of about 12:1 Maximum aggregate data rate is 15 Maximum aggregate data rate is 15

Mbytes/radar/5 min uncompressed = 48 Mbits/sec Mbytes/radar/5 min uncompressed = 48 Mbits/sec compressed for compressed for entire national networkentire national network– bandwidth isn’t the main issue, but rather bandwidth isn’t the main issue, but rather

topology, quality of service, and data storagetopology, quality of service, and data storage 56K lines found to be inadequate in some cases 56K lines found to be inadequate in some cases

(squall lines) – maybe 10% of time(squall lines) – maybe 10% of time– Exceed available bandwidth by 10-20%Exceed available bandwidth by 10-20%– Can create latencies of 10s of minutes (more later)Can create latencies of 10s of minutes (more later)– Up to 26% improvement in compression possible in Up to 26% improvement in compression possible in

BZIP2, but requires changing code at user endBZIP2, but requires changing code at user end– A pre-processing algorithm (NSSL) provides 10% A pre-processing algorithm (NSSL) provides 10%

improvement – no changes in BZIP requiredimprovement – no changes in BZIP required

Technical HighlightsTechnical Highlights

Humble Beginnings…Humble Beginnings…

January, 1999

Eight More in 1 Year Eight More in 1 Year January, 2001

14 Radars

s fn

0

20

40

60

80

100

120

0 5 10 15 20 25 30

Radar Data LatencySEAAMACMBFLGFTWSRPTULLBBFSMOKCTCS

Hour

Latency DataLatency Data

The Idea Catches HoldThe Idea Catches HoldJanuary, 2002

The NWS Makes a Bold The NWS Makes a Bold DecisionDecision

Fall 2004

Regional Collection Concept Regional Collection Concept Proposed 5 Yrs AgoProposed 5 Yrs Ago

NCEP

NCDC

Reiterated Fall 2002Reiterated Fall 2002

NWS is implementing a data delivery system NWS is implementing a data delivery system based upon the CRAFT concept (regional hubs, based upon the CRAFT concept (regional hubs, Abilene as the backbone) Abilene as the backbone)

This infrastructure could become This infrastructure could become THETHE NWS data NWS data and model grid delivery system of the futureand model grid delivery system of the future

Current StatusCurrent Status

We’ve Come a Long Way!We’ve Come a Long Way!Today – 123 Radars!!

– Initiated after 2001 Stakeholders WorkshopInitiated after 2001 Stakeholders Workshop– NWS agreed to let OU provide real time Level II data to NWS agreed to let OU provide real time Level II data to

private companies and others (academia), private companies and others (academia), as R&D as R&D partnerspartners, in exchange for latency and other information , in exchange for latency and other information to help evaluate system performanceto help evaluate system performance

– Any group could participateAny group could participate– Provided an opportunity for private sector partners to Provided an opportunity for private sector partners to

gain experience using Level II data and develop new gain experience using Level II data and develop new capabilities with themcapabilities with them

– Had 6 private sector partnersHad 6 private sector partners– Companies agreed to the collaborative nature of the Companies agreed to the collaborative nature of the

project, i.e., project, i.e., everyone wins when everyone contributeseveryone wins when everyone contributes

Private Sector Joins the Private Sector Joins the EffortEffort

The Benefits – Industry The Benefits – Industry DevelopmentDevelopment

Baron Services, Inc.

A True Public-Private A True Public-Private PartnershipPartnership

Recent NOAA + Pvt Sector Money $3.0M

123 NEXRAD radars are delivering Level II data to University of 123 NEXRAD radars are delivering Level II data to University of Oklahoma in real timeOklahoma in real time– Subsequent distribution to more than 30 users including NCDC, NWS HQ, UCAR Subsequent distribution to more than 30 users including NCDC, NWS HQ, UCAR

Unidata, numerous universities and NOAA laboratories, and several (paying) private Unidata, numerous universities and NOAA laboratories, and several (paying) private companiescompanies

Extremely high reliability (99+%) with latency less than 10 secExtremely high reliability (99+%) with latency less than 10 sec Top-tier nodes announced during week of 12 April 2004Top-tier nodes announced during week of 12 April 2004

– University of OklahomaUniversity of Oklahoma– Purdue UniversityPurdue University– ERCERC

Current StatusCurrent Status

IRaDS will continue to collect and analyzenetwork information for R&D

Data Ingest is Sized to Handle Data Ingest is Sized to Handle Future NEXARD ImprovementsFuture NEXARD Improvements

EnhancementEnhancement Factor of Data Factor of Data Flow Increase Flow Increase (Relative to (Relative to

August 2002)August 2002)

Possible Possible ImplementatioImplementatio

n Daten Date

Cumulative Cumulative Data Flow Data Flow Increase Increase

(Relative to (Relative to August 2002)August 2002)

4.1 min VCP4.1 min VCP 1.241.24 Jan 2003Jan 2003 1.241.24

TCP/IP TCP/IP Wideband Wideband InterfaceInterface

1.001.00 July 2004July 2004 1.241.24

0.25 km 0.25 km Reflectivity Reflectivity DataData

1.561.56 October 2004October 2004 1.931.93

0.5 deg 0.5 deg Azimuth Azimuth SamplingSampling

2.002.00 January 2005January 2005 3.863.86

Doppler Data Doppler Data to End of 2to End of 2ndnd TripTrip

1.491.49 October 2005October 2005 5.765.76

Clutter Filters Clutter Filters and SNR and SNR ThrshldThrshld

1.971.97 January 2005January 2005 11.3511.35

Dual Dual PolarizationPolarization

2.322.32 January 2008January 2008 26.3526.35

OU and NSSL have worked with FAA Mike Monroney OU and NSSL have worked with FAA Mike Monroney Aeronautical Center in OKC to transmit TDWR data in real Aeronautical Center in OKC to transmit TDWR data in real timetime

Can be extended to all TDWR, ASR and ARSR systemsCan be extended to all TDWR, ASR and ARSR systems The idea of using Abilene to transmit data was suggested The idea of using Abilene to transmit data was suggested

by Lincoln Laboratory 2 years after CRAFT was underwayby Lincoln Laboratory 2 years after CRAFT was underway

Weber (2000)

Addition of FAA Air Traffic Addition of FAA Air Traffic RadarsRadars

Baron Services centrally collects data now (Huntsville, AL)Baron Services centrally collects data now (Huntsville, AL) TV stations only scan lowest elevation angle, which would TV stations only scan lowest elevation angle, which would

be very useful in biological target ID and trackingbe very useful in biological target ID and tracking

Possible Addition of TV Station Possible Addition of TV Station RadarsRadars

Baron Services TV Radar Sites

Linked Environments for Linked Environments for Atmospheric Discovery (LEAD)Atmospheric Discovery (LEAD)

A Cyberinfrastucture for Mesoscale Meteorology A Cyberinfrastucture for Mesoscale Meteorology Research and EducationResearch and Education

David McLaughlin

greeting. introduce team present.explain why chandra isn't there.

Analysis/Assimilation

Quality ControlRetrieval of Unobserved

QuantitiesCreation of Gridded Fields

Prediction

PCs to Teraflop Systems

Product Generation, Display,

Dissemination

End Users

NWSPrivate Companies

Students

Traditional PredictionTraditional Prediction

OBSERVATIONSOBSERVATIONS

Radar DataRadar DataMobile MesonetsMobile Mesonets

Surface ObservationsSurface ObservationsUpper-Air BalloonsUpper-Air BalloonsCommercial AircraftCommercial Aircraft

Geostationary and Polar Geostationary and Polar Orbiting SatelliteOrbiting Satellite

Wind ProfilersWind ProfilersGPS SatellitesGPS Satellites




Prediction



Dissemination

End Users


Students

Traditional PredictionTraditional Prediction






The Process is Entirely Serialand Static (Pre-Scheduled): No Response to the Weather!

The Process is Entirely Serialand Static (Pre-Scheduled): No Response to the Weather!




Prediction



Dissemination

End Users


Students

The LEAD Vision: No Longer Serial or StaticThe LEAD Vision: No Longer Serial or Static









Prediction



Dissemination

End Users


Students






The LEAD Vision: No Longer Serial or StaticThe LEAD Vision: No Longer Serial or Static

MesoscaleWeather

LEAD: Users INTERACTING with WeatherLEAD: Users INTERACTING with Weather

NWS National Static Observations & Grids

MesoscaleWeather



MesoscaleWeather

Local Observations


UsersADaM ADAS Tools


MesoscaleWeather

Local Observations


http://www.caps.ou.edu/ARPS/index_flash.html



MesoscaleWeather

Local Observations

Local Physical Resources

Remote Physical (Grid) Resources

Virtual/Digital Resources and Services

MyLEADPortal


Interaction Level I




Experimental DynamicObservations

MesoscaleWeather

Local Observations

MyLEADPortal

Local Physical Resources

Remote Physical (Grid) Resources

Virtual/Digital Resources and Services


Interaction Level II


http://www.nssl.noaa.gov/smartradars/SMART%20radar%20(SR-1)%20at%20NSSL

New NSF Engineering Research Center New NSF Engineering Research Center for Adaptive Sensing of the Atmosphere for Adaptive Sensing of the Atmosphere

(CASA)(CASA) UMass/Amherst and OU are lead partnersUMass/Amherst and OU are lead partners Total funding over 10 years: Total funding over 10 years: $100M$100M Concept: inexpensive, phased array Doppler Concept: inexpensive, phased array Doppler

radars on cell towers and buildingsradars on cell towers and buildings Adaptive dynamic sensing of multiple targets Adaptive dynamic sensing of multiple targets

including aircraftincluding aircraft

Thank You Abilene Thank You Abilene and Internet2!!!and Internet2!!!

http://abilene.internet2.edu/

Documents

Kelvin K. Droegemeier University of Oklahoma Internet2 Spring Meeting 20 April 2004