Grid resources for NWP models at national level in Korea

Grid resources for NWP models at Grid resources for NWP models at national level in Korea national level in Korea

Korean Meteorological Administration

2004. 1. 292004. 1. 29

Super Computer CenterSuper Computer Center

Korea Meteorological Administration Korea Meteorological Administration

APANAPAN

ContentsContents

What the operational weather forecasting isWhat the operational weather forecasting is

Why the operational centers need Grid Technology.Why the operational centers need Grid Technology.

What Grid activities are in the Meteorological field.What Grid activities are in the Meteorological field.

What we have and will do with IT infrastructure.What we have and will do with IT infrastructure.


APANAPAN

What the operational weather What the operational weather forecasting is…forecasting is…


APANAPAN

observationobservation

Technology for the weather Technology for the weather forecastingforecasting

collectioncollection Diss.Diss.Anal.& WFAnal.& WF

NumericalWeatherPrediction

Infrastructure for Earth SystemInfrastructure for Earth System


APANAPAN

Nature

Korean Meteorological

Administration

DATAPRODUCERS

GlobalObserving

System

GOS

DATACONVEYORS

GlobalTelecommunication

System

GTS

DATA USERS

Global DataProcessing

System

GDPS

World Meteorological Organization (187members)World Meteorological Organization (187members)

0

500

1000

1500

2000

2500

3000

3500

4000

GB

/Day

NWP Messages NEXRAD Prod NEXRAD Base

APANAPAN

Current GTS will be replaced with FWIS until 2010Current GTS will be replaced with FWIS until 2010


APANAPAN

연구배경

• Global Scale Study => climatology with Atmos., Ocean, land surface, etc Global Scale Study => climatology with Atmos., Ocean, land surface, etc

• Global Observation and sharing informationGlobal Observation and sharing information• Co-work with international society is importantCo-work with international society is important

Some Characteristics of Earth ScienceSome Characteristics of Earth Science

global science with huge observation data and high performance computing

• Dynamic Equation SetDynamic Equation Set • Numerical RepresentationNumerical Representation

• Super ComputingSuper Computing


APANAPAN

Why we need GridWhy we need Grid


APANAPAN

Archived data volume and the Moore law Archived data volume and the Moore law

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

Te

rao

cte

ts

volume loi de Moore

Source: CAS2k3 , MetroFranceSource: CAS2k3 , MetroFrance


APANAPAN

Source:CAS2K3 DKRZSource:CAS2K3 DKRZ

Daily transfer volumeDaily transfer volume


APANAPAN

0

200

400

600

800

1000

[TB]

Sep

01

Dez 0

1

Mrz

02

Ju

n 0

2

Sep

02

Dez 0

2

Mrz

03

Ju

n 0

3

archive capacity

original

duplicates

0

200

400

600

800

1000

[TB]

Sep

01

Dez 0

1

Mrz

02

Ju

n 0

2

Sep

02

Dez 0

2

Mrz

03

Ju

n 0

3

archive capacity

original

duplicates

0

500

1000

1500

2000

2500

3000

[GB]

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

daily transfer volume

fetch

store

0

500

1000

1500

2000

2500

3000

[GB]

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

daily transfer volume

fetch

store


APANAPAN

Source:CAS2K3 DKRZSource:CAS2K3 DKRZ

Data Size in Operational Center ( DWD )Data Size in Operational Center ( DWD )


APANAPAN

Forget Modeling: How About Case Forget Modeling: How About Case Study Research?Study Research?

Exhaustive study of one or more selected observed weather events, Exhaustive study of one or more selected observed weather events,

sometimes involving the use of models and always involving lots of data!sometimes involving the use of models and always involving lots of data!

We spend We spend 70%70% of our time dealing with of our time dealing with data logisticsdata logistics, i.e., searching, , i.e., searching,

formatting, acquiring, reading, reducing, merging onto common grids and formatting, acquiring, reading, reducing, merging onto common grids and

projections, visualizing, etc…projections, visualizing, etc…

- Radar, satellite, surface nets, GPS, upper-air balloon, commercial - Radar, satellite, surface nets, GPS, upper-air balloon, commercial

aircraft, wind profiler, etc…aircraft, wind profiler, etc…

And only And only ~30%~30% of our time is spent actually studying the event! of our time is spent actually studying the event!

Source: LEAD, Prof. KelvinSource: LEAD, Prof. Kelvin


APANAPAN

WGNE Table of Operational NWP Centres WGNE Table of Operational NWP Centres (November 2002; ref. Kamal Puri)(November 2002; ref. Kamal Puri)

Source : CAS2k3 M.Naughton BOMSource : CAS2k3 M.Naughton BOM


APANAPAN

What Grid activities are in the What Grid activities are in the Meteorological fieldMeteorological field


APANAPAN

The Grid and Meteorology:The Grid and Meteorology:OpportunitiesOpportunities

Inter-personal collaborationInter-personal collaboration

- E.g., Access Grid, CHEF- E.g., Access Grid, CHEF

On-demand access to simulation modelsOn-demand access to simulation models

- E.g., Espresso- E.g., Espresso

Access to, and integration of, data sourcesAccess to, and integration of, data sources

- E.g., Earth System Grid- E.g., Earth System Grid

Dynamic, virtual computing resourcesDynamic, virtual computing resources

- “Metacomputing”- “Metacomputing”

Integration of all of the aboveIntegration of all of the above

- Collaborative, computationally intensive analysis of large quantities of - Collaborative, computationally intensive analysis of large quantities of

online data online data Source : Ian Foster, 16Source : Ian Foster, 16thth APAN Meeting APAN Meeting


APANAPAN

European GridEuropean Grid

www.eurogrid.orgwww.eurogrid.orgKorean Meteorological

Administration

Computational grid exampleComputational grid exampleAPANAPAN


Administration

Computational grid example(cont.)Computational grid example(cont.)APANAPAN


Administration

APANAPAN

Earth System GridEarth System Grid2

Earth System Grid ANL, LBNL, LLNL, NCAR, ORNL, USC/ ISI

Why Climate Changeis a Grid Problem

Climate modeling Data sources: high data rates

– 2MB/s (100 GF/s)– Multiple, high-resolution, multi-component runs: 100

MB/s (@5 TF/s) => 3 PB/yr

User community: large and diverse– O(100+) laboratories, centers, universities– Highly multidisciplinary– International connections

Data analyses: sophisticated, intensive– Multiplicity of tools used for an analyses– Intercomparison of data in different formats


Data grid exampleData grid exampleAPANAPAN

http://lead.ou.eduhttp://lead.ou.edu


APANAPAN


Administration

APANAPAN LEAD ConceptLEAD Concept


APANAPAN LEAD Data CloudLEAD Data Cloud

What we haveWhat we have


APANAPAN

High Speed Network - KORENHigh Speed Network - KOREN


APANAPAN

High Speed Network - KREONETHigh Speed Network - KREONET


APANAPAN

High Speed Network – International LinkHigh Speed Network – International Link


APANAPAN


Administration

APANAPAN KMA Intra NetworkKMA Intra Network

Filer2

Filer1

BackupSystem

Front-EndSystem

GigabitSwitch2

GigabitSwitch1

SX-SX-5/16A5/16A

SX-5/12ASX-5/12A

XYLANF/W

KORENKOREN KMA KMA BackboneBackbone

Fast.EFast.E

MRIMRI

Giga.EGiga.E

G-NEtG-NEt

KMA HPC and NetworkKMA HPC and Network

KREONET


APANAPAN

What we will doWhat we will do


APANAPAN

NEXT KMA HPC SystemNEXT KMA HPC System54 times faster than the current system54 times faster than the current systemInitial stage : 6 times 4Q 2004Initial stage : 6 times 4Q 2004Final stage : 54 times 4Q 2005Final stage : 54 times 4Q 2005More than 80M$ for next 5 yearsMore than 80M$ for next 5 years

Plan for the National Data Center : from 2004Plan for the National Data Center : from 2004 To collect Obs. DataTo collect Obs. Data To disseminate Weather Information and ProductTo disseminate Weather Information and Product

Digital Weather ForecastDigital Weather Forecast Fine Mesh Weather Grid InformationFine Mesh Weather Grid Information Real time DisseminationReal time Dissemination


APANAPAN

KMA Met Satellite at 2008KMA Met Satellite at 2008

Representative of Group on Earth Observation Representative of Group on Earth Observation

V-GISC V-GISC

Future WMO Information SystemFuture WMO Information System

E-Science for Earth SystemE-Science for Earth System

APEC Climate Network APEC Climate Network


APANAPAN

Test for the collaboration in the field of Met. with High Speed Network Test for the collaboration in the field of Met. with High Speed Network

targettargetData Grid Data Grid

For sharing For sharing

Huge dataHuge data

Computational Computational

gridgrid

Access GridAccess Grid

VirtualVirtual

LabLab

Virtual Storage for Huge Virtual Storage for Huge Met. DataMet. Data

User Interface based on User Interface based on 3D, VR3D, VR

Parallel Development of modules Parallel Development of modules

Computational grid Computational grid

Re-locatable and adoptive Re-locatable and adoptive grid modelgrid model

Access grid for parallel Access grid for parallel development of model codesdevelopment of model codes


APANAPAN

연구 방법

Computational grid for Meteo. Application(Distributed Computing)Computational grid for Meteo. Application(Distributed Computing)

Using High Speed Network ( KOREN, KREONET )

Real Time Allocation of computing resource and data resource, sharing model codes

re-locatable grid with friendly UI for high resolution model모 듈 3

Model

모듈 4

Group A

Module 1

Group CModule 3

Group BModule 2

Group DModule 4

DataRes.

ComputingRes.

ComputingRes.

DataRes.

user

모듈 1 모 듈 2module2

Module 1

Module 3 Module 4


APANAPAN

연구 방법

System development for the sharing huge Met. dataSystem development for the sharing huge Met. data

Development of User Interface

virtual storage based on Grid Technology

technique development for Remote Display based on 3D,VR

Storage A

Storage B

Storage C

GridFTP

GridFTP

GridFTPVirtual Storage

Request

Manager

Data

Catalog

Manager

ReplicaCatalogManager

User

Display


APANAPAN

연구 배경

conferenceOff-Line

Media

Group A

Module1Module2Module3

…

meeting E-Mail Phone/Fax

Group B


…

Group C


…

Group D


…

One-Side

Media

Current research status Current research status

– – individual off-line individual off-line


APANAPAN

Display system 3D, VR

연구 방법

Access Grid for Collaboration

Access grid for parallel- development

Parallel computing based on high performance computing

Discussion Forum

Video Conference

Exchange E-Message

Co-operational Project

Local A

Local B

Local C

Access Grid

Computing Grid

Data Grid


APANAPAN

For the High quality weather For the High quality weather informationinformation

- More regional and global Observation Data - More regional and global Observation Data

- Data assimilation- Data assimilation

- High Performance Computing- High Performance Computing

- Ensemble Prediction for regional and global scale- Ensemble Prediction for regional and global scale

- Enhance the research activity- Enhance the research activity

- Training and Education- Training and Education


APANAPAN

Thank you!!Thank you!!


APANAPAN

Data Exchange withData Exchange with

National Information Center for Earthquakes and Disasters,National Information Center for Earthquakes and Disasters,

National Research Institute for Earth Science and Disaster Prevention National Research Institute for Earth Science and Disaster Prevention

Earthquake and ZunamiEarthquake and ZunamiEarthquake and ZunamiEarthquake and Zunami


APANAPAN

Documents

Grid resources for NWP models at national level in Korea