Future GRID Strategy Beijing 7th of September 2004 (CNES, J.P.Antikidis)

Beijing 6th Sept 2004, CNES/J.P.Antikidis

Future GRID StrategyBeijing 7th of September 2004

(CNES, J.P.Antikidis)

- Reminders- GRID evolution and upcoming concepts-Technological trends-Report on WGISS action 17.4- The WGISS WAG experiments- Further on activities


Exemple of distributed knowledge and process environment


IMAGINE….IMAGINE….

That you could plug your computer into the wall and have That you could plug your computer into the wall and have direct access to huge computing resources immediately, direct access to huge computing resources immediately,

just as you plug-in a lamp to get instant light.just as you plug-in a lamp to get instant light.This is the idea the Grid project is about to make into realityThis is the idea the Grid project is about to make into reality

Let’s quote Ian Foster and Carl Kesselman's book (The Grid): Let’s quote Ian Foster and Carl Kesselman's book (The Grid): ""A A computational Grid is a hardware and software computational Grid is a hardware and software infrastructure that provides dependable, consistent , pervasive infrastructure that provides dependable, consistent , pervasive and inexpensive access to high-end computational and inexpensive access to high-end computational capabilitiescapabilities".".

“The Power of Grids”

All data systems All data systems technologies are concernedtechnologies are concerned : :

* * NetworksNetworks

* * Operating systemsOperating systems

* * Large scale dataLarge scale data processing processing

* * Protected architecturesProtected architectures

* * Systems creation andSystems creation and programming programming


GRID

Classical GRID based


Virtual machine defined by Middleware primitives

« Pseudo-machine »

GRID based activitiesGRID: Distributed machines organized by an Operating System GRID: Distributed machines organized by an Operating System

making individual hardware unvisiblemaking individual hardware unvisible

GRID is not an end by itself and coversGRID is not an end by itself and covers

#GRID techno developments#GRID techno developments#GRID based system concept#GRID based system concept

Present paradox with Middleware evolution: Present paradox with Middleware evolution: Make GRID invisible Make GRID invisible from users, so from users, so Who are the users (historical trend) ?:Who are the users (historical trend) ?:

# End users directly connected to GRIDs# End users directly connected to GRIDs

# System developers directly connected to GRID# System developers directly connected to GRID

# System developers willing to make use of GRID high level # System developers willing to make use of GRID high level services (Upperware High level O/S that may compile or emulate services (Upperware High level O/S that may compile or emulate needed systems)needed systems)

Strategic usersStrategic users thinking "thinking "differentlydifferently" because of the GRID advent" because of the GRID advent


The GRID Cook-book

Direct mapping Block module or API primitives

"High level O/S: Upperware

GRID evolution Creation of "Virtual machines" (Upperwares and HEAVEN)Creation of "Virtual machines" (Upperwares and HEAVEN)

Exchange power against flexibiliy (Upcoming upper-wares)Exchange power against flexibiliy (Upcoming upper-wares)

Create "Virtual machines at will"=> Virtual FactoriesCreate "Virtual machines at will"=> Virtual Factories

Several approaches:Several approaches: Data flow virtual descriptionsData flow virtual descriptions

"Virtual machine and processing Emulation" "Virtual machine and processing Emulation"

"Virtual layer of JAVA machines"Virtual layer of JAVA machines

Creation of "transparent" system connectivityCreation of "transparent" system connectivity

Why not leave Telecom and data transfers under O/S control ?Why not leave Telecom and data transfers under O/S control ?

"Distance is not anymore relevant=> Wide Aera Grids"Distance is not anymore relevant=> Wide Aera Grids


Methodology trends

GRID+UpperWare (2002)Processing systems (1960)

SPICE simulators(1985)Electronic circuitry (1914)

INTERNET (1990)MINITEL Physical links(1970)

WINDOWS (1990)Formal programming (1948)

CATIA (1985) (virtual mockup)Blueprint plans (1850)

to « Virtual »From Real =>

"Hosting of Emulated Applications in a Virtual Environnement"

If it is possible to “instruct” a GRID to mimic (or If it is possible to “instruct” a GRID to mimic (or simulate) a computer system,simulate) a computer system,

Shouldn’t be possible to use this simulation as a Shouldn’t be possible to use this simulation as a replacement for the real system by directly running replacement for the real system by directly running the application software within the simulation itself?the application software within the simulation itself?

In short: If one can draw a computer system (that In short: If one can draw a computer system (that could be a large scale and widely distributed could be a large scale and widely distributed implementation) on a piece of paper , will it be implementation) on a piece of paper , will it be possible to create it with the help of a suitable high possible to create it with the help of a suitable high level operating system (called Upperware) level operating system (called Upperware)

In such a way it behave like a real system

If feasible , the consequences on our way to create operational and distributed collaborative system may change drastically and lead to a conceptual change comparable to Internet revolution ...

The HEAVEN paradigm

"The result produced by running an "emulator" is equivalent to the result produced by the real system."HEAVEN can "mimic" any required system

Instead of creating complex processing system, perhaps better describe them in a powerfull and fully flexible computing topology and make the resulting emulation running as a replacement for the real system.

The difficulties experienced by the KRELS are properly solved in a pure mathematic computer world !!!



Virtual enabling facilitiesMost recent approach are connected to:-Compilers and mapping systems able to "tranlate" user requirement into GRIDbasic primitives (efficient but little flexibility and conventional way to treat the inormation

-Emulation of systems embedded into a general purpose supporting GRID creating " virtual machines"

For example the Emulated type approach allows to inject conventional SW and play with system without changing any users'habits (in therory no-one can make the difference in between a Virtual based applicaion and a conventional implementation, but the development cost and effort is several order of magnitude lower


Computers (1950)+ network (1960)=> distributed processing + Packet transmission(1980) + UNIX/WINDOWS O/S(1980)+Mosaic high

level interface (1990) =Internet revolution

Computers(1950)+networks+middleware (1990)= Grid (1995+Upperware (2002)= Virtual defintion and running of processing systems (2005 ….)=

=> Full virtual data management factories (evolution or revolution ?)Advanced Upperwares like HEAVEN is to GRID the equivalent of an advanced O/S like Windows vis à vis a PC-BIOS (inside the machine everything is 8086 protocols but nobody cares anymore).To day using GRID on a direct fashion (trough middleware primitives) is equivalent to code an applicaion in machine language.

From stone-age to modern time


Technology trends and goals

Machines running and software implementaion as general purpose services fully shared on an invisible fashion among all operators

Information, processing, organization of system fully virtual (you just pay for GRID use and access to data sets)

Full analogy with Telecomm:You pay for time use of a system, not anymore for individual components uses.

Data use will not anymore associated to an image or a data set, it could be associated to its use

Applications systems created by the users themselves


"The WAG Experiments"

Response to action 17.25

-"J.P.Antikidis with input from J.Hielkema and Yonsock "J.P.Antikidis with input from J.Hielkema and Yonsock to prepare a paper on the system process for Wide Aera to prepare a paper on the system process for Wide Aera Grid (WAG)"Grid (WAG)"- Meeting with FAO (end of August)Meeting with FAO (end of August)- Coordination with ESA/ESRIN-> MMECoordination with ESA/ESRIN-> MME

(Meris-Modis Experiment)(Meris-Modis Experiment)- CNES organized effort on GRID experimentCNES organized effort on GRID experiment

=>> Proposal to hold demonstrative test based on "real =>> Proposal to hold demonstrative test based on "real size case" illustrating the capability of WAGsize case" illustrating the capability of WAG

Virtual handling of information

GRIDs main characteristics:GRIDs main characteristics:-Flexible topologyFlexible topology-Data flow handlingData flow handling-Computing PowerComputing Power-Distributed processingDistributed processing

=>>Let's add an upper layer "upperware" able to =>>Let's add an upper layer "upperware" able to submit user's jobs to the system= Virtual submit user's jobs to the system= Virtual handling of informationhandling of information

An illustration of Virtual storage handling

ConventionalConventional::

Locate the information (catalogues etc..)Locate the information (catalogues etc..)

Get hold of the data (media, access costs, transmission & delays)Get hold of the data (media, access costs, transmission & delays)

Process the informationProcess the information

Transfert the result to suitable placeTransfert the result to suitable place

WAG based conceptWAG based concept::

Create a "Light" Grid connecting a few centersCreate a "Light" Grid connecting a few centers

Make their archive accessible from this GRIDMake their archive accessible from this GRID

Enable the creation of jobs submitted to this GRIDEnable the creation of jobs submitted to this GRID

=>>Then could be demontrated the feasibility of a virtual factory =>>Then could be demontrated the feasibility of a virtual factory able to dematerialize the handling of informationable to dematerialize the handling of information


GRIDS speciesALL GRID ARE NO EQUIVALENT FROM THE ALL GRID ARE NO EQUIVALENT FROM THE

EO-SCIENCE PERSPECTIVEEO-SCIENCE PERSPECTIVE

- Macro-Grids (ganging of processors)(dynamic hanling of HW resources, eg multi-CPU machines) ~ meters (dynamic hanling of HW resources, eg multi-CPU machines) ~ meters (efficiency)(efficiency)

- Local Grids (Networked computers)with Grapes of computers (local GRIDS, eg:CERN machines) ~decameters with Grapes of computers (local GRIDS, eg:CERN machines) ~decameters (power)(power)

Wide Aera Grids(Widely distributed systems) worldwide basis (integration of services)worldwide basis (integration of services)

"WAG" concept


Virtual Systems internal mechanism

Physical layout(general purpose)

Middleware handling(Basic O/S)

Upperware(high level O/S)

CAD “like” design(Virtual factory)

“Virtual layout”

“Real interface”


Induced conceptual changes

No more knowledge of information or/and data-bank physical location (an archive is just known to be accessible trough its naming)No more need to describe the physical connections in between different locations (since fully supported by the underlying GRID) making in theory any world-wide distributed systems easy to implement.No more physical handling of information sets. This could trigger completly new merchandising concepts (for ex. come close to Telecomm recharge)No need to know the mapping of user ’software enabling virtual service to be easealy created and offered to any-one on an autonomous fashion (virtual services)


W.A.GRID based

Data providers

System designer

User agent

WAG

User Descr.

Data

User data

Real flow

"Virtual elements

GRID

A start : The “Meta Computing”

Process Process localisation is not localisation is not anymore relevantanymore relevant

Job control is not Job control is not fully machine fully machine independant and independant and special care is special care is still needed wih still needed wih respect to respect to Application Application mappingmappingll

A “Proto”-WAG experiment:GRID5000

Grid'5000 is a research effort Grid'5000 is a research effort

developping a large scale nation developping a large scale nation wide infrastructure for Grid wide infrastructure for Grid research.research.

The current plans are to assemble a The current plans are to assemble a physical platform featuring 8 physical platform featuring 8 clusters, each with 100 to a clusters, each with 100 to a thousand PCs, connected by the thousand PCs, connected by the Renater Education and Research Renater Education and Research Network. 7 sites are already Network. 7 sites are already selected. A 8th one sould be selected selected. A 8th one sould be selected soon. All clusters will be connected soon. All clusters will be connected to Renater at 1 Gb/s (10 Gb/s is to Renater at 1 Gb/s (10 Gb/s is expected in near future).expected in near future).

This high collaborative research effort is funded by the French ministry of Education and Research, INRIA, CNRS, the Universities of all sites and some regional councils. 10 laboratories are involved, nation wide, in the objective of providing the community of Grid researchers a test-bed allowing experiments in all the software layers between the network protocols up to the applications:


fMODISMERIS

Experiment(MME)

VirtualFactory

Experiment(VFE)

……

WGISS WAG Experiments

-Archives are not "physically" but logically" Archives are not "physically" but logically" describeddescribed-Processes are described as "virtual actions"Processes are described as "virtual actions"- The underlying HW layout is managed by an The underlying HW layout is managed by an underlying Middleware/Upperwareunderlying Middleware/Upperware


Near future plans: “Earth Science Grid on Demand” Within CEOSWithin CEOS: :

Share data and resources from the different agencies Share data and resources from the different agencies Provide secure and controlled access to the different Provide secure and controlled access to the different

data storage elements data storage elements Share computer resources specific modules Share computer resources specific modules Test and evaluate new processing data fusion Test and evaluate new processing data fusion

algorithmsalgorithms

Specific case: MERIS and MODIS vegetation Specific case: MERIS and MODIS vegetation productsproducts

WAG Experiment

The Virtual Factory ExperimentThe Virtual Factory Experiment Connect several resource center in a WAGConnect several resource center in a WAGMake Process server availableMake Process server availableCreate a virtual distribution schemeCreate a virtual distribution schemeAllow the projection by users of needed processAllow the projection by users of needed processPotential partners: Data and GRID providers Potential partners: Data and GRID providers (Europe+others), Remote job submitter and users (DC (Europe+others), Remote job submitter and users (DC representatives like FAO and UNEP )representatives like FAO and UNEP )

WAG Experiments & WGISS layout -Identification of a potential "task" called WAG Identification of a potential "task" called WAG experiment covering several demonstratives "test experiment covering several demonstratives "test case"case"-Task group composed of test case participants:Task group composed of test case participants:

ESA,CNES,NASA,FAO,UNEP etc…ESA,CNES,NASA,FAO,UNEP etc…

- Activity being blessed by WGISS18 ?- Activity being blessed by WGISS18 ?

- Consolidated proposals with respect to the two - Consolidated proposals with respect to the two first experiments headed by ESA (MME) and first experiments headed by ESA (MME) and CNES (VFE) to be established during CNES (VFE) to be established during WGISS19/20 time frameWGISS19/20 time frame


Documents

Future GRID Strategy Beijing 7th of September 2004 (CNES, J.P.Antikidis)