Upload
issgc-summer-school
View
1.009
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
Introduction to Workflows, APIs and Semantics
Session 37. July 13th, 2009
Oscar Corcho (Universidad Politécnica de Madrid)
Based on slides from all the presenters in the following two days
Work distributed under the license Creative Commons Attribution-Noncommercial-Share Alike 3.0
Themes of the Second Week
Date Theme TechnologyMon 13 July How to solve my problem?
Tue 14 July Higher level APIs: OGSA-DAI, SAGA and metadata management
SAGA,OGSA-DAI,Grid SAM
Wed 15 July Workflows P-GRADE,Semantic Metadata
Thu 16 July Integrating Practical All
Fri 17 July Cloud Computing (lecture)
Principles of job submission and
execution management
Principles of high-throughput computing
Principles of service-oriented
architecture
Principles of distributed data management
Principles of using distributed and
high performance systems
Higher level APIs: OGSA-DAI, SAGA
and metadata management
Workflows
Motivation• Grids are:
– Dynamic:• Version, updates, new resources...
– Heterogenous:• Operating Systems, Libraries, software stack• Middleware service versions and semantics• Administrative policies – access, usage, upgrade
– Complex:• Production level service with high QoS non-trivial• Derived from above as well as inherently
• As described by Steven this morning, operating Grids is still an effort-consuming task, and it is still somehow difficult to develop, program & deploy Grid applications using the existing Grid middleware
• But as you have also seen during last week (and in Morris’ presentation today), there are many commonalities among heterogeneous middleware
In general…• As described by Steven this morning, operating Grids is still an
effort-consuming task, and it is still somehow difficult to develop, program & deploy Grid applications using the existing Grid middleware
• But as you have also seen during last week (and in Morris’ presentation today), there are many commonalities among heterogeneous middleware
• There is a need of:– Programmatic approaches that provide common grid functionality at a
correct level of abstraction for applications– Ability to hide underlying complexity of infrastructure, varying semantics,
heterogeneity and changes from the application-developer– Improved data access and integration mechanisms– Traceable, repeatable analyses of e-Science experiments– Graphical modelling languages for the ease of Grid application
development
e-Science Approach Interoperability• Increasing complexity of e-science applications that embrace
multiple physical models (i.e. multi-physics) & larger scale– Creating a steadily growing demand of compute power– Demand for a ‘United Federation of world-wide Grids’
Balatonfüred, Hungary, 6th-18th July 2008
I. Simple Scripts & Control
II. ScientificApplication plug-ins
III. ComplexWorkflows
IV. InteractiveAccess
V. Interoperability
Grid other Grid type
Grid Middleware
[2] Morris Riedel et al., ‘Classification of Different Approaches for e-Science Applications in Next Generation Infrastructures, Int. Conference on e-Science 2008, Indianapolis, Indiana
SAGA one-slide summary• Simple API for Grid Application – SAGA
– Provide simple and usable programmatic interface that can be widely-adopted for the development of applications for the grid
– Simplicity (80:20 restricted scope)• easy to use, install, administer and maintain
– Uniformity• provides support for different application programming languages as well as
consistent semantics and style for different Grid functionality– Scalability
• Contains mechanisms for the same application (source) code to run on a variety of systems ranging from laptops to HPC resources
– Genericity• adds support for different grid middleware
– Modularity • provides a framework that is easily extendable
• SAGA is not…– Middleware– Service management interface– Does not hide the resources - remote files, job (but the details)
Text
Example: SAGA Job submission
Example: Copy a File (Globus)if (source_url.scheme_type == GLOBUS_URL_SCHEME_GSIFTP || source_url.scheme_type == GLOBUS_URL_SCHEME_FTP ) { globus_ftp_client_operationattr_init (&source_ftp_attr); globus_gass_copy_attr_set_ftp (&source_gass_copy_attr, &source_ftp_attr); } else { globus_gass_transfer_requestattr_init (&source_gass_attr, source_url.scheme); globus_gass_copy_attr_set_gass(&source_gass_copy_attr, &source_gass_attr); } output_file = globus_libc_open ((char*) target, O_WRONLY | O_TRUNC | O_CREAT, S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP); if ( output_file == -1 ) { printf ("could not open the file \"%s\"\n", target); return (-1); } /* convert stdout to be a globus_io_handle */if ( globus_io_file_posix_convert (output_file, 0, &dest_io_handle) != GLOBUS_SUCCESS) { printf ("Error converting the file handle\n"); return (-1); } result = globus_gass_copy_register_url_to_handle ( &gass_copy_handle, (char*)source_URL, &source_gass_copy_attr, &dest_io_handle, my_callback, NULL); if ( result != GLOBUS_SUCCESS ) { printf ("error: %s\n", globus_object_printable_to_string (globus_error_get (result))); return (-1); } globus_url_destroy (&source_url); return (0); }
int copy_file (char const* source, char const* target) {globus_url_t source_url;globus_io_handle_t dest_io_handle;globus_ftp_client_operationattr_t source_ftp_attr;globus_result_t result;globus_gass_transfer_requestattr_t source_gass_attr;globus_gass_copy_attr_t source_gass_copy_attr;globus_gass_copy_handle_t gass_copy_handle;globus_gass_copy_handleattr_t gass_copy_handleattr; globus_ftp_client_handleattr_t ftp_handleattr; globus_io_attr_t io_attr; int output_file = -1; if ( globus_url_parse (source_URL, &source_url) != GLOBUS_SUCCESS ) { printf ("can not parse source_URL \"%s\"\n", source_URL); return (-1); } if ( source_url.scheme_type != GLOBUS_URL_SCHEME_GSIFTP && source_url.scheme_type != GLOBUS_URL_SCHEME_FTP && source_url.scheme_type != GLOBUS_URL_SCHEME_HTTP && source_url.scheme_type != GLOBUS_URL_SCHEME_HTTPS ) { printf ("can not copy from %s - wrong prot\n", source_URL); return (-1); } globus_gass_copy_handleattr_init (&gass_copy_handleattr); globus_gass_copy_attr_init (&source_gass_copy_attr); globus_ftp_client_handleattr_init (&ftp_handleattr); globus_io_fileattr_init (&io_attr); globus_gass_copy_attr_set_io (&source_gass_copy_attr, &io_attr); &io_attr); globus_gass_copy_handleattr_set_ftp_attr (&gass_copy_handleattr, &ftp_handleattr); globus_gass_copy_handle_init (&gass_copy_handle, &gass_copy_handleattr);
Text
Example: Copy a File (SAGA)
#include <string>#include <saga/saga.hpp>
void copy_file(std::string source_url, std::string target_url) { try { saga::file f(source_url); f.copy(target_url); } catch (saga::exception const &e) { std::cerr << e.what() << std::endl; }}
The interface is simple and the actual function calls remain the same
Workflow one-slide summary• Build distributed applications through orchestration of multiple
services– Allows to compose larger applications from individual application
components– The components can be independent or connected by some control
flow/ data flow dependencies.– Scaled up execution over several computational resources
• Integration of multiple teams involved (collaborative work)• Unit of reusage: e-science requires traceable, repetable analysis
– Provide automation: Reproducibility of scientific analyses and processes is at the core of the scientific method
– Support easy analysis modifications– Sharing workflows is an essential element of education, and
acceleration of knowledge dissemination.”– Allows capture and generation of provenance information
• Ease the use of grids: graphical representation– Capture individual data transformation and analysis steps
NSF Workshop on the Challenges of Scientific Workflows, 2006, www.isi.edu/nsf-workflows06Y. Gil, E. Deelman et al, Examining the Challenges of Scientific Workflows. IEEE Computer, 12/2007
Workflow• The automation of a business process, in whole or part, during
which documents, information or tasks are passed from one participant to another for action, according to a set of procedural rules to achieve, or contribute to, an overall business goal.
• Workflow management system (WFMS) is the software that does it
www.wfmc.org
Workflow Reference Model, 19/11/1998
What does a typical Grid WFMS provide?• A level of abstraction above grid processes
– gridftp, lcg-cr, lfc-mkdir, ...– condor-submit, globus-job-run, glite-wms-job-submit, ...– lcg-infosites, ...
• A level of abstraction above “legacy processes”– SQL read/write– HTTP file transfer, …
• Mapping and execution of tasks grid resources– Submission of jobs– Invocation of (Web) services– Manage data – Catalog intermediate and final data products
• Improve successful application execution• Improve application performance• Provide provenance tracking capabilities
http://www.gridworkflow.org/
What does a typical Grid WFMS provide?
Source: Jia Yu and Rajkumar Buyya: A Taxonomy of Workflow Management Systems for Grid Computing, Journal of Grid Computing, Volume 3, Numbers 3-4 / September, 2005
Abstract Workflow Executable Workflow
Describes your workflow at a logical level
Describes your workflow in terms of physical files and paths
Site Independent Site Specific
Captures just the computation that the user wants to do
Has additional jobs for data movement etc.
What does a typical workflow consist of?
• Dataflow graph• Activities
– Definition of Jobs– Specification of services
• Data channels– Data transfer– Coordination
• Cyclic (DAG) /acyclic• Conditional statements
Workflow Lifecycle
Workflow Template
Workflow Instance
Executable Workflow
Data, Metadata, Provenance Information
Data, Metadata Catalogs
Resource, Application Component DescriptionsCompute,
Storage and
Network Resources
Data Products
ExecuteMap to
available resources
Adapt, Modify
Workflow and
Component Libraries
Populate with data
Creation
Mapping
Scheduling/Execution
Reuse
Distributed
Data lifecycle in workflows
Data Discovery
Der
ived
Dat
a an
d
Prov
enan
ce A
rchi
val
Data Processing
Data A
nalysis SetupData Lifecycle
in a Workflow Environment
Metadata Catalogs
Provenance Catalogs
Component Libraries
Workflow Template Libraries
Data Replica CatalogsData Movement Services
Software Catalogs
Workflow Creation
Workflow Mapping andExecution
Workflow Reuse
Workflow Execution
P-GRADE one-slide summary• P-GRADE portal desiderata
– Hide the complexity of the underlying grid middlewares– Provide a high-level graphical user interface that is easy-to-use for e-
scientists– Support many different grid programming approaches:
• Simple Scripts & Control (sequential and MPI job execution)• Scientific Application Plug-ins (based on GEMLCA)• Complex Workflows• Parameter sweep applications: both on job and workflow level• Interoperability: transparent access to grids based on different
middleware technology– Support three levels of parallelism
• History– Started in the Hungarian SuperComputing Grid project in 2003– http://portal.p-grade.hu/– https://sourceforge.net/projects/pgportal/
Workflow sharing: MyExperiment
http://www.myexperiment.org/
Data access and integrationResearcher wants to obtainspecified data from multipledistributed data sources andto supply the result to aprocess and then view itsoutput.
1 Researcher formulates query2 Researcher submits query3 Query system transforms and distributes query4 Data services send back local results5 Query system combines these to form requested data
6 Query system sends data to process7 Process system sends derived data to researcher
OGSA-DAI one-slide summary• Enable the sharing of data resources to support:
– Data access - access to structured data in distributed heterogeneous data resources.
– Data transformation e.g. expose data in schema X to users as data in schema Y.
– Data integration e.g. expose multiple databases to users as a single virtual database
– Data delivery - delivering data to where it's needed by the most appropriate means e.g. web service, e-mail, HTTP, FTP, GridFTP
• History– Started in February 2002 as part of the UK e-Science Grid Core
Program– Part of OMII-UK, a partnership between:
• OMII, The University of Southampton• myGrid, The University of Manchester• OGSA-DAI, The University of Edinburgh
OGSA-DAI Generic web servicesRelational Database
XML Database
Indexed File
• Manipulate data using OGSA-DAI’s generic web services
• Clients see the data in its ‘raw’ format, e.g.– Tables, columns, rows for
relational data– Collections, elements etc. for
XML data• Clients can obtain the schema
of the data• Clients send queries in
appropriate query language, e.g. SQL, XPath
Relational Database
XML Database
Indexed File
OGSA-DAIrequest
data
OGSA-DAI Workflows
• Pipeline, Sequence, Parallel workflows• Composed of activities• Reduces data transfers and web service calls
25
Metadata Management: A Satellite Scenario
SpaceSegment
Ground Segment DMOP files
Product files
SATELLITE FILES:
A Sample File in the Satellite Domain
DATA
METADATA
Metadata can be present in file names… Namefile (Product):RA2_MW__1PNPDK20060201_120535_000000062044_00424_20518_0349.N1" Corresponds to:
27
…and in file headersFILE ; DMOP (generated by FOS Mission Planning System) RECORD fhr FILENAME="DMOP_SOF__VFOS20060124_103709_00000000_00001215_20060131_014048_20060202_035846.N1" DESTINATION="PDCC" PHASE_START=2 CYCLE_START=44 REL_START_ORBIT=404 ABS_START_ORBIT=20498
ENDRECORD fhr................................ RECORD dmop_er RECORD dmop_er_gen_part RECORD gen_event_params
EVENT_TYPE=RA2_MEA EVENT_ID="RA2_MEA_00000000002063" NB_EVENT_PR1=1 NB_EVENT_PR3=0 ORBIT_NUMBER=20521 ELAPSED_TIME=623635 DURATION=41627862 ENDRECORD gen_event_params ENDRECORD dmop_erENDLIST all_dmop_erENDFILE
RECORD ID
RECORD parameters
RECORD parameters corresponding to other RECORD
structure.
Metadata can be exposed• Metadata deserves a better treatment
– In most cases it appears together with files or other resources– It is difficult to deal with– What about trying to query about all the files that deal with instrument X
and where the information was taken from time T1 to T2?
Our goal:Let’s make metadata a FIRST-CLASS CITIZEN in our systemsAnd let’s make it FLEXIBLE to changes
Workflow Lifecycle
Workflow Template
Workflow Instance
Executable Workflow
Data, Metadata, Provenance Information
Data, Metadata Catalogs
Resource, Application Component DescriptionsCompute,
Storage and
Network Resources
Data Products
ExecuteMap to
available resources
Adapt, Modify
Workflow and
Component Libraries
Populate with data
Reuse
Metadata and workflows
• Metadata for describing workflow entities– What is the value added of a given workflow?– What is the task a given service performs?– What are the services that can be associated with a
processor? • Metadata for describing workflow provenance
– How did the execution of a given workflow go?– What this the semantics of a data product?– How many invocations of a given service failed?
Some metadata about a workflow
RDF annotations
Social Tags annotations
Free-text annotations
Reference Ontology1
Reference Ontology2
ReferenceControlled vocabulary
A scientific workflow
Metadata content
Metadata is everywhere• We can attach metadata almost to anything
– Events, notifications, logs– Services and resources– Schemas and catalogue entries – People, meetings, discussions, conference talks– Scientific publications, recommendations, quality comments– Models, codes, builds, workflows, – Data files and data streams– Sensors and sensor data
• But..., what do we mean by metadata???
What is the metadata of this HTML fragment?Based on Dublin CoreThe contributor and creator is the flight booking service “www.flightbookings.com”.The date would be January 1st, 2003, in case that the HTML page has been generated on that specific date.The description would be something like “flight details for a travel between Madrid and Seattle via Chicago on February 8th, 2004”.The document format is “HTML”.The document language is “en”, which stands for English
Based on thesauriMadrid is a reference to the term with ID 7010413 in the thesaurus, which refers to the city of Madrid in Spain.Spain is a reference to the term with ID 1000095, which refers to the kingdom of Spain in Europe.Chicago is a reference to the term with ID 7013596, which refers to the city of Chicago in Illinois, US.United States of America is a reference to the term “United States” with ID 7012149, which refers to the US nation.Seattle is a reference to the term with ID 7014494, which refers to the city of Seattle in Washington, US.
Based on ontologiesConcept instances relate a part of the document to one or several concepts in an ontology. For example, “Flight details” may represent an instance of the concept Flight, and can be named as AA7615_Feb08_2003, although concept instances do not necessarily have a name.Attribute values relate a concept instance with part of the document, which is the value of one of its attributes. For example, “American Airlines” can be the value of the attribute companyName.Relation instances that relate two concept instances by some domain-specific relation. For example, the flight AA7615_Feb08_2003 and the location Madrid can be connected by the relation departurePlace
Need to Add “Semantics”• External agreement on meaning of annotations
– E.g., Dublin Core for annotation of library/bibliographic information
• Use Ontologies to specify meaning of annotations– Ontologies provide a vocabulary of terms, plus– a set of explicit assumptions regarding the intended meaning of the
vocabulary. • Almost always including concepts and their classification• Almost always including properties between concepts• Similar to an object oriented model
– Meaning (semantics) of terms is formally specified– Can also specify relationships between terms in multiple ontologies
• Thus, an ontology describes a formal specification of a certain domain:– Shared understanding of a domain of interest– Formal and machine manipulable model of a domain of interest
S-OGSA Model
Summary• From the lower level of abstraction…
– Difficulties to develop, program & deploy Grid applications using the existing Grid middleware
• To a higher level of abstraction:– High-level APIs and metadata management
• Programmatic approaches that provide common grid functionality at a correct level of abstraction for applications
• Ability to hide underlying complexity of infrastructure, varying semantics, heterogeneity and changes from the application-developer
– Improved data access and integration mechanisms– Workflow management
• Traceable, repetable analyses of e-Science experiments• Graphical modelling languages for the ease of Grid application
development
Introduction to Workflows, APIs and Semantics
Session 37. July 13th, 2009
Oscar Corcho (Universidad Politécnica de Madrid)
Based on slides from all the presenters in the following two days
Work distributed under the license Creative Commons Attribution-Noncommercial-Share Alike 3.0