Upload
ellery
View
33
Download
0
Embed Size (px)
DESCRIPTION
EarthCube – Brokering A Way Forward for Global Multi-disciplinary Data Sharing. Brokering Concept Award Team Steven F. Browdy http://earthcube.ning.com/groups/brokering. Agenda. Introduction of Brokering Concept Award Short history of EarthCube brokering activities - PowerPoint PPT Presentation
Citation preview
ESIP Summer Meeting 2012 Slide 1
EarthCube – BrokeringA Way Forward for Global Multi-disciplinary Data Sharing
Brokering Concept Award TeamSteven F. Browdy
http://earthcube.ning.com/groups/brokering
ESIP Summer Meeting 2012 Slide 2
Agenda
• Introduction of Brokering Concept Award• Short history of EarthCube brokering activities• Current brokering approaches• EuroGEOSS Broker• Brokering Principles• EarthCube Particulars• Future brokering possibilities• Future EarthCube brokering activities• Q & A
ESIP Summer Meeting 2012 Slide 3
Brokering Concept Award
• EarthCube’s goal is “To transform the conduct of research in geosciences by supporting community-based cyberinfrastructure to integrate data and information for knowledge management across the Geosciences.”
• What is needed?– Integration of data, tools and communities through a
cyberinfrastructure– Utilization of current and emerging technologies to create a
transparent infrastructure for the geosciences community
• The EarthCube approach:– “Build it and they will come” doesn’t work– What gets built has to be the result of a community process,
where the community is much more than the IT enthusiasts
ESIP Summer Meeting 2012 Slide 4
Brokering Concept Award
• Our objective– Develop a strategy and roadmap for evaluating promising
brokering technologies in cooperation with a diverse set of US and international partners
• Our approach– Develop and communicate a long-term strategy and roadmap for
fully developing brokering as a core Earth Cube cyber-infrastructure component
– Demonstrate and assess several instances of brokers that mediate search and access to heterogeneous multidisciplinary data sources and services
ESIP Summer Meeting 2012 Slide 5
Brokering Concept Award
• Expectation– Engage the long tail folks, who are part of the 85% of
geoscientists who will be training the next generation– Bring them onboard, understanding at some level what we’re
doing and what is involved, helping to change the way science is done
– Interact with the other EC Groups regarding the utilization of brokering
• Technical assessment responsibility– Pilot studies with different scientific communities– Hack-a-Thons in conjunction with ESIP
ESIP Summer Meeting 2012 Slide 6
Brokering CA Program Plan
4/1/12 6
ESIP Summer Meeting 2012 Slide 7
Agenda
• Introduction of Brokering Concept Award• Short history of EarthCube brokering activities• Current brokering approaches• EuroGEOSS Broker• Brokering Principles• EarthCube Particulars• Future brokering possibilities• Future EarthCube brokering activities• Q & A
ESIP Summer Meeting 2012 Slide 8
Short History
• Roadmap development– An ongoing process
• Hack-a-Thon 1– Used EuroGEOSS Broker
• Pilots– NSIDC Arctic Polar Pilot
• June EarthCube Charrette• Cross-fertilization with other EarthCube groups• July EarthCube PI Meeting
ESIP Summer Meeting 2012 Slide 9
Short History - Roadmap
• Informed by our partners and members, as well as other EarthCube groups and the broader community of geoscientists.
ESIP Summer Meeting 2012 Slide 10
Short History – Roadmap Executive (Meta)Summary
• Developing infrastructure is a complex, dynamic process. (See the NSF report on Understanding Infrastructure: Dynamics, Tensions, and Design)– http://earthcube.ning.com/group/governance/forum/
topics/research-review-understanding-infrastructure-dynamics-tensions
• Brokers help mitigate the tensions by connecting disparate systems without imposing new burdens upon those systems
ESIP Summer Meeting 2012 Slide 11
Short History – Roadmap Executive (Meta)Summary
• Some of the benefits of a Brokering Framework include:– Lowers barriers to participation in distributed systems– Accelerates interconnection of disparate systems– Facilitates sustainability, reusability, extensibility, and
flexibility of the infrastructure– Enhances multi-disciplinary interoperability– Removes need to impose common (e.g. federal, “top-
down”) specifications enabling a more adaptive “bottom-up” evolution of the infrastructure
ESIP Summer Meeting 2012 Slide 12
Short History – Roadmap Executive (Meta)Summary
• Community engagement– It is necessary to very actively engage both the
science and informatics communities, and to employ use-case-driven requirements
– it is impossible to define all requirements in advance of development of an infrastructure
– In the short term we will be addressing small community problems through collaborative pilots
• limited initial capabilities targeted to specific communities• guided by use cases• ongoing user and community feedback
– Evolve into larger prototypes addressing larger intercommunity problems
ESIP Summer Meeting 2012 Slide 13
Short History – Roadmap Executive (Meta)Summary
• Brokering has been demonstrated to be an essential part of a robust, adaptive cyberinfrastructure
• Next Steps– Critical questions of governance and detailed
implementation will be addressed– Continued pilots and community engagement– Expansion of brokering pilots into fully operational
prototypes
ESIP Summer Meeting 2012 Slide 14
Short History – Hack-a-Thon 1
• Featured the EuroGEOSS Broker• All hack-a-thon related material is on the EarthCube Brokering ning
site– http://earthcube.ning.com/group/brokering/page/brokering-hack-a-thon-1
• A webinar was given to introduce the featured broker on May 10• A period of 10 days was used to allow participants to practice with
the broker– Validate services (required assistance)– Validate tools
• A 2-hour WebEx-enabled Hack-a-Thon occurred, after the practice period, on May 24– 15 participants from various EarthCube groups and science disciplines– Resulted in lessons learned and take-a-ways
ESIP Summer Meeting 2012 Slide 15
Short History – Hack-a-Thon 1
• Take-a-ways– Keep the featured broker available post hack-a-thon for
continued “playing”– Setup a wiki area for further discussion and collaboration– Maintain a knowledgebase for broker issues
• Lessons learned– Too narrowly focused
• Future hack-a-thons will include discussions from previous ones
– Need a broader audience– Allow a longer practice period with telecons to maintain
engagement– Poll participants for questions and comments to engage
everyone
ESIP Summer Meeting 2012 Slide 16
Short History - Pilots
• NSIDC Arctic Polar Pilot– Using the EuroGEOSS Broker– Discovering and accessing multidisciplinary
data– Analyzing the broker
• Providing detailed requirements to developers at NSIDC, CNR, and NCAR to enhance capabilities
ESIP Summer Meeting 2012 Slide 17
Short History – etc.
• Attended June EarthCube Charrette– Much interaction with other groups– Very productive in terms of working towards a shared vision, but
more work to do– Started the consensus process on priorities and timelines
• Cross-fertilization with other EarthCube groups– Gave presentations about brokering to other groups– Worked together on common use cases– Understanding areas of potential success for brokering
• July EarthCube PI Meeting– Consensus on reference architecture diagram– Consensus on development of common use cases
ESIP Summer Meeting 2012 Slide 18
Agenda
• Introduction of Brokering Concept Award• Short history of EarthCube brokering activities• Current brokering approaches• EuroGEOSS Broker• Brokering Principles• EarthCube Particulars• Future brokering possibilities• Future EarthCube brokering activities• Q & A
ESIP Summer Meeting 2012 Slide 19
Current Brokering ApproachesSingle Monolithic Instance - Standalone
ESIP Summer Meeting 2012 Slide 20
Current Brokering ApproachesSingle Monolithic Instance - Cloud
ESIP Summer Meeting 2012 Slide 21
Current Brokering ApproachesMultiple Instances – Cloud Farm
ESIP Summer Meeting 2012 Slide 22
Current Brokering ApproachesCommunity Deployment
ESIP Summer Meeting 2012 Slide 23
Agenda
• Introduction of Brokering Concept Award• Short history of EarthCube brokering activities• Current brokering approaches• EuroGEOSS Broker• Brokering Principles• EarthCube Particulars• Future brokering possibilities• Future EarthCube brokering activities• Q & A
ESIP Summer Meeting 2012 Slide 24
Introduction to the EuroGEOSS Broker
• The EuroGEOSS Broker was developed by the Laboratory of Earth and Space Science Informatics (ESSI-Lab) of the Institute of Atmospheric Pollution Research (IIA) of the National Research Council of Italy (CNR)
• It has been deployed in many situations, such as:– GEOSS Common Infrastructure
– NSIDC
– UNEP-Live
– WIS (WMO Information System)
• It’s capabilities include discovery, access, and “light” semantic mediation
• New “accessors” can be added to interoperate with data providers using standards currently unknown to the broker
ESIP Summer Meeting 2012 Slide 25
ESIP Summer Meeting 2012 Slide 26
Introduction to the EuroGEOSS Broker Supported Sources
• OGC WCS 1.0, 1.1, 1.1.2 • OGC WMS 1.3.0, 1.1.1• OGC WFS 1.0.0 • OGC WPS 1.0.0 • OGC SOS 1.0.0 • OGC CSW 2.0.2 Core, AP ISO 1.0,
ebRIM/CIM, ebRIM/EO, CWIC• WAF Web Accessible Folders 1.0 • THREDDS 1.0.1, 1.0.2• THREDDS-NCISO 1.0.1, 1.0.2• THREDDS-NCISO-PLUS 1.0.1, 1.0.2• CDI 1.04, 1.3, 1.4, 1.6• GI-cat 6.x, 7.x• GBIF• GeoNetwork (versions 2.2.0 and 2.4.1)
catalog service• HYDRO• Deegree (version 2.2) catalog service
• ESRI ArcGIS Geoportal (version 10) catalog service
• OpenSearch 1.1 accessor• OAI-PMH 2.0 (support to ISO19139 and
dublin core formats)• NetCDF-CF 1.4 • NCML-CF• NCML-OD• ISO19115-2• GeoRSS 2.0• GDACS• DIF• File system• SITAD (Sistema Informativo Territoriale
Ambientale Diffuso) accessor• INPE
ESIP Summer Meeting 2012 Slide 27
Introduction to the EuroGEOSS Broker Available Catalog Interfaces
• OGC CSW 2.0.2 AP ISO 1.0• OGC CSW 2.0.2 ebRIM EO• OGC CSW 2.0.2 ebRIM CIM• ESRI GEOPORTAL 10• OAI-PMH 2.0• OpenSearch 1.1• OpenSearch 1.1 ESIP• OpenSearch GENESI DR• GI-cat extended interface
ESIP Summer Meeting 2012 Slide 28
Broker Use Cases
View the webinar for Hack-a-Thon 1 to see these use cases demonstrated:https://esipfed.webex.com/esipfed/ldr.php?AT=pb&SP=MC&rID=59212547&rKey=72865c52d24d8992
ESIP Summer Meeting 2012 Slide 29
Use Case 1 – Traditional Query
Preview via Access Broker
Native SourceWCSGeoTiffEPSG:4326
Accessed asWMSPNG
EPSG:4326
ESIP Summer Meeting 2012 Slide 30
Use Case 2 – Semantics-enabled Query
Preview via Access BrokerNative SourceWFSGMLEPSG:4326
Accessed asWMSPNG
EPSG:4326
ESIP Summer Meeting 2012 Slide 31
Use Case 3 – Access on a Common Grid
• Definition of a Common Grid Environment for discovered datasets– Encoding Format– Spatial Resolution– Spatial Extension– Coordinate Reference System– Etc.
• Download datasets on the Common Grid• Visualize Downloaded Datasets on the
Common Grid (using the Unidata IDV tool)
ESIP Summer Meeting 2012 Slide 32
Use Case 4 – Third-party Clients
• Discovery:– GI-go (CSW/ISO Client)– Firefox Search Bar (Opensearch Client)
• Access– Mapshup (WMS Client)
Preview via Access Broker
Native SourceWCSGeoTiffEPSG:4326
Accessed asWMSPNG
(Google CRS) EPSG:3857
ESIP Summer Meeting 2012 Slide 33
Agenda
• Introduction of Brokering Concept Award• Short history of EarthCube brokering activities• Current brokering approaches• EuroGEOSS Broker• Brokering Principles• EarthCube Particulars• Future brokering possibilities• Future EarthCube brokering activities• Q & A
ESIP Summer Meeting 2012 Slide 34
Brokering Framework
ESIP Summer Meeting 2012 Slide 35
Brokering PrinciplesProblem Statement
• Whereas:– Earth system science data are extremely diverse (in
format, description, scale, precision, etc.)– Different data users discover, access, interpret, and
manipulate these data with a wide variety of tools and services that may be specific to the data and each user’s needs
– Interdisciplinary science requires people to access and use data from very different disciplines and communities
– Data providers are increasingly expected to serve communities outside their normal clientele
ESIP Summer Meeting 2012 Slide 36
Brokering PrinciplesProblem Statement
• Whereas:– Both users and data providers experience
impediments in achieving multi-disciplinary interoperability
– Connectivity for multi-disciplinary interoperability should not restrict the autonomous nature of discipline-specific systems
– Technology evolution and maintainability is a great challenge for cyber-infrastructure sustainability and usability
– Brokers should be capable of improving and facilitating access to both “big data” and “long tail” resources, in real time or static environments
ESIP Summer Meeting 2012 Slide 37
Brokering PrinciplesAssertions
• We Believe:– No one information technology or standard will serve
all user needs– Data services (discovery, access, processing,
semantics) should be openly accessible in well-defined, machine-interpretable ways
– Cyberinfrastructure should maximize data and service usability for both data providers and data users
– Flexibility is necessary for incremental expansion of data service mechanisms and to easily achieve scalability
– Brokers can be an effective way to achieve this flexibility
ESIP Summer Meeting 2012 Slide 38
Brokering PrinciplesAssertions
• We Believe:– Brokering [or a brokering framework] provides the
cyberinfrastructure that allows providers and users to better take advantage of the open services
– A brokering framework is best managed and maintained by the cyberinfrastructure (rather than Data Users or Data Providers)
– A brokering framework should be transparent to Data Users and Data Providers
– Not all existing systems will continue, but it is more sustainable to supplement, not supplant, systems mandates and governance arrangements
ESIP Summer Meeting 2012 Slide 39
Brokering Principles Working Definition of a Brokering Framework
• Definition 1:– Brokers are middleware interconnecting client and server
components in the EarthCube cyberinfrastructure– Brokers are services facilitating the run-time interconnection
(sharing of resources) among users and providers in a way that requires little effort on the part of either
– A brokering framework can consist of multiple brokering components to support different capabilities
– A brokering framework may include discovery, semantic enhancement and natural language, data access, processing and publishing
ESIP Summer Meeting 2012 Slide 40
Brokering Principles Working Definition of a Brokering Framework
• Definition 2:– A broker enables at runtime service components to transparently
make and receive requests and responses in a heterogeneous and distributed environment. It is the foundation for building multi-disciplinary applications from distributed service components and for interoperability between applications in heterogeneous and homogeneous environments.
Read our Brokering Principles at the Brokering Concept Award site, and feel free to post comments:http://earthcube.ning.com/group/brokering/forum/topics/brokering-principles-continued
ESIP Summer Meeting 2012 Slide 41
Agenda
• Introduction of Brokering Concept Award• Short history of EarthCube brokering activities• Current brokering approaches• EuroGEOSS Broker• Brokering Principles• EarthCube Particulars• Future brokering possibilities• Future EarthCube brokering activities• Q & A
ESIP Summer Meeting 2012 Slide 42
EarthCube Particulars
• Proposed reference architecture
• Connection with governance
• Common use cases
ESIP Summer Meeting 2012 Slide 43
EarthCube ParticularsReference Architecture
ESIP Summer Meeting 2012 Slide 44
EarthCube ParticularsGovernance
• How should EarthCube CI brokers be deployed and maintained?– What are the requirements for deployment, who sets them, and
what processes are used?– What are the requirements for maintenance, who sets them, and
what processes are used?– What is the funding mechanism?
• What are the policies for brokering across the boundary between communities and the EarthCube CI?
• Standards for publishing data, information, and knowledge?
• What is the process for the evolution and innovation of brokering for the EarthCUbe CI?
• Other issues?
ESIP Summer Meeting 2012 Slide 45
EarthCube ParticularsUse Cases
• Implementation Goals– Interdisciplinary demonstration for diverse providers and users– “Full” range of the EarthCube CI exercised– Collaboration between EarthCube groups– Provide a clear demonstration of benefits– Can be continued as a practical implementation
• Use cases being evaluated– Atmospheric ash– Gulf of Mexico hypoxia– Arctic polar environment
ESIP Summer Meeting 2012 Slide 46
Atmospheric AshSolid Earth Use Case
• Impacts of volcanic emission on human activities
• (Pilot 1 – volcanic supersite)• Pilot 2 – Volcanic ash from Icelandic
eruption• Disciplines – seismology, geodesy, polar,
atmosphere, biodiversity, transportation, Potential EarthCube participants –
• Other
ESIP Summer Meeting 2012 Slide 47
Gulf of Mexico HypoxiaWater Use Case
• Water and Ocean environment• Gulf of Mexico Hypoxic region• Disciplines – land cover, river water quality,
ocean chemistry, weather, biodiversity• Potential EarthCube participants –• Other?
ESIP Summer Meeting 2012 Slide 48
Arctic EnvironmentPolar Use Case
• Arctic Environment
• Data exploration service for discovery of interdisciplinary Arctic data.
• Focus areas – arctic contaminant monitoring, seasonal sea ice predictions, biodiversity
• Potential EarthCube participants –
• Other?
ESIP Summer Meeting 2012 Slide 49
Agenda
• Introduction of Brokering Concept Award• Short history of EarthCube brokering activities• Current brokering approaches• EuroGEOSS Broker• Brokering Principles• EarthCube Particulars• Future brokering possibilities• Future EarthCube brokering activities• Q & A
ESIP Summer Meeting 2012 Slide 50
Future Brokering Possibilities
• Additional functionalities– Enhanced semantics (ontologies)– Multi-natural language processing– Enhanced data transformation– Interoperation with workflows– HPC deployment for modeling
• Architecture/framework alternatives– Interoperability between same brokers– Interoperability between different brokers
ESIP Summer Meeting 2012 Slide 51
Same Broker Interoperability
• Known interoperability mechanisms
• Mutual understanding
ESIP Summer Meeting 2012 Slide 52
Different Broker Interoperability
• Unknown interoperability mechanisms (possibly new interoperability standards)
• New coordination policies needed (publish capabilities)• Possibly more governance issues
ESIP Summer Meeting 2012 Slide 53
Agenda
• Introduction of Brokering Concept Award• Short history of EarthCube brokering activities• Current brokering approaches• EuroGEOSS Broker• Brokering Principles• EarthCube Particulars• Future brokering possibilities• Future EarthCube brokering activities• Q & A
ESIP Summer Meeting 2012 Slide 54
Future EarthCube Brokering Activities
• Hack-a-Thons 2 and 3– Currently being planned– Hack-a-Thon 2 by mid-August– Hack-a-Thon 3 by mid-September– Nominate a broker for a hack-a-thon
• Email Steve Browdy at [email protected]
• More pilots
• Roadmap evolution
ESIP Summer Meeting 2012 Slide 55
Q & A