Interoperability What does it mean and how do we improve it?

  • Upload
    aminia

  • View
    52

  • Download
    4

Embed Size (px)

DESCRIPTION

Interoperability What does it mean and how do we improve it?. Jebb Q Stewart Chapel Hill, NC Thursday, July 11 th , 2013 NOAA Earth System Research Laboratory. Boulder , CO. Affiliated with Colorado State University Cooperative Institute for Research in the Atmosphere (CIRA ). Overview. - PowerPoint PPT Presentation

Citation preview

Slide 1

Interoperability

What does it mean and how do we improve it?Jebb Q StewartChapel Hill, NCThursday, July 11th, 2013

NOAA Earth System Research Laboratory. Boulder , CO.Affiliated with Colorado State University Cooperative Institute for Research in the Atmosphere (CIRA)1OverviewNOAA Earth Information System (NEIS)Climate.gov Data Interoperability ProjectInteroperabilityChallenges and Ideas to Improve Interoperability

NOAA Earth Information System (NEIS)

NEIS -The ConceptNOAA Earth Information System (NEIS) is a framework of layered services designed to help NOAAs mission areas by facilitating the discovery, access, integration, and under-standing of all NOAA (past, present, and future).

Framework provides the capability to answer questions that require data from different data sources regardless of format or location.

Applying existing concepts to NOAA data.Improving the User ExperienceHow does industry make sense of massive amounts of information?

WMSWFSWCSWCSWFSWFSWCSWCSWFS Web Feature Service WMS Web Mapping ServiceWCS Web Coverage ServiceCSW Catalog Service for the WebSOS Sensor Observation Service WPS Web Processing Service

CSWSOSStandardsOpen Geospatial Consortium (OGC)Searching and Metadata

Using Esri Geoportal (http://geoportal.sourceforge.net/) as metadata repository and to harvest metadata from other sources.Using Apache Solr (http://lucene.apache.org/solr/) to provide additional functionality and improve discoverability.Advanced Full-Text Search Capabilities: Facets.Full Document Storing (allows recreation).Supports geospatial/temporal searching.Optimized for High Volume Web Traffic Solr instance indexes information from harvested metadata using Open Geospatial Consortium (OGC) Catalogue Service for the Web (CSW) standard, and directly from data access services.Data AccessCurrently support a variety of data formats:KMLHTMLFIM native grids MoviesERDDAP (Environmental Research Divisions Data Access Program)Open Geospatial Consortium (OGC) Web Mapping Service (WMS)

Built additional data access services to simplify requests for time sequences of rendered data making them available to NEIS/TerraViz.Proxy and cache image requests to improve speed and reliability.

Visualization -- TerraViz3D visualization tool for Earth datasetsDeveloped by NOAA, with full control of source codeWritten in C#; similar to Java but runs nativelyESRL Global Systems Division

http://esrl.noaa.gov/neis

About UnityUnity is a commercial game engine that excels at rendering 3D (and 2D) scenesDevelop once, then run on Windows, Mac, Linux, iOS, Android, consoles, and browsersMillions of dollars in research and developmentGood community support and documentation, tutorials>800,000 registered developers

http://esrl.noaa.gov/neis

ESRL Global Systems Division10

http://esrl.noaa.gov/neis

ESRL Global Systems Division11Partnerships Climate.gov Data Interoperability Team

GroupsOpen Geospatial Consortium (OGC) Standards CommitteesNOAA Environmental Data Management Committee (EDMC)NOAA Data Management Integration Team (DMIT)NOAA Unified Access Framework (UAF) groupNSF Earth CubeFIM/WRF ModelingGIS Committee

ESRL Global Systems DivisionClimate.gov Data Interoperability ProjectInteroperability is the ability for users to discover the available climate data, preview and interact with the data, and acquire the data in common digital formats through a simple web-based interface.VisionPrototyping for select Use CaseBuilt a proof of concept Data Interoperable Platform

The built system is file format agnostic, meaning the pilot system will locate and display the data regardless of what format theyre archived

The web based client was developed using javascript libraries from OpenLayers and JQuery. OpenLayers library provides javascript utilities to interact with a variety of data and metadata services. JQuery provides utilities to construct the layout of the web page itself

Main goal of the pilot initiative was to develop a proof-of-concept, interoperable system for data access and displayThis platform allows users to find, browse and access the NOAAs web-based data products and services that are served through WFS (Web Feature Service), WMS (web mapping services), ESRI Export Service, and will be integrated in the Data section in NOAA Climate.gov

15How It Works

The client allows users to interact with the various services without knowledge of how the services work.The client, and underlying libraries, handle formatting specifics for each service, managing requests against the service, and parsing the response for display to the userUsing the search functionality, users can discover and access with the data aboveOnce the data has been located, it can be loaded to preview a visualization of the dataOnce data has been selected, a user has further controls to filter underlying data based on time, keywords, or geographic locationOnce user has defined constraints, the user can extract this information and download the data to their local system for further analysisThe codes are available to download and fork out in github: https://github.com/ClimateData/interoperability

A user is not limited to a single data set or type and can interact with multiple data sets or types at time

16CPC Data Served through WMS

17AcknowledgementsSudhir Raj Shrestha (NOAA-CPC)Steve Ansari (NOAA-NCDC)Kevin OBrien (NOAA-PMEL)David Herring (NOAA-CPO)Mark Phillips (UNCA)Micah Wengren (NOAA)Mike Halpert (NOAA-CPC)

InteroperabilityInteroperabilityThe ability to discover, access, view, interact, and integrate data regardless of format or physical location.

Components of an Interoperability SystemFormat Agnostic.Owner/Physical Location Agnostic.Platform Agnostic.Preview Capabilities.Semantics/Ontology/Vocabulary.Machine to Machine Communication.

Our view of interoperability is from the machine to machine perspective. In short we want the ability 20BiologicalChemicalPhysical

Biological, Chemical, and Physical data are all interrelated

Carbon Tracker

FIM

Data Integration

Biological Chem and Phy. Understanding is improved if we allow our users to discover and view these data together through one platform21InteroperabilityWhy do we want it?

Improve accessibilityFoster data exploration and use.Decrease complexityProvide framework for new tools and applications. Possibilities are endless.

Interoperability is notIt is not a web page or list of links like ftpNot easily parsed or understood by a machineNo metadata

It is not a serviceSome optional parameters help interoperabilityWMS maxWidth/maxHeight

It is not a search engineFound my data, how do I get it? What is this format?

It is not anything that requires either human intervention or the machine to guess and try and try again

Driving FactorsWhitehouse Open Data Rules to Enhance Government Efficiency and Fuel Economic GrowthOrder requires that, going forward, data generated by the government be made available in open, machine-readable formats, while appropriately safeguarding privacy, confidentiality, and security.

http://www.whitehouse.gov/sites/default/files/omb/memoranda/2013/m-13-13.pdfhttp://project-open-data.github.io/policy-memo/

NOAA Science Advisory Board (SAB) Environmental Information Services Working Group (EISWG) Executive SummaryRecommends developing an Open Weather and Climate Services (WCS) in which both NOAA and the community share equal and full access to NOAA information and development

http://www.sab.noaa.gov/Doc/Towards-Open-Weather-and-Climate-Services-report-and-transmittal_12_23_11.pdf

In addition to user demand, many different groups are advising and recommending open and accessible platforms. We are taking the initiative to explore what it means to be an interoperable system and develop one along the way.24Interoperability Readiness LevelsMeasureable indicators

Credit: NASA ESDS Technology Infusion Working GroupMatrix of measures and a scale indicating how close to goals a component is to the ideal state

Categories of ReadinessCapability EnablementStandardsDataDiscoveryUnderstandingAccess

Details not as important as the process to create a culture where such a thing can be adopted, used and encouraged throughout an organization that desires interoperability..

25ImpactsFramework built towards standards, not data. Important Because:NOAA data ready for action. Services model facilitates agile response to events. Services can be combined or reused quickly, upgraded or modified independently.Any data available through framework can be operated on or combined with other data. Integrated standardized formats and access.New and existing systems have access to wide variety of NOAA data. Any new data added, easy incorporated with minimal to no changes required.

ESRL Global Systems Division

PhysicalChemicalBiologicalBullet one especially important for disaster response26Challenges and Improving InteroperabilityFinding data often requires previous knowledge or understanding of what you are searching forHaving information readily available on data and services for searching. Our environment evolves rapidlyHarvesting versus Aggregated searchingMissing or Incomplete DataDerived or generated products/data not well documented or discoverableFinding and Discovering Data ChallengesSemantics are everywhere. We have semantics for our fields meteorology, oceanography, etc. Then individuals within those fields have different semantics. How would a fisherman, or an emergency manager refer to this data. In order for these groups to find data we have to have an understanding of how they look for data.

Our environment evolves rapidly. Understanding differences between historical, current, or future information is important. How do we understand what a user is looking for. We see this problem in two ways. For example with Polar Orbiting data, a granuale may satisfy our search, only to find out the data doesnt actually cover the area of interest. Similar with Time. A search indicates the server should have data, but upon closer examination, our exact time frame is missing

For derived products, how do we know what is possible. My simple example is Wind Speed. In Meteorology, we store the U and V vectors. Wind Speed is possible if you underlying data has these fields. How do we augment or feedback to our discovery process all the possible data given some base variables.28Metadata must contain information to allow machine to machine communication, such as:

Automated tools to generate MetadatancISOInvestigate how to document possible derived or generated data based on process.

Finding and Discovering Data RecommendationsService InformationNeed a Service EndPointUse ISO SV_ServiceIdentificationTime, periodicity (daily, monthly, yearly) Use EX_TemporalExtent with ISO8601 for Time and Duration UnitsUse ISO 19115 Proposed Revision MD SampleDimension PreviewsISO 19115 MD BrowseGraphicStyling, Ontology/Semantics Next SlidesLack of adherence to specifications and consistency behind standards. Incomplete or Work in Progress Standards.Generation of graphical representations of data. How do I stylize point information? What color palette is the best for this data?Server uptime and data availability. Metadata records say data is available for time but server may be down during access. Derived or generated data, what process to use and how to use it?

Data Access and Display ChallengesMetadata uniformity helps improve user experience need best practices, examples, and/or automated ways to produce information. OGC MetOcean DWG and Others

Dashboard/Automated Testing Systems to give feedback to data providers on outages, adherence to standards, and metadata matchup to capabilities.Rubrics, ncISO, etc

Need information on how data should be displayed, imaged, stylized. Open Geospatial Consortium (OGC) Styled Layer Descriptor (SLD) and Symbology Encoding (SE).Use ISO OnlineResource to point to style information.

Data Access and Display RecommendationsMetadata keywords inconsistent.Lack of meaningful tags.Keyword repetition same keywords are used on several data sets.Semantics, Hierarchy, Taxonomy, Ontology, Relationships All difficult to infer.Missing Information

Examples:Sea Surface Temperature SST Sea Surface TemperatureNOAA vs. National Oceanic Atmospheric Administration, Wind and WindsOcean -> Ocean Circulation -> Ocean Currents

Metadata ChallengesTags and KeywordsBe Aware of existing ontologies and keywordsGCMD Keywords: http://gcmd.nasa.gov/Resources/valids/archives/keyword_list.htmlNASA JPL SWEET Ontology http://sweet.jpl.nasa.gov/Allow user defined keywords (Crowdsourcing)

To what extent can we auto generate this informationncISO, etc

Metadata validationRubrics, Dashboards, etc

MetadataRecommendationsStay Involved and Connected:Organization Groups Within NOAA: Environmental Data Management Committee (EDM), Data Management and Integration Team (DMIT)OGC Committees (Like OGC MetOcean DWG)ESIPNSFAGUOthers? Wikis:NOAA EDM -- https://geo-ide.noaa.gov/wiki/ESIP Others?

Prototypes or Proof of concepts

Communication DiscussionOther ActivitiesImprove discoverability and ease of finding data through:Continued ontology development by linking and improving ontologies.Improved analytics. Learn what is being used and when to provide better results for future searching. Creating profiles for environmental awareness.Providing capability to weight results depending on context.

NEIS Development ActivitiesDiscovery NOAAs Big Data is different. NPP data (~ 4 TB / day and growing)GOES-RNew global forecast models, rapidly increasing in size.Vast amounts of historical data

Data are ever increasing in size. Improve accessibility to Big Data through:Providing tools allowing seamless integration of data across time and space regardless of data size.Minimize data we transfer and avoid data duplication.Move processing close to data.Allow users to collaborate on said data. NEIS Development ActivitiesBig DataContinue development, integrating and leveraging new and emerging technologies to meet NEIS goal any data, any location, any platform, now

Perform processing within cloud environment and with high speed connectivity to data sources, taking advantage of large processing power within cloud infrastructure.

Send graphics and server side processed/rendered/streamed data to GUI, improving bandwidth utilization.

Take advantage of fast networking to make remote requests and processing appear like local application.

Similar to how the concept of the Amazon Silk Browser.

And Beyond

ESRL Global Systems [email protected]://www.esrl.noaa.gov/neis