GLOBALBIODIVERSITY

INFORMATIONFACILITY

David Remsen

ECAT Program Officer

August 2010

WWW.GBIF.ORG

Informatics Infrastructure and Portal (IIP)

Contents

• Publishing– Current developments in DarwinCore, its

extension, and publishing solutions (incl. the Integrated Publishing Toolkit)

• Integration and discovery– Status of tools for the harvesting,

interpretation through controlled vocabularies and plans for the Data Portal

• Communications– Evaluation of the communication platforms,

update on staffing changes and resources

Publishing

• Objectives– Simplify the publication of primary biodiv. data– Support the publication of species-level data– Improve data quality & dataset documentation– Reduce the latency between publishing and

discovery through portals– Support the capacity to extend the published

content– Expand data publishing configuration options

Publishing What

• Species Occurrence Data– Primary Biodiversity Data– Observations / Nat. Hist. Collections

• Species-level Data– Taxonomic Catalogues– Annotated Species Checklists• Floral and Faunal lists• Thematically-defined lists (Red-List, Invasive,

etc.)

• Dataset (Resource) Metadata

Standards and Protocols

• Primary Biodiversity data– Darwin Core via DiGIR protocol– ABCD (Access to Biological Collections

Data) via BioCase protocol– TAPIR protocol– multiple output formats

• Taxonomic data– Taxon Concept Schema (TCS)• Few tools• Low uptake

Protocols impact harvesting latency – Schemas are complex and constrain data scope

Darwin Core

• Ratified in 2009• Significant additions/refinements• Set of terms– http://rs.tdwg.org/dwc/terms/index.htm

• Expressed via XML• Simple Darwin Core (Subset)• Express as Text– http://

rs.tdwg.org/dwc/terms/guides/text/index.htm

Darwin Core Archives (DwC-A)

Extensions are text files

DwC-A Case Study: Ireland

• National Biodiversity Data Centre (Ireland)• Ireland joined GBIF in 2009• Selected DwC-A as the easiest integration• Incorporated into internal systems – Under 2 weeks of development

• Automatic registration through RegistryAPI– http://code.google.com/p/gbif-registry/wiki/ResourceAPI

• 34 Collections today• 450,000 records harvested

Publish via: Direct Export of DwC-A• Requires basic DBA skills and

documentation– Darwin Core Terms– Darwin Core Archive Format– Publishing Taxonomic Catalogues &

Annotated Checklists via DwC-A– Publishing Occurrence Data via DwC-A

• Access to list of terms, supported extensions, and schemas– http://rs.gbif.org (Schema repository)

Status: Documentation release September 2010 via GBIF website

XML Descriptor file

• <archive xmlns="http://rs.tdwg.org/dwc/text/" •          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" •          xsi:schemaLocation=http://rs.tdwg.org/dwc/text/   http://rs.tdwg.org/dwc/text/tdwg_dwc_text.xsd• metadata=”http://www.biodiv.org/docs/metadata/whale_catalogue.eml”>•  •   <core encoding="UTF-8" fieldsTerminatedBy="\t" linesTerminatedBy="\n" fieldsEnclosedBy='' ignoreHeaderLines="0"

rowType="http://rs.tdwg.org/dwc/terms/Taxon">•     <files>•       <location>taxa.txt</location>•     </files>•     <id index="0" />•     <field index="1" term="http://rs.tdwg.org/dwc/terms/kingdom"/>•     <field index="2" term="http://rs.tdwg.org/dwc/terms/phylum"/>•     <field index="3" term="http://rs.tdwg.org/dwc/terms/class"/>•     <field index="4" term="http://rs.tdwg.org/dwc/terms/order"/>•     <field index="5" term="http://rs.tdwg.org/dwc/terms/family"/>•     <field index="6" term="http://rs.tdwg.org/dwc/terms/genus"/>•     <field index="7" term="http://rs.tdwg.org/dwc/terms/species"/>• <field index="8" term="http://rs.tdwg.org/dwc/terms/infraspecies"/>• <field index="9" term="http://rs.tdwg.org/dwc/terms/infraspeciesRank"/>• <field index="10" term="http://rs.tdwg.org/dwc/terms/scientificNameAuthorship"/>•     <field default="ICZN" term="http://rs.tdwg.org/dwc/terms/nomenclaturalCode"/>•   </core>•  •   <extension encoding="UTF-8" fieldsTerminatedBy="\t" linesTerminatedBy="\n" fieldsEnclosedBy='' ignoreHeaderLines="0"

rowType="http://rs.gbif.org/terms/1.0/VernacularName">•     <files>•       <location>vernacular.txt</location>•     </files>•     <coreid index="0" />•     <field index="1" term="http://rs.tdwg.org/dwc/terms/vernacularName"/>•     <field index="2" term="http://purl.org/dc/terms/language"/>•     <field index="3" term="http://rs.tdwg.org/dwc/terms/countryCode"/>•   </extension>• </archive>

Authoring meta.xml

Status: Beta release Sept. 1 http://code.google.com/p/gbif-meta-maker/

Excel Spreadsheet Templates

http://code.google.com/p/gbif-spreadsheet-processor/

Status: Beta release September (by TDWG)

Excel Spreadsheet Templates

http://code.google.com/p/gbif-spreadsheet-processor/

Integrated Publishing Toolkit

• A supported platform for publication of:– Occurrence-level content– Species checklist content– Dataset metadata • Sampling methods• Bibliographic citations• Temporal coverage

• DwC-A compatible– Reduced latency between publishing and

discovery

Integrated Publishing Toolkit

• GBIF Review 2009• “…with regards to software and tool

development…:– Lack of rigorous technical documentation;

open source software must be documented and annotated meticulously in order to take advantage of improvements made by users.

– Release of unstable code that is being worked on still by its initiators to a community who are not made aware that it is not finalised.”

Integrated Publishing Toolkit

• Received good feedback in first year of use

• Primary request: Simplify and “lighten” up the product

• RC4 testing initiating now– Enhanced metadata (still EML)– Darwin Core Archive import–Multiple organisation association (for

hosting centres)– Bug fixing

Integrated Publishing Toolkit

• RC4 will not address feedback, but will be a more stable version for new users

• RC5 development underway to address feedback– Simplification all around (intuitiveness)– Performance improvements– Enriched documentation/examples/webcasts– Server requirements dropping significantly

(target of 256MB of memory)

Integrated Publishing Toolkit

• Following RC5, we will initiate user testing and bug fixing, with no unnecessary functionality changes to move to the target of a stable robust platform by end 2010

• http://code.google.com/p/gbif-providertoolkit/

Vocabulary server

• Drupal implementation developed as a proof of concept– http://vocabularies.gbif.org/

• IPT uses extensions, vocabularies and schemas for the operation– http://vocabularies.gbif.org/

• No well defined workflows yet for community ownership of vocabularies– Discuss at TDWG ’10 – ViBRANT Funding to operationalise

Vocabulary server

Draft New Extensions

Draft New Vocabularies

Publish them

Internationalise them

Indexing and Discovery

• Objectives– Extend the classes of content that can be

discovered– Improve the means to discover (flexible

indexes)– Better determination of fitness for use

• Through dataset metadata

– Annotation / Feedback brokerage– Accurate citation– Reduce the latency between publishing and

discovery through portals

GBIF Registry (GBRDS)

• Index of the technical access points of the datasets comprising the GBIF network

• Captures basic metadata about institutions, datasets, nodes and their relationships

• Enhanced features under development– Improved attribution – Better data provenance declaration– More accurate reporting on the total

participation within the GBIF network– Dynamic definitions of thematic networks– API / Web app for automating registration

http://gbrds.gbif.org/index

GBIF Registry (GBRDS)

http://gbrds.gbif.org/index

Registry: GBIF is complex…

Metadata catalogue

• Collection of XML-based dataset metadata documents (ISO, FGDC, EML, DIV formats)

• Associated with entities known to the GBIF Registry• Common search across content• Currently using Metacat

– Will review this following prototyping

Goal: Enriched documentation, discovery of unpublished datasets

Status: Under development, promoting publication of data documents through “small grant awards”

Harvesting and Indexing Toolkit

• The GBIF harvesting software:– Foundations to harvest DiGIR, BioCASe,

TAPIR, DwC-A– Synchronisation with the GBIF Registry– User interface for controlling and

scheduling harvesting operations–Metrics for the success of harvest runs– Access to logs for diagnostics– Sychronisation against the GBIF Portal

database

Harvesting and Indexing Toolkit

Harvesting and Indexing Toolkit

• In production use in GBIFS only• Some external users are testing – collecting feedback

now• In light of the GBIF review comments, need to assess:

– The need for such a tool – requirements are sought by community

– Resources needed to meet expectation by community (versioning, bug fixing, support, manuals)

• Is it rather a library to aid developers than a product per-say?

• Remember a homogenous network does not require multi protocol support and can be handled far more simply!

Data Portal

• http://data.gbif.org– Little functional development in recent

months• Bug fixing activities only

– Continues to grow in content• Jan 2010: 196 million• Aug 2010: 203 million

– 2500 – 3000 visitors per day (plus web service use)

– US visitors account approx 22% (2010 traffic)• 2nd is UK visitors at 5%

Data Portal Evolutions

• Portal will evolve by end 2011– Improved taxonomic services and content

• Achieved through the Global Names Architecture

– Improved attribution and provenance• Achieve by enhancing the Registry

– Improved occurrence indexing• Scalable solution, richer fields, reduced latency… etc

– Improved map visualisations– Custom information feeds

• Abstracts, repatriation, records modified

– Improved dataset metadata• Determining fitness for use

Portal evolution

Currently Roadmap

Limited to 250,000 records for download

Access to unlimited volume of export formats

23 Darwin core properties available for search

Ability to support multiple indexes (Common, marine, terrestrial Plantae etc)

30 fields available on record detail Full record detail visible

Limited ability to determine fitness for use

Improved access to metadata where available. Improvements in automated determination of fitness for use (spatial resolution)

Poor understanding of the basis of record

Improvements in determining point versus grid based content, ex situ versus in situ records etc.

Limited spatial search Provide means to access content through user defined polygons

Occurrence content

Portal evolution

Currently Roadmap

Synthesised taxonomy assembled from all content

Multiple taxonomic organisation

Assembly methods of synthesised taxonomy poorly documented

Rigorous documentation for taxonomic organisation

Common name search limited Many names sources used to enable common name search

Limited comparison between taxonomies

Services to enable taxonomic comparison (overlap and contradiction)

Limited services for external integration Improved APIs for connecting external systems

Few checklist sources included (4-5?) in current data portal

100s of checklists accessible NOW in Dev Version – integrated into new Data Portal

Taxonomic content and organisation

Portal evolution

Currently Roadmap

Metadata limited to contact information Ability to use rich dataset metadata where available

Only datasets with digitised records discoverable

Datasets described through metadata discoverable

Attribution of data limited to provider and dataset

Better attribution of all parties involved in data publication

Citation in data exported limited to datasets only

Prototype citation services

Feedback delivered on a per record basis by email

Annotation brokerage services

Metadata, attribution and feedback

Checklist Bank Slide

Status: Dev

In-use by ALA

Data Portal Evolution

The Portal is more than just a discovery system. The Portal will be a hub that allows:

a) Data custodians to – Register the existence of biodiversity data sources– Publish their content and in addition, rich

information about the content (e.g. metadata documenting assembly methods)

– Subscribe to annotations made against their content– Subscribe to information about the usage of their

content– Access services of interest to them (e.g. quality

control)

Data Portal Evolution

b) Users to– Search content in real time, through various

customised search options (e.g. Terrestrial plants, marine mammals, natural history collections, protected areas)

– Browse content taxonomically, temporally, geographically etc.

– Define and run reports (not real time) to extract a data subset or derive metrics

– Subscribe to customised information feeds (e.g. Modified Pinaceae specimens in Australia)

– Publish annotations related to record quality, or assertions about that record (e.g. confirmed suitable for 100km modeling)

– Build better information systems that utilise services offered by the Portal

Nodes Portal Toolkit

• Customise-able toolkit to deploy of a National/Regional/Thematic discovery portal

• Technical advisory group for NPT recommend that:– To fully engage the NODES community in the design,

development, testing and deployment of the NPT. – To ensure tight integration of the NPT with the GBIF

Informatics Infrastructure, while taking benefit from a wide array of additional biodiversity-related web services.

– To adopt an open source content management platform such as Drupal, upon which to build and develop specific NPT modules (specifically those for integration, visualisation, and access of biodiversity-related data and information)

• A call for an NPT coordinator is currently in draft• ViBRANT funds will support

Communications

http://community.gbif.org/

Participant forums

13 August Launch

Communications

http://www.gbif.org/

Consolidate tech docs

RSS Feeds

More updates

Secretariat Tech Capacity

• Resources– 3 current openings Java Developers– Vocabularies/Ontology Developer (30

months) –ViBRANT– Taxonomic Publishing Developer (18

months) – i4Life / Catalogue of Life

Summary: Informatics Targets

Data custodians

Registry

Harvesters

Processing

Indexes

APIs(user / machine)

Clients

Dat

a flo

w

Point based occurrences

Grid based occurrences

Checklists

Dataset Metadata

Refined end to end workflows for: