Beispielbild The BioCASe Technology Jörg Holetschek Botanic Garden & Botanical Museum Berlin-Dahlem Dept. of Biodiversity Informatics and Laboratories

BeispielbildThe BioCASe

Technology

Jörg HoletschekBotanic Garden & Botanical Museum Berlin-DahlemDept. of Biodiversity Informatics and LaboratoriesKönigin-Luise-Straße 6-814195 Berlin

BioCASe Workshop, Melbourne, Feb 4-5th 2010

2BioCASe Workshop, Melbourne, Feb 4-5th 2010

Agenda

1. The BioCASe Architecture: An Overview

2. The BioCASe Provider SoftwareFeature, Requirements, Installation, Configuration

3. The ABCD and HISPID data standardsIntentions, Structure, Elements, Use

4. Preparing the database for BioCASe/ABCD

5. Setting Up DatasourcesDatabase connection, Table Setup, Mapping Process;Testing, Data Backups

6. Other Issues

Workshop Wiki: http://hiscom.chah.org.au/wiki/BioCASe_Workshop

Beispielbild

1. BioCASe Technology:

Motivation, Idea and Architecture


Herbaria, Drawings

© J. Holstein et al.


Preserved Specimens



Living Collections



Culture collections



Primary Biodiversity Data

Biological Collection Access Service

Documentation of the occurrence of one species

at a given location at a certain point in time

= Primary Biodiversity Data Record


Data sources worldwide

- Index Herbariorum: 3,293 herbaria, 400 million herbarium sheets- 50-100,000 natural history collections, 1.5-2 billion specimens- With observations added, occurrence records 3+ billion (10b?)

Over 75% of biodiversity information are stored in developed countries.

Est. 75% of all species are found in the developing world.

Source: BARTHLOTT et al. 1999


Accessibility

Stage 0: Only in real world (paper catalogues, just stacks)Only meta information available on the web

Stage 1: Stage 2: Online catalogue Digitalization of specimen


Biodiversity Data

Stage 3: Networking the databases


Architectural Overview

2. Wrapper Software

1. Protocols/Data Standards

Data Quality CheckerDataMining

3. Applications

Data Portal


BioCASe Design Principles

No central database Data remain in the existing DB systems Data Provider gets full credit Full control over published data by collection holder

Partial publication possible

Collection holder can withhold information from publication (e.g., locality data for endangered species) or exclude records (e.g. until research results are published)

Wrapper principle Data remain in original collection management system No changes in workflow for curator/local users


2: The BioCASe ProviderSoftware

Wrapper: BioCASeProvider Software

Protocols/Data Standards


Applications

Data Portal


Software package that „wraps“ around the collection database Equips it with a BioCASe protocol compliant interface

1. Accepts requests from the network

3. Transforms results into ABCD documents and sends them back

BioCASe Provider Software (Wrapper)

Marmotamarmota?

2. Tanslates queries to the collection database

SELECT *FROM specimenWHERE ScientificName LIKE “Marmota marmota%“


BioCASe Provider Software (Wrapper)

Compatible with several protocols (BioCASe, DiGIR) and data schemas (ABCD, ABCD-EFG, ABCD-DNA, DarwinCore)

Works with all SQL-compliant databases (Access, mySQL, Postgres, SQL Server, Oracle, ...)

Currently 84 production installations serving 1.315 collections

Platform independent Support

available!


BioCASe Providers Worldwide

84 production installationsserving 1.315 collections


Requirements

- SQL compliant database (with existing Python connectivity module)

- Webserver (preferrably Apache), allowing the execution of Python scripts

- Possibility to install additional Python packages


Steps

1. Installing Apache

2. Installing Python

3. Downloading BPS (from repository/archive)

4. Installing BPS

5. Creating the link Apache/BPS

6. Testing BPS, Setup of additional packages

7. Changing directory permissions


1. Installing Apache

http://httpd.apache.org/download


2. Installing Python

http://www.python.org/download/


3. Downloading BPS

Archive: http://www.biocase.org/products/provider_software/

Subversion repository

Trunk version: http://ww2.biocase.org/svn/bps2/trunk

Defined version: http://ww2.biocase.org/svn/bps2/tags/release_2.5.2

Linux:

svn co <url> <path>

Windows: Tortoise client


4. Installing the BPS

Setup.py

No files copies,

only adapted!


5. Linking BPS with Apache

http.conf


6. Testing BPS, Installing Additional Packages

http://localhost/biocase Utilities Library Test


6a: mysqldb

http://sourceforge.net/projects/mysql-python/


7. Write permissions

... /bps2/configuration


Changing the Password

... /bps/configuration.ini


3: ABCD, HISPID

Protocols/Data Standards

Wrapper Software


Applications

Data Portal


ABCD Data Schema

Access to Biological Collection Data:

Data schema for all types of primary biodiversity data (living/preserved/observational, botanical/zoological/bacterial/viral, marine/terrestrial)

XML (eXtensible Markup Language) based can be consumed by humans and machines

Highly complex, hierarchical, currently 1,055 data elements almost every data item will fit in

Extendable (plug-in slot for additional information)

standard (currently version 2.06)


ABCD: Structure

Namespace: http://www.tdwg.org/schemas/abcd/2.06


ABCD Metadata: Technical/Content Contact


ABCD Metadata: Intellectual Property Rights


ABCD Metadata: Representation, Owner, ...


ABCD: Triple ID, Record Basis


ABCD: Identification (multiple)


ABCD: Gathering Event


ABCD: Multimedia


ABCD: Unit Associations


ABCD: Specialised Portions

Specimen Unit: Acquisition, Accession, Peparation, Duplicate Distribution, Type Status

Herbarium Unit: Loan Information

Botanical Garden Unit: Location in Garden, Hardiness, Lineage, Cultivation, Planting Date

Other Specialised Subtrees forObservationsCulture CollectionsMycological UnitsZoological UnitsPaleontological UnitsPlant Genetic Resources


ABCD: UnitExtension

Own Namespace for Extension http://www.chah.org.au/schemas/hispid/5

Other Extensions: Extension for Geoscienes (ABCD-EFG) DNA Bank Network (ABCD-DNA)


HISPID

HISPID Gathering-Coordinates DMS-PersonCollector-Substrate/ParentRock-SoilType, Vegetation

HISPID Unit-LifeForm, Phenology-NonComputerisedDataFlag-DonorTyp, ProvenanceType

HISPID Identification

-HigherTaxon: Addition ranks


BioCASe Protocol

Biological Collection Access Service Protocol:

Manages data exchange between data providers (collections) and applications (data portals)

Vehicle for transporting requests: data portal collection and responses (ABCD documents): collection database data portal

XML based


BioCASe Protocol: Capabilities request


BioCASe Protocol: Inventory Request


BioCASe Protocol: Search Request

Beispielbild

4. Preparing the database for BioCASe


4. Reasons for not publishing the live DB

1. Publishing the live DB is not desired creating snapshots for publication

2. DBMS not accessible for the BPS export into another DBMS

3. Performance considerations (too highly normalized) partial, controlled denormalization

4. Repeatable elements kept in columns, not in separate rows Moving repeatable elements to separate records


Each repeatable elements needs its own primary key!

Repeatable elements kept in columns

specimen_id ... class order family

3476 ... Conjugatophyceae Desmidiales Desmidiaceae

3477 ... Conjugatophyceae Desmidiales Desmidiaceae

3478 ... Conjugatophyceae Desmidiales Closteriaceae

specimen_id ...

3476 ...

3477 ...

3478 ...

sp_id ht_entry ht_rank ht_name

3476 456765 class Conjugatophyceae

3476 456766 order Desmidiales

3476 456767 family Desmidiaceae



3477 456770 family Desmidiaceae



3478 456773 family Closteriaceae


CREATE VIEW [dbo].[vwHigherTaxa]

AS

SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rankFROM unit_dataWHERE [kingdom] IS NOT NULL

UNION

SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum'FROM unit_dataWHERE [phylum] IS NOT NULL

UNION

SELECT 'c_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [class], 'class'FROM unit_dataWHERE [class] IS NOT NULL

UNION

...

Example View


Commonly used repeatable elements

- Identification- HigherTaxon- GatheringSite/NamedArea- Metadata/Scope/GeoecologicalTerms- Metadata/Scope/TaxonomicTerms- MultimediaObjects- MeasurementsOrFacts- ...


Controlled Denormalization

insert into [dbo].[abcd_Object]

SELECT dbo.CollectionObject.CollectionObjectID, ISNULL(dbo.CatalogSeries.SeriesName, '') + '-' + ISNULL(CAST(dbo.CollectionObjectCatalog.SubNumber AS nvarchar(20)), '') + '-' + ISNULL(CAST(dbo.CollectionObjectCatalog.CatalogNumber AS nvarchar(20)), ''), dbo.f_getParentID(dbo.CollectionObject.CollectionObjectID), dbo.f_getCollectingEventID(dbo.CollectionObject.CollectionObjectID), dbo.f_getFieldNumber(dbo.CollectionObject.CollectionObjectID), cast(dbo.CollectionObjectCatalog.CatalogNumber as int), dbo.CollectionObject.PreparationMethod, case when Sex = '<No Data>' then NULL else Sex end, case when Stage = '<No Data>' then NULL else Stage end, case when dbo.CollectionObject.Text1 is null then '' else 'Barcode: ' + dbo.CollectionObject.Text1 + '; ' end + case when dbo.Accession.Number is null then '' else 'Specimen Location: ' + dbo.Accession.Number end + case when DerivedFrom.Remarks is null then '' else ' <br> ' + cast(DerivedFrom.Remarks as nvarchar(2000)) end

FROM dbo.BiologicalObjectAttributes RIGHT OUTER JOIN dbo.CollectionObject ON dbo.BiologicalObjectAttributes.BiologicalObjectAttributesID = dbo.f_getParentID(dbo.CollectionObject.CollectionObjectID)

LEFT OUTER JOIN dbo.CollectionObjectCatalog LEFT OUTER JOIN dbo.CatalogSeries ON dbo.CollectionObjectCatalog.CatalogSeriesID = dbo.CatalogSeries.CatalogSeriesID ON dbo.CollectionObject.CollectionObjectID = dbo.CollectionObjectCatalog.CollectionObjectCatalogID

LEFT JOIN dbo.Accession on Accession.AccessionID = CollectionObjectCatalog.AccessionID

LEFT JOIN dbo.CollectionObject AS DerivedFrom ON CollectionObject.DerivedFromID = DerivedFrom.collectionObjectID

WHERE (dbo.f_hasChildObjects(dbo.CollectionObject.CollectionObjectID) = 0) AND ...


How Do I See Someting is Wrong?

Errors in ABCD documents:-Several datasets (one for each unit)-Several units for one specimen record

Reasons:- Repeatable elements not in separate tables (no separate PK several units will be created)- Several records in DB for non-repeatable elements (several ABCD objects are necessary to create a valid document)

Beispielbild

5. Setting Up a BioCASe Data Source:Database connection, Table Setup, Schema Mapping


BPS Datasource

URL for a BioCASe protocol compliant webservice:http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=AlgenEngels

<?xml version='1.0' encoding='UTF-8'?><request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header> <type>search</type> </header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'> http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/ Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search></request>


BPS QueryTools

Tool for sending Scan, Search and Capabilities Requests to a datasource

Choose datasource „Debug“


Steps for Setting Up a Datasource

1. Create a new Datasource

2. Configure Datasource:1. Database Connection

2. Table Setup

3. Create new empty Mapping

4. Edit Mapping:

1. Choose root table

2. Edit mandatory ABCD elements (red)

3. Save Configration, test datasource (QueryTools)

4. Add additional ABCD elements, occasional testing

3. Test Datasource


Datasource Loglevel

The lower the loglevel, the more information is logged.(10=info, 20=debug, 30=warning, 40=error)


How The BPS performs requests

1. Get a ID list of records matching the filter

2. Loading all details for the matching IDs Joining of ALL tables, beginning with the root table (table with UnitID, one record per Unit)


Datasources folder

... /bps/configuration/datasources/<dsname>

querytool_prefs.xmlJust what its name says.

xxxx.pickTemporary files; should be deleted if BPS behaves strangely.

cmf_xxxxxx.xmlConcept mapping; one for each supported schema.

provider_setup_file.xmlDatabase conncetion, table setup, supported schemas.

Regular backup of configuration folder is highly recommended!


Metadata tables

If metadata differ for each or some of the records: several records in metadata table, linked to unit by foreign key

If metadata is unique for all records possible to hold data in one record no reference key is needed static table


Applications

2. Wrapper Software

1. Protocols/Data Standards


3. Applications

Data Portal


Local QueryTool


Distibuted Search vs. Harvesting/Caching

GeoCASe Distributed Search: http://search.biocase.org/geocase


GBIF Registration


GBIF Data Portal


BioCASe European data portal


EDIT Specimen Explorer


Data Mining: Itineraries Project

Goal: Detect itinerary patterns in geo-referenced primary data presumably collected during a collecting event.

1st step:

Try to validate itineraries from well-documented expeditions (literature) against geo-referenced primary biodiversity records with dates/collecting information

2nd step:

Try to find itineraries for collecting events with missing expedition diaries


Data Mining: Ecological Niche Modelling


Jörg Holetschek

Botanischer Garten & Botanisches MuseumAbteilung Biodiversitätsinformatik & Labors

Königin-Luise-Straße 6-814195 Berlin-Dahlem

[email protected]. +49 30 838 50150

0448 831 980

www.bgbm.org/biodivinf

www.biocase.orgsearch.biocase.orgsearch.biocase.de

http://hiscom.chah.org.au/wiki/BioCASe_Workshop

Documents

Beispielbild The BioCASe Technology Jörg Holetschek Botanic Garden & Botanical Museum Berlin-Dahlem Dept. of Biodiversity Informatics and Laboratories