SCAPE

Alexandra Nenadic, David WithersUniversity of Manchester

Practical Tools for Digital Preservation: A Hack-a-thonYork, 28th September 2011

Taverna and myExperimentTools for creating and sharing workflows

Taverna

SCAPEWhat is a workflow?

• Connecting a set of tools/services to create automated and repeatable processing/analysis

SCAPE

Design and run workflows

Taverna

SCAPETaverna Features - Overview

• Access to remote, distributed or local services and resources

• Enables service interoperability and integration • Automates data flow between services• Implicit iteration over data sets, list handling and

control links to determine order of service invocation• Extensible

• Large external developer base• Various third party plugins available

• Data and provenance collection

SCAPETaverna Workbench

• Graphical desktop tool

• Drag-and-drop services into diagram

• Connect services, run, reconnect, rerun

• Integrates diverse set of tools

SCAPEWorkflow Design

Available services

Tree view of the workflow

structure

Workflow diagram

SCAPETaverna Workflows – Features in Detail

• A set of (local and remote) services to analyze or manage data

• Data-links connects services• i.e. output from service A is input to

service B and C• Describes the desired dataflow

instead of process coordination• Nested workflows are also services• Automatic iterations• Parallelization• Can customize list handling and

control links• Fault tolerance

• Retry (with delay and back off)• Failover (alternate services)

SCAPESupported Services

• SOAP/WSDL Web services• REST Web services• SoapLab Web services• R statistical services• Inline Beanshell scripts• External tools and scripts (via ssh or localhost)• Spreadsheet import• XPath and text manipulation services• SADI semantic Web services• Nested workflows (workflow within workflow)• BioMoby• BioMart • … your tool (write your own Taverna plugin)

SCAPEWorkflow Results

Input data and results per port

Progress report

Previous runs

SCAPEWorkflow Provenance

• Information about a workflow run• What happened?• And when?

• Lineage tracing• Which input produced which output

• Intermediate data• Inputs and outputs for each workflow step• Useful for debugging

• Saved in standard format (such as OPM)

SCAPETaverna is Domain-Independent

• Bioinformatics• Biomedicine• Chemistry

PharmacogenomicsAssociation study of Nevirapine-induced skin rash in Thai Population

HIV and TB research in South AfricaSleeping Sickness in African Cattle

Systems Biology for Crop research, BioDiversity

SCAPETaverna is Domain-Independent

• Astronomy• Data and text mining• Digital content preservation (IMPACT)• Social simulations

Astronomy & HelioPhysics

Observing Systems Simulation ExperimentsJPL, NASA

Library Document PreservationBritish Library

SCAPE

Share, discover and reuse workflows

SCAPEmyExperiment

• http://www.myexperiment.org • Social networking for people to share workflows and

collaborate• Makes it easy for people to contribute to a pool of

workflows, build communities and form relationships• Enables people to share, describe, reuse and

repurpose workflows, reduce time-to-production, share expertise and avoid reinvention

SCAPEmyExperiment

SCAPEWorkflow Sharing, Ownership and Attribution

• myExperiment can provide a central location for workflows from one community/group

• myExperiment allows you to say– Who can look at your workflow– Who can download your workflow– Who can modify your workflow– Who can run your workflow

• Workflow ownership and attribution• Users do not need to start from scratch – reuse or modify

existing workflows• Attribute/credit original author

SCAPEUse myExperiment from Taverna

SCAPETraining

• Tutorials and Training• 58+ tutorials to >900 people• >20 Universities, institutes

and networks• Major conferences• Summer schools

• Developer and User Days• Annotation Jamborees

• Undergraduate and Postgraduate Bioinformatics in > 30 Universities

SCAPETaverna and SCAPE

• SCAPE preservation components/actions as services in Taverna workflows

• Use Taverna Workbench to create and test SCAPE preservation workflows on local data

• Then scale-up and run the workflows on a parallelized platform using Hadoop MapReduce

• Share Taverna SCAPE workflows on myExperiment