Wf4Ever: Scientific Workflows and Research Objects as tools for scientific insight and methodology curation

Wf4Ever: Scientific Workflows and Research Objects as tools for scientific insight and methodology curation

Juande Santander-Vela [email protected] de Astrofísica de Andalucía-CSIC

Talk Outline

Introduction

Current challenges for radio astronomy and science

Potential e-Science solutions: Workflows and Research Objects

Final points

Introduction

Who am I?Member of the AMIGA international collaboration, based at IAA-CSIC

Ph.D. on bringing Radio Astronomical data archives and tools into the VO

Applied Scientist at ESO VLT archive, Software Engineer/Astronomy Specialist at ALMA archive (May 2009-Dec 2011)Back to IAA-CSIC as VIA-SKA Project Manager, Radio Astroinformatician

GROUP INTEREST IN TECH DEVELOPMENTS FOR BETTER SCIENCE

Why I’m here?

Collaboration with Stephane Leon and the ALMA Data Management Group

Helping bring the ALMA Science Archive to the VO ⬅ Modelling radio data cubesFinding use cases for workflow technology (see later)

AMIGAAnalysis of the interstellar Medium of Isolated GAlaxies

Multi-wavelength, multi-object study on isolated galaxies with strict isolation criteriaCareful curation of dataVery careful processing of new parameters from

Group’s own observation programs and data reductionLiterature table scanningVirtual Observatory table harvesting and parsing

Emphasis on marrying astronomy and computer science, and buy-in of the VO

E-SCIENCE USERS

AMIGAAnalysis of the interstellar Medium of Isolated GAlaxies

Multi-wavelength, multi-object study on isolated galaxies with strict isolation criteriaCareful curation of dataVery careful processing of new parameters from

Group’s own observation programs and data reductionLiterature table scanningVirtual Observatory table harvesting and parsing

Emphasis on marrying astronomy and computer science, and buy-in of the VO

E-SCIENCE DEVELOPERS!

AMIGA

Project goal: providing a baseline for galaxy properties to compare with other environments

Interaction-free sample, ideal for tracing HI infall: we can use CIG galaxies to detect the cosmic web

Need for very sensitive telescopes able to resolve faint HI ➡ Square Kilometre Array & pathfinders

PARTICIPATING IN SKA.TEL.SDP CONSORTIUMWE NEED TOOLS FOR OUR OWN SCIENCE ANALYSIS

⤷

Current challenges for radio astronomy

and science

Data over-abundanceMoore’s Law for Detectors ➡ Exponential increase of individual and accumulated data sets

We have more data than ever… but we can’t use it:Because we can’t:

Difficult to set up (for sharing)Difficult to find (for using)Difficult to document (both using and sharing)Difficult to deal with (because of size, formatting, purpose…)

Because it is not in our best interest

FULLY

?

Courtesy J.E. Ruiz (AMIGA, Wf4Ever)

Courtesy J.E. Ruiz (AMIGA, Wf4Ever)

Tools!

Data sharingSearch Go

Advanced search

Home News & Comment Research Careers & Jobs Current Issue Archive Audio & Video For Authors

SPECIALS See all specials

Editorial Feature Opinion Elsewhere in Nature

DATA SHARINGSharing data is good. But sharing your own data? That can get complicated. As two researchcommunities who held meetings in May on the issue report their proposals to promote data sharingin biology, a special issue of Nature examines the cultural and technical hurdles that can get in theway of good intentions.

Data SharingSpecials & supplements archiveArchive

DATA FLIRTING

DATA HOARDING

IRREPRODUCIBLE RESEARCH

?

Irreproducible researchSearch Go

Advanced search


SPECIAL See all specials

Editorial News and analysis Comment Perspectives and reviews

CHALLENGES IN IRREPRODUCIBLE RESEARCHNo research paper can ever be considered to be the final word, and the replication andcorroboration of research results is key to the scientific process. In studying complex entities,especially animals and human beings, the complexity of the system and of the techniques can alltoo easily lead to results that seem robust in the lab, and valid to editors and referees of journals,but which do not stand the test of further studies. Nature has published a series of articles aboutthe worrying extent to which research results have been found wanting in this respect. The editorsof Nature and the Nature life sciences research journals have also taken substantive steps to putour own houses in order, in improving the transparency and robustness of what we publish.Journals, research laboratories and institutions and funders all have an interest in tackling issuesof irreproducibility. We hope that the articles contained in this collection will help.

Free full access

Challenges in irreproducible researchSpecials & supplements archiveArchive

nature.com Sitemap Cart Login Register

Search GoAdvanced search

Home News & Comment Research Careers & Jobs Current Issue Audio & Video For Authors


EDITORIAL

NEWS AND ANALYSIS

COMMENT



Free full access

Reducing our irreproducibilityNature 496, 398 ( 25 April 2013 )

Further confirmation neededA new mechanism for independently replicating research findings is one of several changes required toimprove the quality of the biomedical literature.Nature Biotechnology 30, 806 ( 10 September 2012 )

Error proneBiologists must realize the pitfalls of work on massive amounts of data.Nature 487, 406 ( 26 July 2012 )

Must try harderToo many sloppy mistakes are creeping into scientific papers. Lab heads must look more rigorously at thedata — and at themselves.Nature 483, 509 ( 29 March 2012 )

Independent labs to verify high-profile papersMonya BakerNature News ( 14 August 2012 )

Power failure: why small sample size undermines the reliability of neuroscienceKatherine S. Button, John P. A. Ioannidis et al.Nature Reviews Neuroscience 14, 365-376 ( 15 April 2013 )

Replication studies: Bad copyEd YongNature 485, 298-300 ( 17 May 2012 )

Why animal research needs to improveMalcolm MacleodNature 477, 511 ( 29 September 2011 )

Reliability of 'new drug target' claims called into questionAsher MullardNature Reviews Drug Discovery 10, 643-644 ( September 2011 )

If a job is worth doing, it is worth doing twiceJonathan F. RussellNature 496, 7 ( 04 April 2013 )


Irreproducible researchSearch Go

Advanced search





Free full access


nature.com Sitemap Cart Login Register

Search GoAdvanced search

Home News & Comment Research Careers & Jobs Current Issue Audio & Video For Authors


EDITORIAL

NEWS AND ANALYSIS

COMMENT



Free full access

Reducing our irreproducibilityNature 496, 398 ( 25 April 2013 )

Further confirmation neededA new mechanism for independently replicating research findings is one of several changes required toimprove the quality of the biomedical literature.Nature Biotechnology 30, 806 ( 10 September 2012 )

Error proneBiologists must realize the pitfalls of work on massive amounts of data.Nature 487, 406 ( 26 July 2012 )

Must try harderToo many sloppy mistakes are creeping into scientific papers. Lab heads must look more rigorously at thedata — and at themselves.Nature 483, 509 ( 29 March 2012 )

Independent labs to verify high-profile papersMonya BakerNature News ( 14 August 2012 )

Power failure: why small sample size undermines the reliability of neuroscienceKatherine S. Button, John P. A. Ioannidis et al.Nature Reviews Neuroscience 14, 365-376 ( 15 April 2013 )

Replication studies: Bad copyEd YongNature 485, 298-300 ( 17 May 2012 )

Why animal research needs to improveMalcolm MacleodNature 477, 511 ( 29 September 2011 )

Reliability of 'new drug target' claims called into questionAsher MullardNature Reviews Drug Discovery 10, 643-644 ( September 2011 )

If a job is worth doing, it is worth doing twiceJonathan F. RussellNature 496, 7 ( 04 April 2013 )


Tool over-abundance

++

Starship Asterisk*APOD and General Astronomy Discussion Forum

Board index ‹ Learning & Resources ‹ The Engineering Deck: Astrophysics Source Code Library

FAQ Register Login

Search this forum… Search 671 topics • Page 1 of 7 • 1 2 3 4 5 ... 7

The Engineering Deck: Astrophysics Source Code Library

Search… SearchAdvanced search

Post a new topic

ANNOUNCEMENTS REPLIES VIEWS LAST POST

Welcome & Rules (please read before posting) by RJN » Mon Jan 18, 2010 7:40 pm 0 15666 by RJN

Mon Jan 18, 2010 7:40 pm

TOPICS REPLIES VIEWS LAST POST

Guide to the Astrophysics Source Code Library by RJN » Sat Jul 24, 2010 8:01 pm 13 17027 by owlice

Mon Jul 01, 2013 3:32 am

1 2Papers of Possible Interest to Astronomical Software Users by owlice » Tue Oct 12, 2010 7:02 am 27 7056 by owlice

Wed May 15, 2013 1:31 pm

1 2The Astrophysics Source Code Library: New codes welcome by RJN » Sat Jul 24, 2010 8:01 pm 26 5273 by Eran Ofek

Thu Dec 13, 2012 9:20 pm

*Web Resources and Tools for Astrophysicists/Astronomers* by owlice » Sat Jul 16, 2011 12:01 pm 22 2750 by owlice

Fri May 10, 2013 12:12 pm

2011 and 2012 Additions to the ASCL by owlice » Thu Feb 24, 2011 11:26 pm 23 1693 by owlice

Sat Dec 08, 2012 8:09 pm

21cmFAST: Simulation of the High-Redshift 21-cm Signal by owlice » Thu Feb 17, 2011 10:47 pm 0 3443 by owlice

Thu Feb 17, 2011 10:47 pm

2LPTIC: 2nd-order Lagrangian Perturbation Theory Initial Con by owlice » Tue Jan 03, 2012 5:27 am 0 855 by owlice

Tue Jan 03, 2012 5:27 am

2MASS Kit: 2MASS Catalog Server Kit by owlice » Sun Mar 17, 2013 5:16 pm 0 214 by owlice

Sun Mar 17, 2013 5:16 pm

3DEX: Fast Fourier-Bessel Decomposition of Spherical 3D Surv by owlice » Sat Nov 26, 2011 4:00 pm 0 741 by owlice

Sat Nov 26, 2011 4:00 pm

AAOGlimpse: Three-dimensional Data Viewer by owlice » Sat Oct 15, 2011 11:29 am 0 1034 by owlice

Sat Oct 15, 2011 11:29 am

ACORNS-ADI: Calibration, Registration and Nulling in Imaging by kcd » Sat Mar 30, 2013 7:40 am 0 177 by kcd

Sat Mar 30, 2013 7:40 am

ACS: ALMA Common Software by kcd » Sat Feb 09, 2013 3:44 am 0 269 by kcd

Sat Feb 09, 2013 3:44 am

AdaptaHOP: Subclump finder by kcd

671 topics • Page 1 of 7 •

Services too!

How to deal with all this?

++All of this compounds the problems of reproducibility, methodology assessment, result dissemination…


AND THE CODE?

WHAT SOFTWARE DOES IT DEPEND ON?

WHICH CODE DID WHAT?

NOT

A GOOD

SOLUTION

TRADITIONALLY…


++ORCHESTATION,ENCAPSULATION,DATA ACCESS,PROVENANCE,

ANNOTATION…

Why Workflows?SCIENTIFIC

Workflows define computationsEvents & ProcessesDependencies

ResourcesLocal & Remote ProcessesSequencesConcurrencesTriggers

FORMALLY,OR AT LEAST

MACHINE READABLE

➡

WORKFLOWDEFINITIONLANGUAGES

Workflows enable distributed computing

Distributed computing paradigmMove computation to the dataComputing services

Collaborative environments

Linked data ｝FOR SCIENTIFIC DISCUSSION &

SCIENCE EXTRACTION

➡ Science-computing

Workflows enable distributed computing

Data can be anywhere

Workflows can be constructed hierarchicaly

Each workflow does useful work on its own

The data flow can be easily followed

Workflows enable interactive computing

Each workflow run records it’s inputs, outputs, and intermediate results

You can build and run workflows incrementally

You can get (almost) immediate feedback on changes

Tools for workflow storage and discovery

About | Give us Feedback | Publications Juandesant

New Workflow GO Workflows Search

View

Download (v7)

Taverna 2

OriginalUploader

PaulFisher

View

Download (v4)

Taverna 1

OriginalUploader

MarcoRoos

View

Download (v3)

Taverna 1

OriginalUploader

HamishMcWilliam

View

Download (v4)

Taverna 1

OriginalUploader

MarkBorkum

View

Download (v4)

Taverna 1

OriginalUploader

PaulFisher

View

Download (v5)

Taverna 1

OriginalUploader

PaulFisher

View

Download (v4)

Taverna 1

OriginalUploader

AlanWilliams

View

Download (v1)

Taverna 1

OriginalUploader

M.B.Mon...

View

Download (v1)

Taverna 1

OriginalUploader

AlanWilliams

View

Download (v3)

Taverna 1

OriginalUploader

AndreaWiggins

Sort by: Rank« Previous 1 2 3 4 5 … 221 Next »

1111562243

43342624231813

223103102102

827775727167

268908268645345353533

1472339317

251310

9631

264776142322725222221

88432524241917161514

42302723191918171311

Search filter terms

Filter by type Taverna 2 Taverna 1 RapidMiner Kepler Bioclipse Scri… LONI Pipeline GWorkflowDL KNIME BioExtract Ser… Galaxy

Filter by tag example mygrid bioinformatics localworker cheminformatics benchmarks protein workflow votable ebi

Filter by user Alan Williams Paul Fisher Antoon Goderis Peter Li Anja Le Blanc Hamish McWil… Francois Belle… Rafael C. Jim… Stian Soiland-… Egon Willigha…

Filter by licence by-sa by-nd by BSD GPL Apache LGPL by-nc-sa CC0 MIT

Filter by group myGrid HELIO IST 600 SCAPE SabrOndexPr… AMIGA e-LICO BioSemantics MediGRID TextGrid

Filter by wsdl http://soap.ge… http://xml.nig.a… http://phoebus… http://eutils.nc… http://xml.nig.a… http://www.ebi… http://www.ebi… http://www.ch… http://cagnode… http://festung1…

Filter by curation Decommissio… Requires exa… Requires des… Runnable Obsolete - de… Test workflow Example data Not runnable Tutorial / exa… Whole solution

« Previous 1 2 3 4 5 … 221 Next »Sort by: Rank

Results per page: 10

Home Users Groups Workflows Files Packs Topics

Home > Workflows

Workflows

Showing 2207 results. Use the filters on the left and the search box below to refine the results.

Search

Pathways and Gene annotations for QTL region (7)

Created: 19/11/09 @ 18:18:52 | Last updated: 07/09/12 @ 18:23:36

Credits: Paul Fisher

License: Creative Commons Attribution-Share Alike 3.0 Unported License

This workflow searches for genes which reside in a QTL (Quantitative Trait Loci) region in themouse, Mus musculus. The workflow requires an input of: a chromosome name or number; a QTLstart base pair position; QTL end base pair position. Data is then extracted from BioMart to annotateeach of the genes found in this region. The Entrez and UniProt identifiers are then sent to KEGG toobtain KEGG gene identifiers. The KEGG gene identifiers are then used to searcg for pathways inthe KEGG path...

Rating: 4.6 / 5 (10 ratings) | Versions: 7 | Reviews: 1 | Comments: 7 | Citations: 1

Viewed: 7291 times | Downloaded: 1283 times

Tags (22):

adasd | chromosome | data-driven | disease | ensembl | entrez | gene | genes | genotype | kegg | mouse | nbiconworkflows | pathway |pathway-driven | pathways | phenotype | qtl | shim | subworkflow | uniprot

BioAID_DiseaseDiscovery_RatHumanMouseUnipr... (4)


Credits: Marco Roos AID


This workflow finds disease relevant to the query string via the following steps: 1. A user query:a list of terms or boolean query - look at the Apache Lucene project for all details. E.g.: (EZH2OR "Enhancer of Zeste" +(mutation chromatin) -clinical); consider adding

'ProteinSynonymsToQuery' in front of the input if your query is a protein. 2. Retrieve documents: finds'maximumNumberOfHits' relevant documents (abstract+title) based on query (the AIDA service inside is based onApa...



Tags (9):

AIDA | BioAID | biorange_nl | disease | pkna | protein | text_mining | text_mining_network | VL-e

EBI_InterProScan (3)


Credits: Hamish McWilliam

License: Creative Commons Attribution 3.0 Unported License

Note: the WSInterProScan web service used by this workflow is no longer available haveing beenreplaced by the EMBL-EBI's InterProScan (REST)(http://www.ebi.ac.uk/Tools/webservices/services/pfa/iprscan_rest) and InterProScan (SOAP)(http://www.ebi.ac.uk/Tools/webservices/services/pfa/iprscan_soap) web services. Thus theworkflow described here no longer works, see the alternative workflows for the InterProScan(SOAP) service for workflows which use the new services. Perform an InterProScan ...



Tags (10):

bioinformatics | ebi | gff | go terms | interpro | interproscan | protein annotation | protein domain | protein family | protein motif

QR code (matrix code) generator (4)



The latest (and greatest) version of the QR code generator workflow. This workflow uses theQR code service provided by the ChemTools project.



Tags (4):

barcode | chemtools | qr | qrcode

HUMAN Microarray CEL file to candidate pat... (4)


License: Creative Commons Attribution-No Derivative Works 3.0 Unported License

This workflow takes in a CEL file and a normalisation method then returns a series ofimages/graphs which represent the same output obtained using the MADAT software package(MicroArray Data Analysis Tool) [http://www.bioinf.manchester.ac.uk/MADAT/index.html]. Alsoreturned by this workflow are a list of the top differentialy expressed genes (size dependant onthe number specified as input - geneNumber), which are then used to find the candidatepathways which may be influencing the observed ...



Tags (12):

affymetrix | cel | file | genotype | kegg | microarray | nbiconworkflows | pathway-driven | pathways | phenotype | shim

Entrez Gene to KEGG Pathway (5)


Credits: Paul Fisher


This workflow takes in Entrez gene ids then adds the string "ncbi-geneid:" to the start of each gene id.These gene ids are then cross-referenced to KEGG gene ids. Each KEGG gene id is then sent to theKEGG pathway database and its relevant pathways returned.



Tags (9):

entrez | genotype | kegg | ncbi | pathway | pathway-driven | pathways | phenotype | shim

BiomartAndEMBOSSAnalysis (4)

Created: 15/09/09 @ 15:11:11


Using Biomart and EMBOSS soaplab services, This workflow retrieves a number of sequencesfrom 3 species: mouse, human, rat; align them, and returns a plot of the alignment result.Corresponding sequence ids are also returned.



Tags (6):

bioinformatics | biomart | emboss | example | mygrid | taverna

Workflow for Protein Sequence Analysis (1)


Credits: M.B.Monteiro

Attributions: BLAST using DDBJ service Simplify a BLAST text file conditional branch


This workflow performs a generic protein sequence analysis. In order to do that a novel proteinsequence enters into the software along with a list of known protein identifiers chosen by thebiologist to perform a homology search, followed by a multiple sequence alignment and finally aphylogenetic analysis.



Tags (10):

analysis | BLAST | homology | multiple sequence alignment | phylogenetic tree | protein | sequence | tree | wpsa

Example of a conditional execution workflow (1)


Credits: Tomoinn


If the input is true then the string 'foo' is emited, if false then 'bar'. Just a simple example to show howthe monster works, so to speak. Note that this type of workflow is not suitable for Taverna 2. Analternative mechanism is recommended.



Tags (5):

condition | design pattern | example | mygrid | taverna

Success-Abandonment-Classification (3)


Credits: Andrea Wiggins James Howison


Retrieves data from FLOSSmole and from the Notre Dame SourceForge repository to computeproject statistics based on releases, downloads and project lifespan. Project statistics are thenused to classify projects according to the criteria set up in English & Schweik, but comparisoncriteria are parameterized so that a different set of criterion thresholds can be used to evaluatethe project characteristics.



Tags (6):

classification | draft | floss | open source software | oss | social sciences

Copyright © 2007 - 2012 The University of Manchester and University of Southampton

Front PageHomeInvite people to myExperimentHelp pages

About UsNews and EventsMailing ListContact UsDevelopersPublications

Taverna Workflow WorkbenchmyGridBioCatalogueTridentGoogle Coop Search

EPSRCJISC

Microsoft

Powered by:

Icons: Silk icon set 1.3

Tools for workflow storage and discovery


New Workflow GO Astrotaverna Workflows Search

View

Download (v3)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v1)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v1)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v1)

Taverna 2

OriginalUploader

JoseEnrique

Ruiz

View

Download (v2)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v2)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v1)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v3)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v1)

Taverna 2

OriginalUploader

JoseEnrique

Ruiz

View

Download (v2)

Taverna 2

OriginalUploader

JoseEnrique

Ruiz

Sort by: Relevance« Previous 1 2 3 4 5 Next »

44

4342402623

99955

2717

404

164

Search filter terms

Filter by type Taverna 2

Filter by tag astronomy astrotaverna votable virtual observ… starter pack local processes taverna workflow galfit sextractor

Filter by user Jose Enrique … Julian Garrido

Filter by licence by-sa BSD

Filter by group AMIGA Wf4Ever

« Previous 1 2 3 4 5 Next »Sort by: Relevance



Home > Workflows

Workflows


Astrotaverna Search

Remove search query

Cocatenates several VOTables into one (3)


Credits: Julian Garrido


Snippet showing how to use AstroTaverna tool for concatenating several VOTables. The inputis four VOTables with the same number of columns. The result if using sample values providedwill be a four times vertically duplicated VOTable.



Tags (4):

astronomy | astrotaverna | cat | votable

Create configuration files from a template... (1)




This workflow uses astrotaverna artifacts. It creates files by using a template whose keys arereplaced by data from a votable. A configuration file is created for every row in the votable. Thekeys must appear also in the vocabulary file and match column names in the votable. A columnin the votable must contain the name of the result configuration file.



Tags (4):

astronomy | astrotaverna | local processes | votable

Simulates the physical, dynamical, and che... (1)

Created: 17/05/13 @ 08:03:13



Simulates the physical, dynamical, and chemical structure of a shocked layer of the ISM. Itcalls an asynchronous PDL service at Paris Server using a very simple loop condition (until

status is finished).


Viewed: 1 time | Downloaded: 0 times

Tags (4):

astrotaverna | ism | pdl service | simulation

Astronomical object name to equatorial coo... (1)


Credits: Jose Enrique Ruiz

License: BSD License

Resolve equatorial coordinates of a list of objects (names) with AstroTaverna plugin



Tags (5):

astronomy | astrotaverna | coordinates | virtual observatory | votable

Run scripts from a column in a votable (2)




It runs scripts whose pathname are values of a column in a votable



Tags (6):

astronomy | astrotaverna | ellipse | iraf | pyraf | votable

Create votable from ellipse results (2)




The workflow takes from a votable the names of files where the ellipse result is stored. These filescontain data for several ellipses and the workflow takes the center from the inner ellipse and theellipticitiy from one of the outter ones. It uses astrotaverna plugin(http://wf4ever.github.com/astrotaverna/).



Tags (7):

astronomy | astrotaverna | ellipse | iraf | pyraf | sextractor | votable

Add columns to a votable resulting from ex... (1)

Created: 30/08/12 @ 09:33:55



Addcolumns that are needed to run galfit and ellipse. it requires columns coming fromsextractor and the astrotaverna plugin.



Tags (7):

astronomy | astrotaverna | galaxy | sextractor | taverna | votable | workflow

Extract a column from a VOTable into a List (3)


Credits: Julian Garrido Susana


Snippet showing how to use AstroTaverna tool for extracting a column from a VOTable into asingle List.


Viewed: 26 times | Downloaded: 1 time

Tags (5):

astronomy | astrotaverna | starter pack | virtual observatory | votable

Discovery of Brown Dwarfs mining the 2MASS... (1)

Created: 12/04/13 @ 16:37:41



Brown dwarfs are objects occupying the gap between the least massive stars and the mostmassive planets. They are intrinsically faint objects so their detection is not straighforward and,in fact, was almost impossible until the advent of global surveys at deep optical and near-infrared bands like SDSS, 2MASS or DENIS among others. Here we propose to mine theSDSS and 2MASS databases to identify T-type brown dwarfs through an appropriate

combination of colours in the optical and the infra-red...



Tags (5):


Crossmatching VOTables (2)




Snippet showing how to use AstroTaverna tool for crossmatching two VOTables VOTablesamples may be downloaded from MyExperimenthttp://www.myexperiment.org/files/911/versions/1/download/SDSS7.vot

http://www.myexperiment.org/files/910/versions/1/download/2MASS.vot



Tags (5):






EPSRCJISC

Microsoft

Powered by:




View

Download (v3)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v1)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v1)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v1)

Taverna 2

OriginalUploader

JoseEnrique

Ruiz

View

Download (v2)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v2)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v1)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v3)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v1)

Taverna 2

OriginalUploader

JoseEnrique

Ruiz

View

Download (v2)

Taverna 2

OriginalUploader

JoseEnrique

Ruiz


44

4342402623

99955

2717

404

164

Search filter terms









Home > Workflows

Workflows


Astrotaverna Search

Remove search query








Tags (4):









Tags (4):



Created: 17/05/13 @ 08:03:13







Tags (4):









Tags (5):









Tags (6):









Tags (7):



Created: 30/08/12 @ 09:33:55






Tags (7):









Tags (5):



Created: 12/04/13 @ 16:37:41







Tags (5):










Tags (5):






EPSRCJISC

Microsoft

Powered by:




View

Download (v3)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v1)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v1)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v1)

Taverna 2

OriginalUploader

JoseEnrique

Ruiz

View

Download (v2)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v2)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v1)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v3)

Taverna 2

OriginalUploader

JulianGarrido

View

Download (v1)

Taverna 2

OriginalUploader

JoseEnrique

Ruiz

View

Download (v2)

Taverna 2

OriginalUploader

JoseEnrique

Ruiz


44

4342402623

99955

2717

404

164

Search filter terms









Home > Workflows

Workflows


Astrotaverna Search

Remove search query








Tags (4):









Tags (4):



Created: 17/05/13 @ 08:03:13







Tags (4):









Tags (5):









Tags (6):









Tags (7):



Created: 30/08/12 @ 09:33:55






Tags (7):









Tags (5):



Created: 12/04/13 @ 16:37:41







Tags (5):










Tags (5):






EPSRCJISC

Microsoft

Powered by:



New Workflow GO All Search

What is this?Linked Data

Version 3 (latest) (of 3) View version: 3 (latest)

Version created on: 22/04/13 @ 16:52:00 by: Julian Garrido

Title: Cocatenates several VOTables into oneType: Taverna 2

Preview

(Click on the image to get the full size)

Download Scalable Diagram (SVG)

Description

Snippet showing how to use AstroTaverna tool for concatenating several VOTables. The input is fourVOTables with the same number of columns. The result if using sample values provided will be a four timesvertically duplicated VOTable.

Download

Download Workflow File/Package (T2FLOW)

Download Workflow as a Galaxy tool

Run

Run this Workflow in the Taverna Workbench...

Option 1:

Copy and paste this link into File > 'Open workflow location...'http://www.myexperiment.org/workflows/3130/download?version=3

[ More Info ]

Workflow Components

Authors (1)

Titles (1)

Descriptions (1)

Dependencies (0)

Inputs (4)

Processors (1)

Beanshells (0)

Outputs (1)

Datalinks (5)

Coordinations (0)

Workflow TypeTaverna 2

Original Uploader

JulianGarrido

LicenseAll versions of this Workflow are licensed

under:

Credits (1)(People/Groups)

Julian Garrido

Attributions (0)(Workflows/Files)

None

Tags (4)

Original Uploader tags

astronomy | astrotaverna | cat |

votableAdd Tags

Shared with Groups (1)

AMIGA

Featured In Packs (1)

AstroTaverna Starter Pack

Ratings (0)Hover and click to rate

Current:

0.0 / 5(0 ratings)

You haven't rated yet

Breakdown

Attributed By (0)(Workflows/Files)

None

Favourited By (0)No one

Add to your Favourites

Statistics53 viewings

75 downloads[ see breakdown ]

More


Home > Workflows > Cocatenates several VOTables into one

Workflow Entry: Cocatenates several VOTables into oneCreated at: 30/08/12 @ 10:05:29 Last updated: 22/04/13 @ 16:52:00

| License | Credits (1) | Attributions (0) | Tags (4) | Featured in Packs (1) | Ratings (0) | Attributed By (0) | Favourited By (0) | | Citations (0) | Version History | Reviews (0) | Comments (0) |

Citations (0)

None

Version History

In chronological order:

1. Cat several votables into one

Created by Julian Garrido on Thursday 30 August 2012 @ 10:05:29 (UTC)

2. Cocatenates several VOTables into one

Created by Julian Garrido on Monday 15 April 2013 @ 08:52:18 (UTC)

Revision comment:

Addition of new metadata



Reviews (0)

No reviews yet

Be the first to review!

Comments (0)

View Timeline

No comments yet

Make a Comment:

Submit Comment

Other workflows that use similar services (0)

There are no workflows in myExperiment that use similar services to this Workflow.

Non-Information Resource URI: http://www.myexperiment.org/workflows/3130

Alternative Formats





EPSRCJISC

Microsoft

Powered by:








Preview



Description


Download



Run


Option 1:


[ More Info ]

Workflow Components

Authors (1)

Titles (1)

Descriptions (1)

Dependencies (0)

Inputs (4)

Processors (1)

Beanshells (0)

Outputs (1)

Datalinks (5)

Coordinations (0)


Original Uploader

JulianGarrido


under:


Julian Garrido


None

Tags (4)



votableAdd Tags


AMIGA




Current:

0.0 / 5(0 ratings)


Breakdown


None





More





Citations (0)

None

Version History






Revision comment:




Reviews (0)

No reviews yet


Comments (0)

View Timeline

No comments yet

Make a Comment:

Submit Comment




Alternative Formats





EPSRCJISC

Microsoft

Powered by:








Preview



Description


Download



Run


Option 1:


[ More Info ]

Workflow Components

Authors (1)

Titles (1)

Descriptions (1)

Dependencies (0)

Inputs (4)

Processors (1)

Beanshells (0)

Outputs (1)

Datalinks (5)

Coordinations (0)


Original Uploader

JulianGarrido


under:


Julian Garrido


None

Tags (4)



votableAdd Tags


AMIGA




Current:

0.0 / 5(0 ratings)


Breakdown


None





More





Citations (0)

None

Version History






Revision comment:




Reviews (0)

No reviews yet


Comments (0)

View Timeline

No comments yet

Make a Comment:

Submit Comment




Alternative Formats





EPSRCJISC

Microsoft

Powered by:


That’s not enough!

FOR ASTRONOMERS

FOR REPRODUCIBILITY AND REUSE

3

7 4

16

52

1. Intelligent Software Components (iSOCO, Spain)

2. University of Manchester (UNIMAN, UK)

3. Universidad Politécnica de Madrid (UPM, Spain)

4. Poznan Supercomputing and Networking Centre (PSNC, Poland)

5. University of Oxford (OXF, UK)6. Instituto de Astrofísica de Andalucía

(IAA, Spain)7. Leiden University Medical Centre

(LUMC, NL)

EU FUNDED FP7 STREP PROJECTDECEMBER 2010 – DECEMBER 2013

• Astronomy (IAA-CSIC)• Genome-wide Analysis and Biobanking

Case Studies

Archival, classification, and indexing of scientific workflows and their associated materials in scalable semantic repositories, providing advanced access and recommendation capabilities

Creation of scientific communities to collaboratively share, reuse, and evolve workflows and their parts, stimulating the development of new scientific knowledge

Goals

• Digital Libraries• Workflow Management • Semantic Web• Integrity & Authenticity• Provenance• Information Quality

Core Competencies (Tech)

• One SME• Six public organisations

Partners

Technological infrastructure for the preservation and efficient retrieval and reuse of scientific workflows in a range of disciplines

TARGETING ALREADY ESTABLISHED

COMMUNITIES: MYEXPERIMENT,

VIRTUAL OBSERVATORY

3

What is a Scientific Workflow? Workflows to Access and Massage VO Data

»  A mechanism for coordinating the execution of services and codes, and linking together resources.

»  The combination of data and processes into a configurable, modular, structured set of steps that implement semi-automated computational solutions in scientific problem-solving.

»  The implementation of a scientific method.

COURTESY J.E. RUIZ

NOT A PIPELINE!

AMIGA4GAS3D KINEMATICAL MODELING

INPUT FILES

ROTCUR

12 RUNSPOSSIBLE COMBINATIONS

IN INPUT PARAMETERS

12 ASCII FILES

GALMOD

12 CUBES 4 APPROACHING

4 RECEEDING 4 BOTH COPY

8 CUBES 4 APPROACHING + RECEEDING

4 BOTH

MOMENTS

8 VELOCITY MAPS

1 DATACUBE1 VELOCITY MAP

1 CONFIG FILE ROTCUR 1 CONFIG FILE GALMOD

SUB

8 RESIDUAL CUBES8 RESIDUAL MAPS

SUB

MNMX 8 VALUES FOR PEAKS IN CUBES8 VALUES FOR PEAKS IN MAPS

VARIABLEPARAMS

INSETRADII, WIDTHS

WEIGHTTOLERANCE

DENSNVZ0

VDISP

How do we build workflows?

AstroTavernaTaverna plugin for retrieving and manipulating VO Data + Catalogs on HTML Pages

VO Services: ConeSearch, SIA, SSA, TAP coming soonTabular Data (VOTables, converters from other formats)

Crossmatching, Filtering, NameResolving, Coordinates and reference system transformation, Data massage.. (STILTS)Source catalog overplotting on Images and filtering, overplot circles, ellipses, etc. as a function of physical magnitude. Resampling, crops, blinks, mosaics, movies, blinks, RGBs, fusion, diff.. (through Aladin)VO Table rendering, SAMP for final inspection

Image support, Spectra not yet PLUS ADDITIONAL ANALYSIS USING

SCRIPTS

Service discovery

Data massaging

Data massagingX-Matching

Calculation

Additions

Filtering

Access

Data curation

X-Matching

Calculation

Additions

Filtering

Access

Data curation

X-Matching

Calculation

Additions

Filtering

Access

Data curation

X-Matching

Calculation

Additions

Filtering

Access

Aladin scripting

Interactive data inspection

Interactive data inspection

Learning examples

Not yet enough!FOR REPRODUCIBILITY

AND REUSE

Home RO at 5000 feet Examples Ontologies Tools Collaboration Publications History About

Search

Research Objects

Research ObjectsContent

Process (workflows), data, external resources and bibliographyExecution environment set-up and local software dependenciesExperimental protocol followedRoles, types and relationships among all digital componentsProvenance of intermediate and final resultsDecomposable attribution and authoringFine-grained access control and permissionsExample datasets for demonstration, reproducibility, monitoring, etc

TemplatesPlaceholders to ease the aggregation processCompleteness checking/quality assessment

Research ObjectsTarget Audiencies

Scientists [producers] who want to share their research outcomes so that they are more reusable and reproducible – ease of sharing and citation.Scientists [consumers] who want to understand, reuse, validate and further extend existing RO’s.Publishers can adopt the concept and principles of Research Object to enable the sharing of and access to the actual data and methods.Librarians who want to support research preservation.

Semantic annotationsAuthor of an annotation Author and co-authors of a workflow; reference link to a re-used workflow and its author Who has performed the execution of a workflow leading to the results provided in the RO Computing execution environment of the RO and local software dependencies Special access requirements to web services Datasets provider: person, webpage, survey, data release, etc. How much time does it take to run a workflow using the full data and the provided subsampleThe number of elements of the sample dataset where one workflow and/or RO iteratesPrevious and subsequent workflows to be executed, as in the experimental protocolResearch institution, country, and scientific domain of the RO The actual size of the RO and/or a folder

Semantic model

DataLink

MULTI DISCIPLINARY

RO data organisation

Recommended organisation provides automatic semantics for some items

It makes it easier for both people and machines to understand the RO

ROs in AstronomyADSLabs Research Objects

Authors

Publications

Journals

Objects SIMBAD

Tabular data behind the plots CDS

ASCL reference of used software

Observing time Proposals

Used facilities, surveys or missions

NOT JUST FROM WORKFLOWS

POTENTIAL FOR RESEARCH OBJECT INDEXING IN ADS

RO IncentivePAPERS WITH DATA LINKS ARE CITED MORE THAN THOSE WITHOUT

Effect of E-printing on Citation Rates in Astronomy and Physics2006. Edwin A. Henneken et al.

RO IncentivePAPERS WITH DATA LINKS ARE CITED MORE THAN THOSE WITHOUT

Effect of E-printing on Citation Rates in Astronomy and Physics2006. Edwin A. Henneken et al.

NOW YOU CAN CITE DATA AND

PROCESSES, TOO

RoadmapAstroTaverna, mostly ready: you can publish workflows and packs to myExperiment from Taverna

myExperiment, building support for ROs

ADS will populate myExperiment with literature-ROs

Taverna will be able to publish ROs to myExperiment

Final points

We need something like workflows to describe computations in a distributed environment

Workflows are not enough for supporting reuse and methodology preservation

Research Objects are meaningful associations of data, operations, provenance, which can also be cited CAN EMBED

COMPUTATIONS IN SCIENCE ARCHIVES

Useful Links

http://www.wf4ever-project.org

http://www.myexperiment.org

http://www.researchobject.org

http://wf4ever.github.io/astrotaverna/

http://amiga.iaa.es

Thank you!

Technology

Wf4Ever: Scientific Workflows and Research Objects as tools for scientific insight and methodology curation