Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam

The SEALS Platform for Scholarly Research

Asun Gómez-Pérezasun@fi.upm.es

Ontology Engineering GroupUniversidad Politécnica de Madrid

Acknowledge to: Miguel Esteban Gutiérrez, Raul García-Castro, Nandana Mihindukulasooriya

and the SEALS consortium11.04.23

1

http://www.seals-project.eu/

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam2

Ed Hovy Requirements

• A kind of framework that graphically depicts the whole scholarly environment/workflow - on this, a set of markers, like Post Its notes, that each indicate a technology available for use at the relevant point - also on this, another set of Post Its notes that each indicate some kind of bottleneck or gap

• A kind of system that helps me to reproduce experiments that appear in research papers and compare my technology with other technologies

11.04.23

2

Asun Gómez-Pérez Requirements

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam3

11.04.23

3

Scenario

Tool1

Tool4

Tool5

Tool1

Tool4

Tool5

Tool1Tool2Tool1Tool2

Tool3Tool3 Tool1Tool4Tool5

Tool1Tool4Tool5

Tool1Tool1

Tool2Tool6Tool2Tool6

Tool6Tool6 Tool6Tool7Tool6Tool7

Scholarly workflows

How do I know which tools are appropriate?: Reproduce Experiments (unbiased, repeatable, …)

ToolsTools

Test dataTest dataEvaluationsEvaluations

ResultsResults

My toolMy tool My resultsMy results

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam

Heterogeneity in the tools to be compared

• Data Sets– Levels of complexity– Scalability– Synthetic, hand-crafted

and real world data• Tools

– Execution environment– Functionalities– Criteria to evaluate:

scalability, interoperability, etc.

• Metrics– Precision/recall, information

losses of data, etc.• Evaluations

– Fully automatic– The user is in the loop– Hybrid approach– Extrinsic and intrisic

• Starting point– From scratch– Extend and adapt

preliminar approaches

11.04.23

4

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam

Test dataselection

Metricsselection

Execution requirements

selection

Evaluation services

implementation

Interpretationselection

Goalsselection

Tool featuresselection

Toolsselection

Evaluation workflows

implementation

Assumptionsselection

Criteriaselection

Dimensions

Evaluationdesign

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam

SEALS platform for technology evaluation

• SEALS Platform Features

6

• Open (everybody can use it)• Scalable (to users, data size)• Extensible (to more tests, different technology,

more measures)• Sustainable (beyond SEALS)• Independent (unbiased)• Repeatable (evaluations can be reproduced)

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam7

Test DataTest Data

EvaluationsEvaluations

ToolsTools

The SEALS platform

7

ResultsResults

11.04.23

Semantic Technology Recommendation

Framework

Results Comparisons

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam8

SEALS Service Manager

SEALS Service Manager

Runtime Evaluation

Service

Runtime Evaluation

Service

SEALS PortalSEALS Portal

Test Data Repository

Service

Test Data Repository

Service

Tools Repository

Service

Tools Repository

Service

Results Repository

Service

Results Repository

Service

Evaluation Descriptions

Repository Service

Evaluation Descriptions

Repository Service

Technology Providers

Evaluation Organisers

Technology Adopters

Software agents

SEALS Repositories

SEALS Logical Architecture

11.04.23

8

Virtualization:two execution environments

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam

Evaluation Execution Process at 10,000ft

9

ER

10.07.2012

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam

Evaluation services

10

Exploit resultsExploit results

11.04.23

ToolsTools

Test dataTest dataEvaluationsEvaluations

ResultsResults

Reproduce experiments

My toolMy tool My resultsMy results

My test dataMy test data My resultsMy resultsToolsToolsEvaluationsEvaluationsUse your dataset to evaluate others tool

ToolsTools Test dataTest data

My evaluationMy evaluation

My toolMy tool

My resultsMy results

My test dataMy test data

Define your own experiment

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam11

Achievements

11.04.23

Specification of the Platform Architecture

Workflows Implemented in BPEL

Test Data

Repository Service

Test Data

Repository Service

Tools Repository Service

Tools Repository Service

Results Repository Service

Results Repository Service

Evaluation Descriptions Repository

Service

Evaluation Descriptions Repository

Service

Implementation and Evaluation of Repositories

http://www.seals-project.eu/ontologies/

SEALS entities (dataset, tools, evaluations and results) are described using Ontologies

Semantic Technology Recommendation Framework

Results Comparisons

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam12

SEALS support to scholarly workflows

12

SEALS Service ManagerSEALS Service Manager

Runtime Evaluation Service

Runtime Evaluation Service

Test Data RepositoryTest Data Repository

Tools RepositoryTools Repository

Results RepositoryResults Repository

Evaluation Descriptions Repository

Evaluation Descriptions Repository

SEALS Repositories

Scholarly WorkflowRepository

Scholarly Portal

Tool1

Tool2Tool1

Tool2Tool3Tool3 Tool1

Tool4Tool5

Tool1

Tool4Tool5

Tool1Tool1

Tool2

Tool6Tool2

Tool6Tool6Tool6 Tool6

Tool7Tool6

Tool7Tool1

Tool4

Tool5

Tool1

Tool4

Tool5

Asun Gómez-Pérez. The SEALS Platform for Scholarly Research. Beyond the PDF2 Conference. 19-20th March 2013. Amsterdam

Thank you for your attention!