February 12, 2011

NEMO All-Hands Meeting:Database and Portal

Kurt Mueller and Jason Sydes

http://nemo.nic.uoregon.edu

Agenda

• FTP site / file database overview

• NEMO portal– Present– Future

Database – FTP site

• Place to store files– Raw waveforms (input files)– Intermediate analysis results– Parameter files– Final output files

Organization

• Organize files by lab, experiment, file type:

LAB DIRECTORYEXPERIMENT DIRECTORY

INPUT_DATA DIRECTORYANALYSIS_1 DIRECTORYANALYSIS_2 DIRECTORY

Example

JTC (lab-level folder)JTC-PMN1 (experiment-level folder)

JTC-PMN1_Input_Data (subdir)'JTC-PMN1.raw' (input data

file)JTC-PMN1_Decomposition_2011-01-15 (subdir)

'JTC-PMN1_tPCA.m' (metascript file)'JTC-PMN1_DecompObj-20110115.mat' (MATLAB

output file)'JTC-PMN1_tPCA.raw' (simple binary output

data file)

Uploading files – ftp client

Data in FTP is both managed and unmanaged

• Mixture of two different areas – Unmanaged– Portal managed

• Possible area for documentation?• What role will FTP server will serve in future?• Can browse and download portal data via FTP– Readonly!

Data in portal accessible via FTP

Graphical ftp clients

• Mac OS X + Windows + Linux– Filezilla - http://filezilla-project.org/

• Mac OS X + Windows– Cyberduck – free - http://cyberduck.ch/

• Mac OS X– Transmit - $34 - http://www.panic.com/transmit/

Goals of portal

1. Online record of experiment metadata2. Repository of experiment data files3. Tight coupling to NEMO ontology

4. Provenance of all data5. NEMO analysis pipeline execution 6. Querying on metadata

now

in progre

ss

Record experiment metadata

ExperimentConditions Sessions

Publications Subject groups Data files

Task

Stimuli

Responses

Quick Sampling of Portal

Portal database tables

Repository of experiment data files

• New upload widget provides progress feedback (standard http upload does not)

OpenID access to NEMO resources

• SSO: Single Sign On• Login once, get access to– NEMO portal– NEMO wiki– NEMO sourceforge page

• Learn about it: http://openid.net/• How to create a new SSO account:http://nemo.nic.uoregon.edu/wiki/New_Nemo_User

Role-based access control (RBAC)

Database coupled to OntologyExample Table: Stimulus

Database coupled to Ontology Example “Ontology”: Stimulus

Database coupled to Ontology

Database coupled to OntologyExample mapping: Stimulus

Database coupled to Ontology Example term: stimulus_intensity

Database coupled to OntologyExample tooltip

Database coupled to OntologyQuick syncing

Open Provenance Model(http://openprovenance.org/)

Open Provenance Model– subset used –

Open Provenance ModelExample

Open Provenance ModelSimplified Example

Workflow – MATLAB in portal

• Current workflow:1. Conduct experiment, producing raw waveforms2. Process data with NEMO Toolkit scripts, locally3. Upload raw and processed data to ftp or portal4. Create an experiment in the portal, associate

data with experiment

Worfklow – MATLAB in portal

• Future workflow (in design)1. Conduct experiment, producing raw waveforms2. Create an experiment in the portal3. Upload raw data through portal4. Execute MATLAB pipeline through portal• Automatic storage of output data• Automatic provenance tracking• Single place for MATLAB scripts to live• Run on big iron

Questions?

Thank you!