ICTP, 23-28 April 2007 CIMA in Australia Ian Atkinson [email protected] HPRC Manager, ITR School of Maths, Physics and IT James Cook University

ICTP, 23-28 April 2007QuickTime™ and a

TIFF (Uncompressed) decompressorare needed to see this picture.

CIMA in Australia

Ian Atkinson

[email protected]

HPRC Manager, ITRSchool of Maths, Physics and IT

James Cook University

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.



http://www.instrumentmiddleware.org/





ICTP, 23-28 April 2007

QuickTime™ and aTIFF (LZW) decompressor













DART: Dataset Acquisition, Acquisition & e-Research Technologies

• A DEST-SII funded project (http://www.dart.edu.au & http://plone.jcu.edu.au/dart) to develop tools to handle the data & information management needs of diverse research environments.

• DART is developing tools to handle typical research data & information management needs, such as:

1. collecting data 2. managing data 3. analyzing that data to produce useful information 4. managing that information 5. collaboration & annotation on the information 6. publishing information 7. searching on the information

• DART has built three demonstrators: • Climate Research • X-ray crystallography • Indigenous Digital History





DART - workflows






Instrument Taxonomy

Rick McMullen


Goal: Develop a Grid enabled and Web services based portal system for collaborative remote access to scientific instruments and their data.

Why?

Maximise resource utilization and return on investment/s Outreach, diffusion, training and education

Grid?Access to compute cycles for processing and modeling and flexible distributed storage management that may further facilitate remote access

Multi-Site CollaborationGoal for production level service: Significant collaboration between USyd, JCU, IU (and many others: UAdel, Monash, UQ, DART…)Deployment and user focus issues (this is hard)

Remote Instrumentation


Common Instrument Middleware Architecture (CIMA)

Offers a general and re-usable model for instrument access and management Offers a general and re-usable model for instrument access and management using Web/Grid services; adaptable to different instrument settingsusing Web/Grid services; adaptable to different instrument settings

Flexible and extensible with modular use of plug-ins, Flexible and extensible with modular use of plug-ins, Common programmable interface – reusable code baseCommon programmable interface – reusable code base

Publish-subscribe or registration modelPublish-subscribe or registration model

Basis for a Basis for a standardisedstandardised implementation/deployment system implementation/deployment system

Grid enablement of instruments & sensorsGrid enablement of instruments & sensors






• Integrate instruments and sensors (e.g. real-time data sources) into a grid environment with Web Services interfaces

• Abstract instrument capabilities and functions to reduce data acquisition and analysis applications dependence on specialized knowledge about particular instruments

• Move production of metadata as close to instruments as possible and facilitate the automatic production of metadata

• Develop a standard, reusable methodology for grid enabling instruments

Rick McMullen, Knowledge Acquisition and Projection Laboratory, Indiana University



are needed to see this picture.QuickTime™ and aTIFF (LZW) decompressor



CIMA Elements• Schema for instrument functionality in WSDL • Data model for representing instrument metrics and

calibration • A small, high performance Web Services stack

– Service implementation for accessing the instruments functionality

• Ability to dynamically insert new protocols into running instances

• OGSA compliant functions to register with a directory service, authenticate users, provide access control to instrument controls and data, and co- schedule the instrument into a Grid computing and storage context;




CIMA X-ray Portal

www.instrumentmiddleware.org.au


Indiana CIMA X-Ray system 4 Components

i. Instrument Representative, ii. My Manager, iii. Data Manager, iv. Portal Interface

● Instrument Rep, My Manager, Data Manager Service implemented in c++, gSoap, Perl

● Storage – ext3 file-system, MySQL database● Portal – jrs168 portlets, gridsphere

Issues – My Manager & Portal implementations domain specific

i.e. Crystallography vs oceanographic monitoring


Video feeds

CIMAInstrument

Representative

Data Manager

CIM

A

SO

AP

Inte

rfa

ce

GridSphere Portal

SRB data repository (data and metadata)HSM

Storage

SRB MCAT

Data Processing Applications, Annontation and visualisation

Instruments

Live data feeds

CIMA core code from IUDART/JCU/UQ/Usyd/MU Implementation

Workflow & comp pipelines. Automatic metadata generation.

PGL: SRB access& schemas

SOAP

DART Implementation


NFS and MySQL replaced with Storage Resource Broker (SRB)

Kepler workflow used for MyManager Ability to customise

data storage via a workflow system

Highly extensible, not restricted to SRB as repository

Data Management - workflow


Data Grid middlewarefrom SDSC (San Diego Supercomputing Centre)

•Uniform interface to heterogeneous resources over a network.•Logical View of data•User defined metadata structures•Works in conjunction with MCAT.

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

Storage Resource Broker (SRB)Storage Resource Broker (SRB)

http://www.sdsc.edu/srb

Data Management - datagrid


Use of Personal Grid Library (PGL) for SRB data manipulation / metadata display

Metadata schema definitions applied to experimental data - chitter chatter

Stored experimental data is able to be easily retrieved, secured and annotated

Visual diagnostics for sensors (e.g. Graphs, Visual Indicators)

Live CCD image analysis (experimental status)

DART - User Portal







ICTP, 23-28 April 2007A. Beitz, et al. Monash

ICTP, 23-28 April 2007J. Hunter, et al. UQ


Initial Instrument Representatives deployed at JCU, USyd, Monash & UQ

Goal Each site having their

own Data Manager and SRB storage facility which is federated across sites

Shibbolith based AAA and Virtual Organisations

Federate/Replicate data into aa national SRB store…

Data Federations with SRB


– Data federations (SRB)– Security and Rights Management

• Shibboleth

– Analysis tools (Portals/portlets)– Workflows

• Goal is an end-to-end workbench• Extensions to larger and smaller instruments

(e.g. sensors)

Future Developments


Acknowledgements:

JCU Sydney Uni Indiana Uni Adelaide Uni S. Ness D. du Boulay K. Chiu (SUNNY) A. WendelbornM. Morgan C. Chee P. CoddingtonF. Eilert D. Zhang

Department of Education Science and Training: Dataset Acquisition, Accessibility, and Annotation e-Research Technologies (DART) project

Australian Research Council: e-Research Seed Funding Programme and the Research Networks Programme (MMSN: Molecular and Materials Structure Network).


Documents

ICTP, 23-28 April 2007 CIMA in Australia Ian Atkinson [email protected] HPRC Manager, ITR School of Maths, Physics and IT James Cook University