Upload
makara
View
40
Download
0
Embed Size (px)
DESCRIPTION
ICAT Integration at DLS. Alun Ashton. What were the requirements?. Integrate with current business system Collect Data and Metadata relating to a proposal Security Long term storage of data (Multi?) Institutional repository Searchable Metadata - PowerPoint PPT Presentation
Citation preview
ICAT Integration at DLS.
Alun Ashton
What were the requirements?
• Integrate with current business system• Collect Data and Metadata relating to a proposal• Security• Long term storage of data• (Multi?) Institutional repository• Searchable Metadata• Potentially extendable with future e-science
infrastructure• Scalable
What were the restrictions?
• Only relies on Beamline Network.• Must not prevent scientist collecting data.• Data is collected by putting files onto a UNIX file system.• Use e-Science research output.
• Interest, vision, priorities and finances.
What was the solution? Diamond, Modified by e-Science
GDA
DArc andStoragedNexus
& Data
DUO DLS ICAT SRB
DataPortalDiamondProposal
Web pages
AtlasData Store
DUO Desk
Active Directory
People DB
IKittenData /
metadata
What was the solution? Diamond, Modified by e-Science
GDA
DArc and StoragedNexus
& Data
DUO DLS ICAT SRB
DataPortalDiamondProposal
Web pages
AtlasData Store
DUO Desk
Active Directory
People DB
IKittenData /
metadata
What goes into the ICAT/Ikitten
• Who• When• Where• ‘what’
– Abstract– Visit
• Triggers I think moves the data to the iKitten
DLS ICATDUO Desk
IKitten
What does the GDA read from the IKitten
• Metadata read only– Fedid – the GDA knows the users Fedid and asks the
DB for the current visit for that user, used for setting up data paths.
– Instrument, abstract
GDA
IKittenData /
metadata
Role Based access and application sharing.
• Definable access• Definable role• If this is not
right…
What does the GDA write out?
• GDA does not currently write a NeXus file,
– Detector files and anything…..
• Currently implemented only on MX beamlines
GDANexus& Data
‘GDA’ needs to initiate next step
• In the MX GDA whenever an image file is created it runs a script
• Bash script creates an XML file as input to DArc• The GDA can produce a script itself but no beamlines
currently do today… maybe next week?
GDANexus& Data
DArc
DArc
Watches a directory
Converts file to XML Ingest format
Initiates storagedCopies a filelistTo a location
Moves XML to A log
• Python Based EDNA • Server that checks <>5s• http://www.edna-site.org
Runs XMLingest
Success?
The XML file, minimum!<?xml version="1.0" ?><icat version="1.0 RC6"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="icatXSD.xsd">
<study> <investigation> <inv_number>MX307</inv_number> <visit_id>MX307-27</visit_id> <instrument>i04</instrument> <title>dont need it</title> <inv_type>experiment</inv_type> <dataset> <name>icr/fernando</name> <dataset_type>EXPERIMENT_RAW</dataset_type> <description>unknown</description> <datafile> <name>FG2_3_MS_3_154.img</name> <location>srb://srb-mcat-i18.esc.rl.ac.uk:5518/dls-2/i04/data/data/2009/mx307-27/icr/
fernando/FG2_3_MS_3_154.img</location> <description>unknown</description> <datafile_version>1.0</datafile_version> <datafile_create_time>2009-08-11T02:58:21</datafile_create_time> <datafile_modify_time>2009-08-11T02:58:21</datafile_modify_time> </datafile> </dataset>
What is our status?
• https://dataportal.diamond.ac.uk
New Eclipse Based GDA
Viewing and choosing the Metatdata
Relation to NeXus
By the by…https://ispyb.diamond.ac.uk
• Did try a view, now use a script
DLS ICATDUO Desk
ISPyB
IKitten
ISPyB and ICAT
• ICAT– Archive– Generic
• ISPyB– LIMS– Domain specific
How robust is the system?
GDA
DArc andStoragedNexus
& Data
DUO DLS ICAT SRB
DataPortalDiamondProposal
Web pages
AtlasData Store
DUO Desk
Active Directory
People DB
IKittenData /
metadata
How robust is the system?
GDA
DArc andstoragedNexus
& Data
DUO DLS ICAT SRB
DataPortalDiamondProposal
Web pages
AtlasData Store
DUO Desk
Active Directory
People DB
IKittenData /
metadata
Future Work
• Use it!!!
• Portal and API integration into in house tools• Sample information from user office (top level
information about the sample, and ISPyB (the instance of the sample).– Internal ICAT.
• Reduced data and richer metadata.
• Archive old data
Acknowledgements • Diamond Light Source: Tobias Richter, Stuart Campbell,
Karl Levik, Marc Basham, Karl Levik, Bill Pulford (and others in groups who have contributed at various times on various beamlines)
• STFC e-Science: Michael Gleaves, Roger Downing, Rik Tyer, Glen Drinkwater, Shoaib Sufi, Phil Couch, Kerstin Kleese Van Dam, Keir Hawker, Carmine Cioffe, Gordon Brown, Lisa Blanshard, Kevin O’Neill.
• STFC Daresbury: Steve Kinder and Karen Ackroyd
• But especially, Roger, Keir, Carmine, Michael, Mark and Tobias.
• BTW we have 64Tb of raw data already