The State of NeXus

The State of NeXus

P.F. Peterson and F.A. Akeroydon behalf of the NeXus International

Advisory Committee

NeSSI: Collaborative Software and Other Mythical Creatures

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

Soft

NeS

S’94

Soft

NeS

S’95

Soft

NeS

S’96

v1.1

.0, v

1.2.

0, v

1.2.

1, v

1.3.

0

v1.3

.1, v

1.3.

2, v

1.3.

3

v2.0

.0

v2.1

.0

v3.0

.0

v4.0

.0

NIA

C Pa

sade

na, C

A

NIA

C Vi

llige

n, S

witz

erla

nd

NIA

C Sa

nta

Fe, N

M

NIA

C G

reno

ble,

Fra

nce

NIA

C Be

rkel

ey, C

A

NIA

C Be

rlin,

Ger

man

y

NIA

C Sy

dney

, Aus

tral

ia

NeSSI: Collaborative Software and Other Mythical Creatures

NIAC Members• Frederick Akeroyd, Rutherford Appleton Laboratory, UK (ISIS, Executive Secretary) • Stephen Cottrell, Rutherford Appleton Laboratory, UK (Muon Representative) • Matthias Drochner, Forschungzentrum Jülich, Germany • Emmanuel Farhi, Institut Laue Langevin • Andrew Götz, European Synchrotron Radiation Facility, France • Nick Hauser, Australian Nuclear Science and Technology Organisation, Australia

(Documentation Release Manager, netCDF contact) • Jens-Uwe Hoffmann, Helmholtz Zentrum Berlin (renamed HMI), Germany • Pete Jemian, Advanced Photon Source, USA • Paul Kienzle, DANSE, USA • Mark Könnecke, Paul Scherrer Institut, Switzerland (Technical Committee Chair) • Nick Maliszewskyj - NIST Center for Neutron Research, USA • Raymond Osborn, Argonne National Laboratory, USA • Peter Peterson, Spallation Neutron Source, USA (Chair) • Stephane Poirier, Synchrotron Soleil, France • Thomas Proffen, Los Alamos National Laboratory, USA • Jiro Suzuki, KEK, Japan

Principles of NeXus

Mission: NeXus enables comprehension of data amongst the neutron, x-ray, and muon science communities through a standard data format.

Reduced need for local expertise

• nexus@nexusformat.org• nexus-developers@nexusformat.org• http://www.nexusformat.org

• http://svn.nexusformat.org• http://trac.nexusformat.org

Simple Plotting

NXdataNXdata

Reduced need for conversion

Is it based on anything?

• Hierarchical Data Format (4 and 5)– NXopen(filename,NXACC_CREATE4,handle)– NXopen(filename,NXACC_CREATE5,handle)

• eXtensible Markup Language– NXopen(filename,NXACC_CREATEX,handle)

– NXopen(filename,NXACC_RDWR,handle)– NXopen(filename,NXACC_READ,handle)

Who is committed to NeXus

SNS at ORNLSNS at ORNLNCNR at NISTNCNR at NISTISIS at RAL (neutrons and muons)ISIS at RAL (neutrons and muons)DIAMOND at RALDIAMOND at RALBI at ANSTOBI at ANSTOFRM2 at TUMFRM2 at TUMKENS at KEKKENS at KEKJ-PARCJ-PARC

APS at ANLAPS at ANLHMI in BerlinHMI in BerlinMLNSC at LANLMLNSC at LANLSINQ at PSISINQ at PSIILL in GrenobleILL in GrenobleLLB at SaclayLLB at Saclay

Existing HDF4/5 tools

• HDFLook – HDF based plotter (???)• Java HDF Viewer (NCSA)• Scientific Data Browser – view data over the

web (NCSA)

• Any web browser• Eclipse with WTP plugin

Existing XML tools

NeXus Utilities• NXbrowse – CLI tool to interactively explore NeXus files• NXdir – CLI tool that lists contents of NeXus file and convert

selected portions to ASCII• NXextract• NXtoDTD – Creates a definition file from an existing NeXus file• NXtoXML – Writes an XML version of an existing NeXus file• NXtranslate – anything to NeXus converter NXvalid – GUI tool to

interactively explore, plot, and validate NeXus files• NXtree – Create a html based tree view of NeXus files• NXvalid – GUI tool to view and validate NeXus files• NXvalidate – command line tool for validating structure of

NeXus files

NXvalid

NXtranslate• Anything to NeXus converter

– Binary dump– FRM2– IPNS run– NeXus– Spec– XML

• Uses an XML-based translation file to determine where to get information from

• Additional information can be supplied in the translation file directly• CLI to encourage batch usage• Simple macro language to allow translating several files using one

translation file

Tools and Utilities• addit, subi – programs to add or subtract powder data and write files suitable for Rietveld programs (SINQ)• Amortool – simple CLI tool for Reflectometer data analysis (SINQ)• anatric, cami4pcd – four circle diffractometer data analysis (SINQ)• BerSANS – data analysis for SANS through adapter (SINQ)• DANSE – Data Analysis Architecture for Neutron Scattering Experiments (CalTech)• DAVE (future) - Data Analysis and Visualization Environment (NCNR)• FABLE (ESRF)• fit – home grown fitting program (SINQ)• GumTree – scientific workbench for instrument control (BI)• hdfb.sav and h5b.sav – browser for any 1D/2D/3D data saved in HDF (APS)• ISAW – Integrated Spectral Analysis Workbench (IPNS)• KUPLOT – General purpose plotting program (MLNSC)• LAMP – Large Array Manipulation Program (ILL)• MANTID – Manipulation Toolkit for ISIS Data (ISIS)• McStas – instrument modeling program (SINQ/ILL)• Nathan – TOF data analysis for FOCUS (IDL based) (SINQ)• NDS – NeXus data server, publish NeXus files on TCP/IP, read only (SINQ)• ninx – inx (ILL) adapted to read FOCUS TOF files (SINQ)• NNDB – Java program to browse NeXus files published through NDS (SINQ)• NXviewer – OpenDave based viewer (FRM2)• OpenGenie – General purpose neutron and muon scattering data reduction and analysis tool (ISIS)• Redas – data analysis for Reflectometer, based on Scilab (SINQ) • ReflPak – data analysis for Reflectometer (future) (NCNR)• Scilab – general matlab like data analysis tool - with a NeXus interface for that (SINQ)• SNS Portal (SNS)• UDA and WIMDA – General purpose muon scattering analysis packages (ISIS)

nexusformat.org

NeXus API v4.2

NeXusDefinitionLanguage

Validation of NeXus files using XMl Schema

Freddie AkeroydNIAC

Why Validation?

Standardised definitions make sharing files and analysis programs easier

However, developers need a way to confirm their files comply to the standard

NeXus Meta-DTD Format

Not really a true DTD, more a template XML file

However there are a few issues:Syntax is a little “cryptic” – required training

to readNot easily machine parsableValidated by humans

Base classes /dictionaries

Base classes /dictionaries

Application definitionsApplication definitions

DocumentationDocumentationValidation formValidation form

schemaschema

<definition name="NXfermi_chopper" type=“group” extends="NXchopper"> <field name="type" type="NX_CHAR">

<doc>fchopper type</doc> </field> <field name="radius" type="NX_LENGTH">

<doc>radius of chopper</doc> <attribute name=“colour” /></field> <group type="NXgeometry">

<doc>geometry of the fermi chopper</doc> </group> …

New Definition Format

NXDL post XSLT transform

NeXus Schema (XSD) File

Advantages of XML schemaHave built-in data types and allow new types

Support extension and restriction of typesMany tools exist for using schema to validate

XML filesTools exist for generating language bindings

and data reading libraries from schemaPossible to add constraints on values via

enumerations, patterns and rangesSupports keys and uniqueness constraints

How are definitions built? Each NX* base class is defined in a separate file

This separation will make life easier for future base class editors

Instrument definitions restrict NXentryType by marking required items as mandatory

Inheritance used to ensure consistency and avoid duplication: “object”, “component”, “chopper base”

Enumerations created for chopper types, units, shapes, etc.

Special “Types with units” defined, such as floatAngleDataType and floatLengthDataType

XML fileXML file

NXD Definition

file

NXD Definition

file

NeXus fileNeXus file

XML schema

XML schema

Validation (xmllint)

Validation (xmllint)

NXVALIDATE

Questions?Questions?

NeXus homepagehttp://www.nexusformat.org

Feature requestshttp://trac.nexusformat.org