LDAQ – the New Lujan Center Data Acquisition Application

Operated by Los Alamos National Security, LLC for NNSAU N C L A S S I F I E D

LDAQ – the New Lujan Center Data Acquisition Application

Frans Trouw, Gary Cooper and Paul Lewis

Lujan Center, Los Alamos National Laboratory


Outline

• Brief Overview of Lujan Center• Description of hardware• Existing data acquisition application (LOS)• Issues with LOS (reminder, see Paul’s talk)• Guiding principles for new application (LDAQ)• Overview of existing new LDAQ application• Experiences with LDAQ (good!)• Summary


Data Acquisition at the Lujan Center - hardware

• Detectors are all 3He gas tubes or area detectors

• Pre-amps at detectors generate a differential signal

• Custom “Time-of-Flight” boards take differential input and convert into events in a buffer

• VME (VXI) crate with single board computer collects events from TOF units


Overview of Hardware Configuration

Detector & Preamp

CP

UR

OC

TO

FT

OF

TO

FT

OF

TO

FT

OF

TO

F

TO

FT

OF

TO

F

CP

UR

OC

TO

FT

OF

TO

FT

OF

TO

FT

OF

TO

F

TO

FT

OF

TO

F

CP

UR

OC

TO

FT

OF

TO

FT

OF

TO

FT

OF

TO

F

TO

FT

OF

TO

F

T0 from accelerator

Private DAQNetwork

100 Mbit/s

DAQ Host PCWindows

ChopperEPICS IOC

VXI Crates

VME Crate ProcessorsVxWorks

"Public" LANLNetwork

100 Mbit/s LANSCE FacilityEPICS Gateway

Data ArchivePC Panels

CP

UR

OC

TO

FT

OF

TO

FT

OF

TO

FT

OF

TO

F

TO

FT

OF

TO

F

CP

UR

OC

TO

FT

OF

TO

FT

OF

TO

FT

OF

TO

F

TO

FT

OF

TO

F

CP

UR

OC

TO

FT

OF

TO

FT

OF

TO

FT

OF

TO

F

TO

FT

OF

TO

F

CP

UR

OC

TO

FT

OF

TO

FT

OF

TO

FT

OF

TO

F

TO

FT

OF

TO

F

CP

UR

OC

TO

FT

OF

TO

FT

OF

TO

FT

OF

TO

F

TO

FT

OF

TO

F

CP

UR

OC

TO

FT

OF

TO

FT

OF

TO

FT

OF

TO

F

TO

FT

OF

TO

F

CP

UR

OC

TO

FT

OF

TO

FT

OF

TO

FT

OF

TO

F

TO

FT

OF

TO

F

T0 from accelerator

Private DAQNetwork

100 Mbit/s

DAQ Host PCWindows XP

Remote access

ChopperEPICS IOC

VXI Crates

VME Crate ProcessorsVxWorks

"Public" LANLNetwork

100 Mbit/s LANSCE FacilityEPICS Gateway

Data ArchivePC Controls PC Sample

Environments


Crate contains SBC, ROC module, and 1-11 TOF units

TOF unitROC module

SBC

To Detectors


LOS Development Methodology

Domain:Domain:

class diagram,class diagram,

state charts,state charts,

action languageaction language

Domain:Domain:

class diagram,class diagram,

state charts,state charts,

action languageaction language

Domain X:Domain X: class diagram,class diagram, state charts,state charts, action languageaction language

““Model compiler”:Model compiler”: templatestemplates generation scripts (rules)generation scripts (rules)

Source code Source code generatorgenerator

BuildBuild

““Model compiler”:Model compiler”: run-time libraryrun-time library

Imported Imported softwaresoftware

Source fileSource fileSource fileSource file


Source fileSource file


Build scriptBuild script

ExecutableExecutableExecutableExecutable

ExecutableExecutable


xtUML is in principle attractive

• High level description generates consistent code and less errors

• Extending or changing model generates new code that takes into account interdependencies

• Less hand-coding => less manpower

• Quality is consistent and assured


xtUML has Significant Drawbacks

• Requires the desire to think at a very high level of abstraction – requires expert

• Not all of the code can be generated

• Highly dependent on message passing

• The community using xtUML is very small (< 100 at SMUG) - “early adopters”

• Only available for Windows

• Debugging next to impossible!


xtUML – communication risky?

• Message passing (MP) is critical to xtUML

• MP is done using Windows message queues or sockets

• A single communication failure kills the whole system

• VxWorks 5 network stack obsolete

• Windows network stack documentation limited (being generous here)


LOS move to Linux – MP problems!

• As Windows is so opaque and a single user environment, xtUML system libraries were ported to Linux

• Communication was implemented using a typical client server model (TCP/IP sockets)

• Linux and VxWorks 5 network stacks are incompatible


LDAQ “design principles”

• VXI crate LDAQ processes should be independent entities – fault tolerant

• No “host” computer controlling task

• Minimal code base

• Homogenous and state of the art operating system environment

• Open source preferred – transparency, quality & cost


What is LDAQ?

• Threw out all of LOS except hand coded pieces that do VME communication

• Moved VXI CPU’s from VxWorks to Linux

• “Host” computer is Linux

• Wrote a minimal application that integrates the VME calls – about 1 month of effort

• LDAQ runs independently as a user application on each VXI crate


LDAQ visual description

LDAQMySQL

(on host)ROC module

TOF units

20 HzLocal NetworkComputerLANL

Trigger

StartStop

Ready

Data


LDAQ benefits versus LOS

• Code can be changed and recompiled in seconds using Eclipse (> 1/2h for LOS)

• LDAQ failure on any VXI crate does not affect others – can be restarted

• Remote cross-debugging using gdb successful

• Linux is a true multi-user system (security)

• Open source – no licensing costs


Single Event and Histogram

• LDAQ does single event “out of the box” – temporary NFS limitations

• Single event -> large amount of data, use only when useful

• LOS histograms implemented in LDAQ

• “Live data” transfer call implemented

• Defined “time-focusing” histogram – e.g. live GSAS coming very soon!


Experience with LDAQ• Pharos has run LDAQ for most of this year –

only failures were power cuts!• HIPD and more recently HIPPO are successfully

using LDAQ with legacy scripting• LDAQ is a user application; changing SBC’s is

straightforward (e.g. PPC -> Intel) – cost, effort• Restart takes seconds• Infrequent/development VME bus “hangs” now

limiting reliability


Why is LDAQ successful?

• Each VXI crate is an independent agent – does not know about others

• No controlling task

• Minimal features, implement bells and whistles externally (e.g. run timing)

• Linux is robust and transparent

• Keep it simple!


LDAQ development - observations

• Exclusively using Linux - economical (up front and manpower) – cultural challenges

• Does require significant Linux expertise

• MySQL straightforward, robust & free

• Agile development, develop features as needed – requires IS collaboration

• Close collaboration between physical and computer scientists


LDAQ - future

• LDAQ development is essentially done!• Documentation• Needs external setup & control scripts• Scripting – leave to IS, provide, or port

existing control scripts?• Integration with slow controls – EPICS• Visualization toolkit• Real time analysis

Documents

LDAQ – the New Lujan Center Data Acquisition Application