7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
An Overview ofDOOCS
andDAQ
O. Hensler
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Olaf HenslerDESY – MCS
Working in the controls groupfor FLASH and XFEL
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
FLASH
Linac is 280 m long up to 1GeV energy 2.5 kA peak current 5 Kryo Modules with eight 9-cell cavities 4 Klystrons to drive Gun and Modules VUV FEL light down to 6nm wavelength Undulator length is almost 30 m shared between Users, XFEL and ILC
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
3.3 km
HERA
DESY
Hamburg City Centre (7 km)
European X-Ray Free Electron Laser (XFEL)
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Accelerator schematic layoutIn
ject
or13
0 M
eVB
C I
500
MeV
BC
II2
GeV
Mai
nLi
nac
10-2
0 G
eV
Und
ulat
ors
Pho
ton
Bea
mlin
es
Col
limat
ion
Dia
gnos
ticF
eedb
ack
Bea
m D
istr
ibut
ion
installed modules 101
installed cavities 808
acc. gradient 23.6 MV/m
installed klystrons 26
RF pulse length 1.4 ms
beam pulse length 0.65 ms
repetition rate 10 Hz
max. average beam power 600 kW
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
X-ray Free Electron Laser an European project at DESY Linac will 1500 m long up to 25 GeV energy ~100 Kryo Modules with
eight 9-cell cavities each 1 Klystrons to drive 4 Modules 1.3 GHz operation XFEL light down to 0.1 nm wavelength German government announced to spend
50% of the budget == ~350 Mio Euro start of civil engineering in 2008
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Some DOOCS features Distributed Object Oriented Control System Client / Server system with ONC-RPC Implemented mainly in C++ Local HDD for archiving and configuration Every server has its local .conf and .log file Hierarchical name space Name calls -> query all locations or properties Most source of a server is inside a central C++ library No central DataBase For Solaris, Linux and Win2000
http://doocs.desy.de
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
DOOCS features con,t A commen client API for
DOOCS ONC-RPC
TINE
EPICS
On top of this API
DOOCS Data Display DDD
Generic Applications and command line tools
MatLab
LabView
ROOT
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
DOOCS features con,t
Native Java API : jDOOCS on top of Java
jddd
Alarm & Info System
jDTool ( generic access )
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
DOOCS architecture• 3 layers• common APIs• modular design• multi protocol
– RPC– TINE– EPICS– Shared Memory
• Device level– ~ 200 server types– Local cfg, arch
• Middle layer– FSM, FB– DAQ
• User applications
– GUI, Web, etc.
http://doocs.desy.de
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Nomenclature
FACILITY == Accelerator
DEVICE == Type of a device
LOCATION == position inside the accelerator
PROPERTY == list of properties
Example:
TTF2.VAC/ION_PUMP/1ACC1.WG/STS.ONLINE
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
DOOCS implementations, tools• API in C++, Java, MATLAB, LABView• SolarisOs, Linux• Multi threaded• Long time archiving (>30000 channels)• Devices (servers > 300)
– synchronized (> 50 VME crates + ~ 60 ICCD & CCD cameras)– not synchronized (CAN, ProfiBus, SEDAC, GPIB, Ethernet etc.)
• ML servers– DAQ (~ 20 Feedback and Monitoring ML servers)– Alarm system– Web Log book
• Applications– DDD, jDDD, jDTool– MATLAB, LabView, ROOT– Electronic Log book
•
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
DOOCS implementations, toolsDOOCS Data Display (DDD)
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
DOOCS implementations, tools
Transition from DDD (X-Window and OpenLook) to jDDD (Java)
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
DOOCS implementations, tools jDDD
view layers
http://jddd.desy.de
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
DOOCS implementations, toolsjDTool
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Concept Overview
XML backup
Alarm and Info server
Device IO
RPC call for device info
XML viaRPC
DOOCS device server
Info
error
status
GUIXML based Alarm and Info Server Architecture
Req / resp.
Dev. Sev. Msg.Time
Dev. Sev. Msg.Time
Fac.Dev.
Loc.Prop.
JMS
JMSServer
XML device history
CentralDOMTree
XML
Collectthread
JNI
Event&session
Queue
ApacheTomcat
JNIServlets
1.2.
3.
jDOOCS(opt.)
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
The Java GUI
Device Tree
Online Ticker
History
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Some pictures : LLRF DSP
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Some pictures : LLRF ADC
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Some pictures : Diagnostic
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Some Pictures : CPU
500 Mhz SPARC CPU
512 MB RAM
SCSI HD Interface
no fan
PMC
1 VME Slot
Solaris8
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Some pictures : HardDisk
SCSI HardDisk
5V und 12V from VME Bus
2 VME Slots needed
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Fast ADC
Build at DESY-Zeuthen
8 Channels
14Bit ADC
10Mhz
parallel clocked
memory for 16 bunches
170 items at FLASH
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Some pictures : Industry Pack
IP-Stepper
IP-DIGI IO
IP-Timer
IP-ADC
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
DOOCS server design idea
DOOCS modulates a certain hardware into a C++ class
This means : the class describes the properties of one hardware device
Example : DOOCS writes software for one ion-pump
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
DOOCS server design idea
read
convert
Voltage
archive
read
convert
Pressure
archive
read
convert
Voltage
archive
read
convert
Pressure
archive
Powersupply
HV I
Powersupply
HV I
Ion pump at location A Ion pump at location B
...
Ethernet
HardwareDevice
DOOCSServer
To program
classes ofServer lib
Location
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
DOOCS server design idea
does all data taking, calculations, process logic and storing independent from the network, cause of local harddisk for
operating system server binaries configuration files archiving
At startup all objects, locations are created from the configuration file Two main classes
EqFct class => the container class for the location D_fct classes => types for the properties
Most functionality is inside a central server library easy to extend good debugged/tested
The server :
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
DOOCS server design idea
Ser
ver
Pro
gram
Ethernet
Archive Configuration
VME access
Hardware I/O
Enet, ProfiBus, CAN, Sedac
Overload
Specific
Ethernet layer TCP/IP
RPC/XDR (Remote ProcedureCalls,External Data Rep.)Server RPC Interface
Address and Data handling
Equipment Functions standard and
Data Functions (D_float, D_int, D_hist ...
DO
OC
S c
lass
esO
S TINE calls
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
DOOCS server design idea
_SVR 1BC1 2BC2 UND1
List of EqFct == locations
List
of D
_fct
==
pro
pert
ies
SVR.NAME
ERROR
Log
Rate
STS
NAME
ERROR
SIN
Value
STS
NAME
ERROR
SIN
Value
STS
NAME
ERROR
SIN
Value
STS
Standard Properties derivedfrom the base class EqFct
Properties added toEqFct<MY>
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
DOOCS server design idea
MemoryBuffer
.FILT
RingBuffer
.EGU
.DESC
D_hist class
Value
set
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Timing Sequence FLASH
} Events
Injector Laser
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Fast ADC
ADC
ADC
TIMER
SH
AR
ED
ME
MO
RYDMA
TIMER DRIVER
SCOPE_SERVER
BPM_SERVER
RF_SERVER
CH_DATA
DAQ
VM
E B
usVME Crate
Trigger
DMA_SERVER
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Shared Memory
FAST ADC CHANNELS
1 2 NBUF INFO NEXT ADC
CURRENT BUFFERSHM INFO NEXT BUF
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
DAQ integrationinterrupt_usr1( )TIMER
LOC THREADD_adcdaq
FILL_BUF()
DEVICE LOCDEVICE DATA
SENDER LOCEqSender
CHECK DAQ EVENT
SEND BUF
READY
BUF PAR
LOC THREADD_adcdaq
FILL_BUF()
DEVICE LOCDEVICE DATA
A6
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
ATCA
ATCA == Advanced Telecom Computer Architecture
probably new crate standard XFEL
serial, switched bus (GB Enet, PCI-x)
redundant, monitored 48V powersupply
standardised monitoring and management (IPMI)
complete redundant setup possible
no IO from industry yet
too expensive for now
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Hardware: xTCATiming receiver
board size
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Future Shelfs : ATCA
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
FLASH DAQ Design
BM
FCFC SC
FCDS FCML
OPERATOR GUI
DAQ Server
EVB
dCache
LOCAL GUI
REMOTE GUI
Storage
RC
DB
RC GUI
multicast
Fast data (every shot)Beam relevant measurements: ADCs (BPM, BLM, TOR, etc) CAMERAs
DOOCS(TINE)
Slow data (max 1Hz) Data from slow ADCs(MAG, V, etc.) DOOCS channels (Masks, params, etc.)
45 days of linac data+ GMD , Exp Data
LINAC
FAST
ADC
IMAGE SLOW
ADC
DISK
DCCP DISKROOT
data streams
OracleXML files
48 hours of linac data
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
FLASH DAQ Design
DISK
dCache
Storage
BM
FC SC
DS FCML
OPERATOR GUI
DAQ Server
EVB
LOCALE GUI
REMOTE GUI
RC SA
RC SA GUI
BM
FC SC
DS ML
OPERATOR GUIEVB
LOCALE GUI
REMOTE GUI
RC
DB
RC GUI
DAQ for LINACDAQ for experiments
FAST
ADC
IMAGE SLOW
ADC
LINAC
FAST
ADC
IMAGE SLOWADC
Experiment
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
FLASH DAQ Design
8 double core SPARC CPUs32 GB common memory6 (3used by DAQ) x 1Gbit Ethernet1.7 TB local storagefully redundant fan, PS, …
• 2 dual-core AMD Opteron processors 2.6 GHz• 16 GB common memory• 24 TB local storage; 48 SATA HDD • 4 x 1Gbit Ethernet• 1 GBps from disks to network • 2 GBps from disk to memory
DAQ server hardware
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Experiments at FLASH
Hardware (custom)
C1class
C2class
C3class
C4class
C5class
C6class
Basic Camera Class
USB FireWire Ethernet FrameGrabber
DOOCSCamera
Server
DAQ
Andor PCO
Acqiris DC282 cPCI High-Speed 10-bit PXI/CompactPCI 4-ch, 2-8 GS/s,256-1024 kpt, 32-128 Mptor 256-1024 Mpt
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Experiments at FLASH
Control, monitoring Beam diagnostics panels
DDD panels MATLAB panels
Experiment panels DDD panels
7. April 2008 UCL 2008 London Olaf Hensler DESY-MCS
XFELThe EuropeanX-Ray Laser Project
X-Ray Free-Electron Laser
Hints for a hardware design
try to find a one to one relation between hardware and software
try to define an independent device
allow to read back all register, avoid write only