Sep. 17, 2002BESIII Review Meeting BESIII DAQ System BESIII Review Meeting IHEP · Beijing · China Sep. 16 - 17, 2002

Sep. 17, 2002Sep. 17, 2002 BESIII Review MeetingBESIII Review Meeting

BESIII DAQ SystemBESIII DAQ System

BESIII Review MeetingBESIII Review Meeting

IHEP · Beijing · China Sep. 16 - 17, 2002IHEP · Beijing · China Sep. 16 - 17, 2002

Sep. 17, 2002 BESIII Review Meeting

BESIII DAQ Scheme

. . .

Branch 1

PowerPC VME Modul e

. . .

Tape

Branch M

.. .

VME Crate VME Crate

Tri gger System

.. .

VME Crate. . .

100MEthernet

1000MEthernet

Tri ggerSi gnal

Fi l e Server

Farm Node 1 Farm Node 2 Farm Node N

1000M Swi tch

Readout PC

100M Swi tch

Readout PC

100M Swi tch

Readout PC

100M Swi tch

Farm Supervi sor

Run Control EventDi spl ay

Hi stogramDi spl ay

Sl ow Control

100M Swi tch


Readout Crate System Controller – MVME2431-1Readout Crate System Controller – MVME2431-1

PowerPC 750™ 32-bit PowerPC 750™ 32-bit microprocessormicroprocessor

32MB of on-board ECC SDRAM32MB of on-board ECC SDRAM Up to 1MB capacity for on-board Up to 1MB capacity for on-board

firmware firmware 8MB on-board Flash memory 8MB on-board Flash memory One asynchronous serial debug One asynchronous serial debug

portport 10/100Mbps Ethernet interface10/100Mbps Ethernet interface 4-level requester, 7-level 4-level requester, 7-level

interrupter, and 7-level interrupt interrupter, and 7-level interrupt handler for VMEbushandler for VMEbus


3 Levels of Event Builder3 Levels of Event Builder

... ... ...

... ...

Slice

Fragment

Sub-event

Event

Event Bui l der

FEEs

ReadoutCrates

Readout PCs(Branch)

OnlineFarm

Hardware

Level-1 trigger

1 GbSwitch

100 MbSwitch


BESIII DAQ TasksBESIII DAQ Tasks Event Data Readout from FEEEvent Data Readout from FEE Event BuildingEvent Building Event FilteringEvent Filtering Event Recording to Persistent MediaEvent Recording to Persistent Media Run Control SystemRun Control System Monitoring Monitoring (event, histogram display ...)(event, histogram display ...) Message ReportingMessage Reporting


Data Rate EstimationData Rate Estimation Level-1 Trigger Rate: 4KHz Level-1 Trigger Rate: 4KHz (2KHz Good (2KHz Good

Events)Events)

Event Size: 12KBytesEvent Size: 12KBytes

Data Rate to be Processed by DAQ: Data Rate to be Processed by DAQ: 48MByte/sec48MByte/sec

To be Recorded on Disk Array/Tape: To be Recorded on Disk Array/Tape: 36MByte/sec36MByte/sec


TechnologiesTechnologiesMulti-level BufferingMulti-level BufferingSwitch NetworkSwitch NetworkParallel ComputingParallel ComputingEasily Upgrade and PortEasily Upgrade and PortEasily ScaleEasily ScaleReliable / Stable / Error RecoveryReliable / Stable / Error Recovery


Online Data FlowOnline Data FlowFarm Node 1

Readout PC

111

2

3 3 3

4 5

6

666

Farm Supervi sor1000M Swi tch

100M Swi tch

Run Control Event Di spl ay

Hi stogram Di spl ay

Fi l e Server

Tape

6

3

Network Connecti on

Control Command

Event Data

Farm Node 2 Farm Node N

A B C D E F G HSELECTED

ON-LINE


ON-LINE


PowerPC/VxWorks Based Test EnvironmentPowerPC/VxWorks Based Test Environment


Test Results – Programmed I/OTest Results – Programmed I/O

System Controller: MVME2431-1 (350 MHz System Controller: MVME2431-1 (350 MHz PowerPCPowerPC 750) 750)

Operating System: VxWorks 5.4Operating System: VxWorks 5.4

Writing from CPU DRAM into VMEbusWriting from CPU DRAM into VMEbus

32-bit transfers: 400 ns/32-bit write,32-bit transfers: 400 ns/32-bit write, 10MB/s 10MB/s

16-bit transfers: 400 ns/16-bit write, 16-bit transfers: 400 ns/16-bit write, 5MB/s5MB/s

Reading from VMEbus into CPU DRAMReading from VMEbus into CPU DRAM

32-bit transfers: 1,100 ns/32-bit read,32-bit transfers: 1,100 ns/32-bit read, 3.6MB/s 3.6MB/s

16-bit transfers: 1,100 ns/16-bit read,16-bit transfers: 1,100 ns/16-bit read, 1.8MB/s 1.8MB/s


Note:Note:

The above VME access speeds were The above VME access speeds were Programmed I/O speeds.Programmed I/O speeds.

The program was written in C language and The program was written in C language and

run in the context of the VxWorks shell.run in the context of the VxWorks shell.

The tests were conducted on an idle system The tests were conducted on an idle system

with no other competing processes nor any with no other competing processes nor any

concurrent activity on the VMEbus. concurrent activity on the VMEbus.


Question: Question:

Can we use Can we use Programmed ReadProgrammed Read to read to read out data from ADCs/TDCs modules?out data from ADCs/TDCs modules?


Estimation of One VME Readout Crate (1)Estimation of One VME Readout Crate (1)

Basic Assumptions:Basic Assumptions: 1 crate = 1 crate = 1616 modules modules 1 module = 1 module = 3232 channels channels Trigger rate: Trigger rate: 4000 Hz4000 Hz When one L1 trigger comes, When one L1 trigger comes, 10%10% channels have channels have

signals.signals. Each channel has its own Buffer containing Each channel has its own Buffer containing 16-bit 16-bit

datadata to be read. to be read. Each module has a Hit Map (Each module has a Hit Map (32-bit32-bit) ) Data readout is in Data readout is in A24/D16A24/D16 mode mode and 1.1 and 1.1

microsec/16bit readmicrosec/16bit read


Estimation of One VME Readout Crate (1) Estimation of One VME Readout Crate (1) (con’t)(con’t)

For each module:For each module:Data Ready Flag:Data Ready Flag: 1 word1 word

Trigger Number:Trigger Number: 1 word1 word

Hit Map:Hit Map: 2 word 2 word

Data:Data: 32 × 10% = 3.2 words32 × 10% = 3.2 words

For each crate:For each crate: 16 × ( 1 + 1 + 2 + 3.2 ) = 115.2 words16 × ( 1 + 1 + 2 + 3.2 ) = 115.2 words

When trigger rate is 4000 Hz, readout time will be:When trigger rate is 4000 Hz, readout time will be:

4000 × 115.2 bytes 4000 × 115.2 bytes ×× 1.1 microsec = 1.1 microsec = 0.51 sec0.51 sec


Estimation of One VME Readout Crate (2)Estimation of One VME Readout Crate (2)

Basic Assumptions:Basic Assumptions: 1 crate = 1 crate = 1616 modules modules 1 module = 1 module = 3232 channelschannels Trigger rate: Trigger rate: 4000 Hz4000 Hz When one L1 trigger comes, When one L1 trigger comes, 10%10% channels have channels have

signals.signals. Each module has a Each module has a Global BufferGlobal Buffer containing containing 16-16-

bit channel numberbit channel number and and 16-bit data16-bit data to be read. to be read. Data readout is in Data readout is in A24/D32A24/D32 mode mode and and 1.11.1

microsec/32bit readmicrosec/32bit read


Global Buffer Data FormatGlobal Buffer Data Format31 031 0

……

Event DataEvent DataChannel NumberChannel Number

…………

…………

TailTail

……


Trigger NumberTrigger Number

TailTail



Trigger NumberTrigger Number


Estimation of One VME Readout Crate (2)Estimation of One VME Readout Crate (2)(con’t)(con’t)

For each module:For each module:

Data Ready Flag:Data Ready Flag: 1 DWord1 DWord

Header:Header: 1 DWord 1 DWord

Data:Data: 32 × 10% = 3.2 32 × 10% = 3.2 DWordsDWords

Tail:Tail: 1 DWord1 DWord

For each crate:For each crate:

16 × ( 1 + 1 + 3.2 + 1 ) = 99.2 DWords16 × ( 1 + 1 + 3.2 + 1 ) = 99.2 DWords

When trigger rate is 4000 Hz, readout time will be:When trigger rate is 4000 Hz, readout time will be:

4000 × 99.2 bytes 4000 × 99.2 bytes ×× 1.1 microsec = 1.1 microsec = 0.44 sec0.44 sec


So, the answer is:So, the answer is:

The system controller does not have enough The system controller does not have enough time to read out data from ADCs/TDCs by time to read out data from ADCs/TDCs by using Programmed Read.using Programmed Read.

The way to get faster reads is to use the The way to get faster reads is to use the DMA engine resident on the Universe VME DMA engine resident on the Universe VME <=> PCI bridge chip. It is designed to copy <=> PCI bridge chip. It is designed to copy larger chunks of data from one bus to the larger chunks of data from one bus to the other. other.


MVME2431 DMA ControllerMVME2431 DMA Controller The Universe II has a DMA controller for high performance The Universe II has a DMA controller for high performance

data transfer between the PCI bus and VMEbus.data transfer between the PCI bus and VMEbus. Transfer sizeTransfer size

Any Number of bytes from Any Number of bytes from 1 byte to 16 Mbytes1 byte to 16 Mbytes.. Transfer Data WidthTransfer Data Width

D08 (EO)D08 (EO) D16D16 D16BLTD16BLT D32D32 D64D64 D32BLTD32BLT D64BLT (MBLT)D64BLT (MBLT)


Readout Mode: Global Buffer + DMAReadout Mode: Global Buffer + DMA

Block size must be counted Block size must be counted somewhere on board. somewhere on board.

To avoid too many DMA To avoid too many DMA operations (DMA Interrupts), operations (DMA Interrupts), it’s better for each ADC/TDC it’s better for each ADC/TDC board to board to buffer data of buffer data of multiple triggersmultiple triggers (e.g. 64, 128 (e.g. 64, 128 triggers), and then triggers), and then interruptinterrupt the system controller to start a the system controller to start a DMA operation.DMA operation.


DMA TransferDMA Transfer

Some typical values for DMA transfers:Some typical values for DMA transfers:

D32 Read:D32 Read: 20 -- 23 MB/s20 -- 23 MB/s

D32 Write:D32 Write: 22 -- 25 MB/s22 -- 25 MB/s

D64 Read:D64 Read: 40 -- 45 MB/s40 -- 45 MB/s

D64 Write:D64 Write: 45 -- 50 MB/s45 -- 50 MB/s

Note:Note: This is the raw speed of the hardware This is the raw speed of the hardware without any software overhead.without any software overhead.


Main Tasks of ReadoutMain Tasks of Readout

DMA transferDMA transfer CalculationCalculation Data unpacking & packingData unpacking & packing Network transmission.Network transmission.


R & D PrototypeR & D Prototype

Online Farm

Node 1 Node 2

ReadoutPCs

Farm Supervisor

100Mbps Switch100Mbps Switch

Single EventDisplay

File Server


ON-LINE


ON-LINE

Node 1 Node 2


File ServerFile Server

DELL PowerEdge 2500 Dual Intel Pentium III DELL PowerEdge 2500 Dual Intel Pentium III 1.4 GHz Processors with 512K Cache1.4 GHz Processors with 512K Cache

Integrated 10/100 Ethernet AdapterIntegrated 10/100 Ethernet Adapter 1 GBytes SDRAM1 GBytes SDRAM SCSI RAID ControllerSCSI RAID Controller Six 18GB SCSI Hard Drive, 10K RPMSix 18GB SCSI Hard Drive, 10K RPM Intel Pro 1000 Fiber Network Adapter Intel Pro 1000 Fiber Network Adapter Red Hat Linux 7.3 Kernel 2.4.18-3 smpRed Hat Linux 7.3 Kernel 2.4.18-3 smp


Online Farm & Readout PC NodesOnline Farm & Readout PC Nodes

DELL GX260 Intel Pentium 4 1.8 GHz ProcesDELL GX260 Intel Pentium 4 1.8 GHz Processor with 256K Cachesor with 256K Cache

Integrated 10/100/1000 Ethernet AdapterIntegrated 10/100/1000 Ethernet Adapter 512 MBytes SDRAM512 MBytes SDRAM 40GB EIDE 7200 RPM Hard Drive40GB EIDE 7200 RPM Hard Drive Red Hat Linux 7.3 Kernel 2.4.18-3Red Hat Linux 7.3 Kernel 2.4.18-3


Ethernet SwitchEthernet Switch

Catalyst 2948G-L3Catalyst 2948G-L3 48 Ports of 10/100-Mbps Fast Ethernet48 Ports of 10/100-Mbps Fast Ethernet 2 Ports of 1-Gbps Gigabit Ethernet2 Ports of 1-Gbps Gigabit Ethernet


Network Performance Network Performance (Preliminary)(Preliminary)

0

20

40

60

80

100

0 32 64 96 128

Bl ock si ze (KBytes)

Thro

ughp

ut (

Mbps

)

Block sizeBlock size ThroughputThroughput Block sizeBlock size ThroughputThroughput Block sizeBlock size ThroughputThroughput

(Bytes)(Bytes) (Mbps)(Mbps) (Bytes)(Bytes) (Mbps)(Mbps) (Bytes)(Bytes) (Mbps)(Mbps)

88 0.470.47 10241024 26.1526.15 131072131072 87.6987.69

1616 0.930.93 20482048 36.5136.51 262144262144 88.7788.77

3232 1.821.82 40964096 52.5852.58 524288524288 89.2689.26

6464 3.543.54 81928192 65.6065.60 10485761048576 89.5289.52

128128 6.546.54 1638416384 75.7675.76 20971522097152 89.6689.66

256256 11.8111.81 3276832768 81.9181.91 41943044194304 89.7289.72

512512 18.6118.61 6553665536 85.9685.96 83886088388608 89.7689.76


More Studies on PrototypeMore Studies on Prototype Performance StudiesPerformance Studies

Network performance in parallel modeNetwork performance in parallel mode Raid disk performanceRaid disk performance

write/read speed vs. config (raid0/raid5)write/read speed vs. config (raid0/raid5) Data flow control softwareData flow control software Online filter softwareOnline filter software Distributed histogram processingDistributed histogram processing Single event displaySingle event display


Thanks!Thanks!

Documents

Sep. 17, 2002BESIII Review Meeting BESIII DAQ System BESIII Review Meeting IHEP · Beijing · China Sep. 16 - 17, 2002