RodModule Class Library

Tom Meyer [email protected] 25 September 2002

RodModule Class LibraryRodModule Class Library

Tom Meyer

Iowa State University

[email protected]


ContextContext

RodModule is the class library providing driver routines for the ROD VME module.

ConsumerProcess 2

ConsumerProcess 2

Other Hardware

AncillaryManagers

ROD PrimList Managers


ROD Module

VME Interface

ROD Module

VME Interface

ActionList Handler

CommandPackage

ROD CrateManagers

(one per crate)

CrateManager

ROD Module

VME Interface

TIM Module

VME Interface

Other VME

VME Interface

Other Hardware

Run Control

AncillaryManagers

Data Manager

OtherConsumerProcesses

Real TimeMonitorProcess

User

RCC Process

SCT/PixelHost Process

ROD DAQ Overview :Co n tro l an d Data F lo w

State Trans itionsand res pons es

Commandsand res pons es

Pr imitiv es Replies

Selec tedData ty pes


Ev ent data (a ll ty pes )Pr imLis t Reply L is t

Pr imLis t Reply L is t/Ev ent data

Commands

Detector ConfigData Bases

DetectorControl (DDC)

Other Services:GUI (IUI)Messaging (MRS)Information D istributor (IS)E lectronic Logbook

Version 0.5T. Meyer, et al.

15 Feb 02

Key:ORANGE: DAQ-1GREEN: HOST PROCESSBLUE: RCC PROCESSRED: CORBA (OR TCP/IP)YELLOW: CONSUMERBLACK: OTHER

.

.


Present StatusPresent Status ROD has been working in Ames since 1 August. Debugging has covered most of the basic functionality.

Create a primitive and add it to a primitive list Send a primitive list and retrieve the reply buffer if it exists Read an unsolicited text buffer

Tools: g++ 2.96, gdb 5.0rh-5 with Insight GUI 5.1.1 (from Red Hat web site) Some problems have been uncovered

Writing to HPID with auto-increment of HPIA does not increment HPIA. It works fine for reading with auto-increment. Problem exists even with the RCC_Rel_2 vme_test utility. We suspect a hardware problem specific to this board. A work-around exists, but is inefficient.

Block reads and writes do not return correct values and block writes do not increment HPIA. A work-around is to do single word reads. Less efficient, but works.

Program crashes when I delete the RCCVmeInterface object. Since this is the last statement of the program, it is easy to get around by aborting via the debugger just before this is executed. It needs to be understood, though.


Directory Structure on SBC (rcc0)Directory Structure on SBC (rcc0)

/RodCrate RodModule.h, -.cxx RodPrimitive.h, -.cxx RodPrimList.h, -.cxx RodOutList.h, -.cxx RodRegisterBits.h RodDspAddresses.h RodVmeAddresses.h Various RodModule test

programs

/VmeInterface RCCVmeInterface.h, -.cxx VmeModule.h, -.cxx VmePort.h libRCCVmeInterface.a

/CommonWithDspprocessor.hsmSendTxtBuff.htxtBuffer.hcomRegDfns.hmemoryPartitions.hprimParams.hsctStructure.hutilities.hvmeAddressMap.h

/CommonForHostscommandListDefinitions.herrorCodes.hflashUtility.hhostDefinitions.hprimListTableDefinitions.h


Methods Inherited from VmeModuleMethods Inherited from VmeModule

initialize() Reset the ROD (bit 7 of FPGA Control Register 2) Read ROD serial number Initialize Master HPIC register and read it back to verify Wait for MasterDSP init Read reply buffer addresses Initialize state variables

? reset() Reset the ROD (bit 7 of FPGA Control Register 2) Clear IDRAM Reinitialize state variables

? status() Under development Creates a RodStatus object For now just print to standard output What information do we want?, Levels of info? How to display it?

Key: = well tested; ? = untested; ! = known problems


VME AddressesVME Addresses

DSP Host Port Interface (HPI) Command Register (HPIC): Sets half-word order. Set once at start of

program and leave it alone. (0x00000000/VME) Address Register (HPIA): Address in DSP space to be accessed. Data Registers: Data port (0x02000000/VME)

o HPID_AUTO: Auto increments HPIA after access (0x04000000/VME)

o HPID_NOAUTO: Leaves HPIA unchanged (0x06000000/VME) FPGA Status and Control Registers

They access the FPGAs directly for configuration, reset, status, halt, etc.o FPGA_CONTROL_REG_REL_ADDR[8] (0x00C00000/VME)o FPGA_STATUS_REG_REL_ADDR[8] (0x00C00024/VME)

R odM odule

P rog R esetM G R32/16

M asterD S P

R O D32 b its

32 b its 16 b its

16 b its

HPI


DSP AddressesDSP Addresses Primitive buffers

PRIM_BUFF_BASE (0x02040000/MDSP) PRIM_BUFF_SIZE (0x28000)

Reply buffers REPLY_BUFF_BASE (0x02068000/MDSP) REPLY_BUFF_SIZE (0x28000)

Text Buffers TEXT_BUFF_SIZE (0x8000) ERR_BUFF_BASE (0x02020000/MDSP) INFO_BUFF_BASE (0x02028000/MDSP) DIAG_BUFF_BASE (0x02030000/MDSP) XFER_BUFF_BASE (0x02038000/MDSP)

ROD Status Registers STATUS_REG[3] (0x80000000/MDSP)

VME Command Registers COMMAND_REG[2] (0x8000000C/MDSP)


ConstructorConstructor

RodModule( unsigned long baseAddr, unsigned long mapSize, VmeInterface & ourInterface, long slot, long numSlaves);

There are no copy or assignment operators. The constructor creates a VmePort object and registers it with its

VmeInterface object.


Shared Header FilesShared Header Files

The C-code DSP programs and the TestStand program use shared header files to specify addresses and bit assignments. We would like to use these same headers for C++.

We want to specify these details in only one place and propagate the information.

In C, these are set with #define statements. C++ can use C headers, but it is more in keeping with C++ principles

to declare the constants as const unsigned long variables. This lets the compiler catch any attempts to change them, and it does not expose the code to global text substitutions from #define statements.

I have wrapped the C headers with C++ header files that expose only data of interest to RodModule and use const unsigned long values. This also isolates C++ code from changes in C headers, and hides C-only parts of the header.


RodModule: Data MembersRodModule: Data Members

Board Params: long m_slot; //! Slot number in VME crate unsigned long m_serialNumber; //! Board serial number VmePort* m_myVmePort; //! VME Port handle long m_numSlaves; //! Number of slave DSPs on this ROD long m_finBufferSize; //! Size of file input buffer, in bytes. Default=4096 ?std::string m_masterImageName; //! string holding name of master image file ?std::string m_slaveImageName[4]; //! string array holding names of slave image filesRegister copies: unsigned long m_vmeCommandReg[2]; //! Cached copy of VME command registers unsigned long m_rodStatusReg[3]; //! Cached copy of ROD Status registersPrimitive handling: RodOutList* m_myOutList; //! A pointer to the array holding a reply buffer PrimState m_myPrimState; //! State variable for sending primitive listsText buffer handling: TextBuffState m_myTextState; //! State variable for reading text buffers TEXT_BUFFER_TYPE m_textType; //! Text buffer type struct TXTBUFFER m_txtBuffer; //! A struct holding text buffer info from txtbuffer.h char m_textData[TEXT_BUFF_SIZE]; //! A local buffer to hold the text data unsigned long* m_textBuff[N_TXT_BUFFS]; //! Array of pointers to the text buffers in MDSP space


RodModule: HPI Access MethodsRodModule: HPI Access Methods

//! Load a 32-bit value into an HPI register.

void hpiLoad(const unsigned long hpiReg, const unsigned long hpiValue);

//! Fetch a 32-bit value from an HPI register.

unsigned long hpiFetch(const unsigned long hpiReg);


RodModule: DSP-space Access MethodsRodModule: DSP-space Access Methods

//! Read to and write from Master DSP address space

unsigned long mdspSingleRead(unsigned long address);

void mdspSingleWrite(unsigned long address, unsigned long value);

! void mdspBlockRead(unsigned long address, unsigned long buffer[], long count);

! void mdspBlockWrite(unsigned long address, unsigned long buffer[], long count); Note: multi-block transfers are not yet tested.

//! Read to and write from a slave DSP memory address

? Unsigned long slaveSingleRead(long slaveNum, unsigned long dspAddr);

? void slaveSingleWrite(long slaveNum, unsigned long address, unsigned long value);

? void slaveBlockRead(long slaveNum, unsigned long address, unsigned long buffer[], long count);

? void slaveBlockWrite(long slaveNum, unsigned long address, unsigned long buffer[], long count);


RodModule: DSP Reset MethodsRodModule: DSP Reset Methods

//! Reset the Master DSP via FPGA Control Reg 2

void resetMasterDsp(); //! Reset a Slave DSP via FPGA Control Reg 2

? void resetSlaveDsp(long slaveNumber);

//! Reset all DSPs via FPGA Control Reg 2

? void resetAllDsps();


RodModule: PrimitivesRodModule: Primitives

//! Send a primitive list

void sendPrimList(RodPrimList *list);

//! State machine to handle primlist dialog with ROD

PrimState& primHandler();

//! Delete reply list retrieved by primHandler

void deleteOutList();


RodModule: Text BuffersRodModule: Text Buffers

//! Get a text buffer

void getTextBuffer(char * buffer, long length);

//! State machine to handle text dialog with ROD

TextBuffState textHandler();

Note: reading a wrapped-around buffer not yet tested

//! Clear the text buffer when done

void clearTextBuffer();


RodModule: UtilitiesRodModule: Utilities

//! Calculate a checksum (bit-wise XOR)

long checkSum(const unsigned long *sourceArray, const long wordCount);

//! Go to sleep for a specified number of milliseconds

void sleep(const double milliSecs);

//! Reverse the endian-ness of a data word if a flag is set

unsigned long endianReverse32(const unsigned long inValue);

These methods may be moved to a RodUtilities class in the future.


RodPrimitive ClassRodPrimitive Class

A simple class to hold Primitive objects Data members:

long m_length long m_index long m_id long m_version long *m_body

Methods: Only the usual data accessor methods (“get” and “put”).

Prim Version B

Prim Version A

List Version

Body A

PrimList Format

Version 2.518 Sep 2002

List Length

List Index

Checksum

Prim Length A

NumPrims

Prim Length B

Body B

Prim Index A

Prim Id A

Prim Id B

Prim Index B

List Length


RodPrimList ClassRodPrimList Class

Derived from the C++ Standard Template Library (STL) list class. This gives us almost all the functionality we need for free.

Data members: unsigned long m_index Unsigned long m_version unsigned long *m_buffer unsigned long m_listLength

Methods (other than data accessors) unsigned long numWords() unsigned long checkSum() void bufferBuild() void clear()

Prim Version B

Prim Version A

List Version

Body A

PrimList Format

Version 2.518 Sep 2002

List Length

List Index

Checksum

Prim Length A

NumPrims

Prim Length B

Body B

Prim Index A

Prim Id A

Prim Id B

Prim Index B

List Length


RodOutList ClassRodOutList Class

Contains reply lists generated by the MDSP in response to an associated primitive list.

Structure is the same as a PrimList Data members:

long m_length unsigned long* m_body

Methods: unsigned long* getBody() long getLength()

RodModule makes no attempt to understand the reply list, it merely passes it to a user thread for further handling.


Main Programs for TestingMain Programs for Testing

EchoTest.cc The first program I wrote sends a simple “ECHO” primitive, retrieves the reply, and prints it on the console. It also detects and prints messages from text buffers (mainly error).

Useful for debugging RodModule code and as a model for later programs.

In real RODDAQ, this functionality will be spread between threads and processes on the host and RCC machines.

Other programs are (or will be) MdspDump.cxx - Dumps values from a range of DSP addresses to

stout. Echo2Test.cxx – Test primlist with multiple echo primitives ? LedTest.cxx – Flashes a front panel LED via slave DSP ? TextTest.cxx – Fill a text buffer via a primitive …


Sequence DiagramSequence Diagram

CommandThread

BkgndThread

Primitive PrimList RodModule

UserCommand

Create Primitive

Add Primitive

Send PrimList

Iterate

write HPIC and HPIA

send inList

primHandler()

read dspAck

if (dspAck=0 and LOADED):set inListRdy=1

STATE=IDLE

setState(LOADED)

if (dspAck=1 and EXECUTING):read outListRdy

Return(LOADED)

Clear PrimList

Delete PrimitiveIterate

setState(EXECUTING)

Primitive List:User Command to ROD

Version 1.7T. Meyer4 Sep 2002

OutList

return(EXECUTING)

primHandler()

if (outListRdy=1):fetch outList

setState(WAITING)

if (outListRdy=1)construct OutList

return(WAITING)

outListRdy

processoutlist

if (Outlist) delete OutListdelete OutList

Reply to user

1

2

3

4a

4b

5

6

7

8

9

10

ROD

setState(IDLE)

getOutList

return(IDLE)

read dspAck

ReplyThread


Text BuffersText Buffers Text buffers are more useful for debugging than for data taking. If a DSP CPU crashes, we cannot execute primitives. But the on-board

extended memory interface (EMIF) circuitry still lets us read the SDRAM contents.

Text Buffers are blocks of SDRAM memory to which we can write ASCII records with traceback information, or simple diagnostic info for later analysis. (Recall debugging FORTRAN with print statements.)

Even if DSP is fine, it may be useful to pass ASCII data into a message stream (MRS) in ATLAS.

There are four text buffers defined for the Master DSP Err: for reporting fatal software error tracebacks Info: for non-priority information messages Diag: for diagnostic messages, “print” to it like we used to print out

FORTRAN debug messages. Xfer: A temporary MDSP buffer to relay text buffers from the Slave DSPs


Text Buffer HandlingText Buffer Handling

User TextThread

BkgndThread

RodModule

read xxxBufferNotEmpty

set xxxBufferReadRequest

textHandler()

read buffer header

read buffer contents

STATE=IDLE

setState(RQ_SET)

clear xxxBufferReadRequest

Return(RQ_SET)

clearTxtBuffer()

Iterate

setState(READOUT)

Text Buffers:User Thread to ROD

Version 1.3T. Meyer5 Jun 2002

return(READOUT)

setState(IDLE)

bufferReady

processbuffer

1

2

3

4

5

6

7

8

9

ROD

getTxtBuffer(buffer, length, type))

textHandler()

read xxxBufferNotEmpty

"1"

"0"

buffer


Short- and Medium-Term FutureShort- and Medium-Term Future Finish RodModule to first order (3 weeks)

Test slave routines (LED primitives, Memory read/write, DSP reset) Test text-generator primitive, use to test wrap-around readout Write prototype status() – in progress Fix block transfer and HPID++ problems Test multi-block transfers

Implement threaded version (3 weeks) Command thread Reply handling thread Error handling thread Background thread

Drive threads from DAQ-1(2 weeks) Send results to analysis routine(s) (worked on in parallel to above – who?)

Define interface for data Define transport method (IPC at first, CORBA later) Write analysis program (root? PAW? Other?)

Pixel calibration program Hard-wired commands and configuration Need ROD+BOC+TIM+Module => do at LBNL I propose a one or two week visit to LBNL of relevant parties in late November to mid

January time frame.


Phase 1: Pixel Calibration ProgramPhase 1: Pixel Calibration Program

1 ROD (=>1 crate) Posix threads DAQ-1 Hard-wired, or very simple,

configuration No data base Simple displays Analysis on host computer (no

CORBA)

P hase O ne

D AQ -1R un C ontro l

C om m andLayer

R O D M oduleD river

C ++ VM E Interface

C VM E Interface

U ser

VM E M odules

VM E calls

R O D Prim itives R eplies

S tateT ransitions R eplies

Analys is T askand D isp lay

D ata U ser


A Possible Path to Phase 1A Possible Path to Phase 1

Phase 1 “RODfest”@ LBNL

RodModule Debugged

DAQ-1

pThreads

Analysis/archiving

“reply” thread/IPC

ROD+BOC+TIM+Module

Simple Config


Longer-term FutureLonger-term Future

Multi-Rod and multi-crate support

Configure dynamically from data base

Dynamic control of operations CORBA for command

distribution and event publishing

Full implementation of plan from February workshop

ConsumerProcess 2

ConsumerProcess 2

Other Hardware

AncillaryManagers



ROD Module

VME Interface

ROD Module

VME Interface

ActionList Handler

CommandPackage

ROD CrateManagers

(one per crate)

CrateManager

ROD Module

VME Interface

TIM Module

VME Interface

Other VME

VME Interface

Other Hardware

Run Control

AncillaryManagers

Data Manager

OtherConsumerProcesses

Real TimeMonitorProcess

User

RCC Process

SCT/PixelHost Process

ROD DAQ Overview :Co n tro l an d Data F lo w

State Trans itionsand res pons es

Commandsand res pons es

Pr imitiv es Replies



Ev ent data (a ll ty pes )Pr imLis t Reply L is t

Pr imLis t Reply L is t/Ev ent data

Commands

Detector ConfigData Bases

DetectorControl (DDC)

Other Services:GUI (IUI)Messaging (MRS)Information D istributor (IS)E lectronic Logbook

Version 0.5T. Meyer, et al.

15 Feb 02

Key:ORANGE: DAQ-1GREEN: HOST PROCESSBLUE: RCC PROCESSRED: CORBA (OR TCP/IP)YELLOW: CONSUMERBLACK: OTHER

.

.

Documents

RodModule Class Library