Juno proposal for PMT readout – GCU - INFN Sezione di Padova · INFN and University of Padova -...

INFNandUniversityofPadova-GCUProposal

1

JunoproposalforPMTreadout–GCU

M.Bellato1,R.Brugnera2,F.DalCorso1,S.Dusini1,A.Garfagnini2,R.Isocrate1,I.Lippi1,G.Meng1,D.Pedretti3

1INFN–SezionediPadova2Universita’diPadova–DipartimentodiFisicaG.Galilei3INFN–Lab.NazionalidiLegnaro

1.Introduction

Thescopeofthisdocumentistodescribeapossiblearchitectureforphoto-multiplier(PMT)readout, following the indication of a baseline structure agreed at the Paduameeting ofOctober2015.Essentially,thebaselinestructurestatesthateachPMTshouldembedboththeHighVoltageand the Readout electronics in a standalone manner, with sufficient I/O for PMT signalprocessing,highvoltageinterface,triggeringsupportanddatareadout.2.Architecture

TheJunocollaborationrecommendsabasicstructureinwhichPMTreadout,triggerprimitivegeneration,fragmentbuffering,selectivedatareadoutandHVinterfacetakesplaceonaperPMT basis. One possible solution is to embed these complex tasks on the PMT itself, byaugmenting the PMT physical volume with a water-tight box housing HV and digitalelectronicsandcommunicatingwiththeexternalworldbymeansofcoppercableswithanestimatedlengthof~100meters.Giventhehigh(~20.000)numberofPMTsinvolved,theoverallcostofthedigitalboardandthenumberofcablesaretobeassumedasparameterstobeminimized.Theconceptualschemefortriggerandreadoutelectronicscanbesummarizedasinfigure1.EachPMTsignalgetscontinuouslydigitizedbyacustommadeASICandbufferedinalocalmemorywhilelookingontheflyformeaningfuldatawiththresholdcomparison:insuchacaseatriggerrequestisgeneratedandroutedtoaGlobalTriggerProcessorviaadedicatedlink.

INFNandUniversityofPadova-GCUProposal

2

Figure1DataarestoredwaitingforatriggeracceptthatmaycomeviaanEthernetinterface:insuchacase,aprocessingelementwillinspectthedatastoragelookingforaneventfragment(orabunchofevent fragments)matching the requestedone(s) inapreset timewindow;uponpositivematch,thecurrenttimestampandacentrallyassignedeventtagareassociatedwiththematchedfragmentandsenttotheDataAcquisitionSystemthroughtheEthernetlink.This conceptual scheme is widely deployed in most modern physics experiments and iseffectiveinreducingthebandwidthrequirementsofthedatalink.Theworstcasescenariointermsofbandwidthrequirementcomesfromtriggersduetodarkcurrent,whoseratemayreach ~50KHz; assuming 30ns of sampling period at 1 Gsample/s with 16 bit words, themaximumdatarateisintheorderof24Mb/s,wellintherangeofFastEthernet.HencethecostofthemediumcouldbegrantedbyaninexpensiveCAT5eEthernetcable.AcandidateschemeforlocalreadoutisillustratedinFig.2.

INFNandUniversityofPadova-GCUProposal

3

Figure2.Accordingtofig.2thecorecomponentsoftheGCUare:

• ADU–acustombuiltASICwith1Gs/sADCs,analogpreamplifieranddigitalI/FforPMTsignalprocessing;thedigitalI/FisLVDSdoubledatarate(DDR)at500MHz

• Storage:2GBytehighbandwidth,lowcostRAMchipsstoringupto1sofeventfragments

• EthernetPHY:a100Mb/sEthernetphysicallayerchipfordatareadoutandcontrol• PoE+transformer:a48V,30WIEEE802.3atcompliantchip,receivingpowersupply

throughEthernetcableinaccordancewithAlternativeA• ADC/DCconverterfeedingGCUwithappropriatevoltagerails• Linedriverandreceiver:suitablebuffersforclockreceptionandtriggertransmission

via~100mcoppercablepairs.• FPGA:gluelogic,highperformancestatemachines,communication,ASICreadout,

HVslowcontrol• Flashmemory:longtermstorageforFPGAconfigurationbit-stream;redundantfor

disasterrecoveryTo reduce the number of copper pairs reaching the GCU, the IEEE 802.3at Alternative Astandard for remotepower supplyof (usually) telecomhardwaremayhelp. The standardforeseestheinjectionofa~48Vbiasonthecentraltapsoftheisolationtransformersforthetransmissionandreceptionpairsof10/100Mb/sEthernetasshowninfig.3

INFNandUniversityofPadova-GCUProposal

4

Figure3.The remotepowereddeviceextracts the commonmodevoltageand feedsa localDC/DCconverterforappropriateuse,sinkingamaximumpowerof30.5W.Low cost commercial hardware is readily available to act as Ethernet Switch and PowerSourcingEquipmentforasmanyas48ports[3].ThesamepowersourcecanbeusedtosupplypowertotheHighVoltagemodule,althoughsomeefforthastobeenvisagedtoshieldtheGCUfrompossibleHVdischarges.TheEthernetmediumrequiresprotocolstobeofanypracticaluse,andJunoisnoexception.InthecontextoftheCMScollaboration[2],anefforthasbeenmadetodevelopanddeployanUDPbasedIP(IntellectualProperty)corethatabstractsanhardwarebusontopoftheInternetProtocolandiscalled“IPbus”[1].IPbusbringstheUDPprotocolattheFPGAleveland leveragestheubiquityof IP fordirectcommunicationwithdistributedFPGAsthroughEthernet. The IPbus ecosystem is comprised also of software APIs for a further layer ofabstraction,thatmakesdataacquisitionandslowcontrolforahighnumberofpeerdeviceseasiertomanage.IPbusisdeployedbyseveralprojectsandexperiments:CMS,ATLAS,LHCb,Compass,CBM,amongothersandneutrinoexperimentslikeSOLIDandDUNE.3.SynchronizationThe problem of GCU synchronization in Juno is related to the distributed nature of datareadout.EachGCUcollectsafragmentofaneventwhichhastobejoinedwithpotentiallymanyotherfragmentsofthesameeventbeforeprocessing.Abarrelsorterapproachthroughcommercialtelecom switches is a proven and efficient method to accomplish parallel eventreconstruction in an event building farm, as schematically shown in fig. 4. Most physicsexperimentsdataacquisitionsystemsadoptthistechniquebecauseofscalability,excellent

INFNandUniversityofPadova-GCUProposal

5

performanceatmoderatecosts,duetoreuseofcommoditywidespreadhardware.InJunoitmakesevenmoresenseifoneadoptsEthernetreadoutattheveryfront-end.

Figure4.ACentralTriggerProcessor(CTP)collectstriggerprimitivesgeneratedatCGUlevelwithfixedlatency and applies suitable algorithms (e.g. coincidence, multiplicity, windowing, etc.)according to physics constraints. Upon reception of a suitable number of primitives thatqualify a potentially interesting event, the CTP generates a tag related to a Global TimeCounter(GTC)thatmustbesharedamongCTP,LCUsandGCUs.ThetimetagwillthenbeforwardedtoallGCUsviaslowcontrolordedicatedlinkforreducedlatency.Uponreceptionofatime-taggedeventvalidation,GCUswilllookforamatchintheirstorageofbufferedeventfragments:thematchmayoccurinapresettimewindow,e.g.incloseproximityofthetimespecifiedbythetag. Incaseofamatch,GCUswillretrievethefragmentsfromtheirmemoriesandforwardthemtothedataacquisitionsystem(DAQ)viatheIPbusdedicatedlink.

INFNandUniversityofPadova-GCUProposal

6

Thisreadoutmechanismfeaturesvirtuallynodeadtimebutreliesdeeplyontheavailabilityof a global time at the level ofGCUs. In several experiments [4], [5], [6], the global timeavailabilityisbasedonthedistributionofatimecounterwithvaryingresolutionsbymeansofhighspeed(usuallyoptical)seriallinksthatembedalsootherusefulservices(clock,resets,etc.).InJuno,thereferenceimplementationisbasedaroundareducednumberoflonglength(~100m)copperlinks,thusavoidingthedeployofmultigigabitseriallinks.In thiscase,adifferentdistributionschemeforglobal timemustbeenvisaged.GCU’swillreceiveaglobalclockviaadedicateddifferentialpairandtheywillcountlocallythisclockthatwillactasalocalcopyoftheGlobalTimeCounter(GTC).ButeveryGCUwillexperienceanoffsetintheircopyofGTC.Thisoffsethastwocontributions:

• thestartof thecounting isnotsynchronizedamongGCUs.DuetodifferentpowercyclesanddifferenttimeofarrivalofresetcommandsinGCUs,theirGTCswillstartatdifferenttimesandwillkeepthesamepace.

• ThecableslinkingLCUswithGCUsarepresumablyofdifferentlengths:thevelocityfactorincat5cablesvaryfrom0.4cto0.7c,e.g.asignalpropagationspeedof~5ns/m.With an external clock of 62.5MHz, ~3m of length mismatch in clock cables aresufficientforaclockpulseslidingamongGCUs.

INFNandUniversityofPadova-GCUProposal

7

Figure5ThefigureshowsthemechanismofglobaltimesynchronizationbetweenoneLCUanditssubordinatedGCUsanditcanbesummarizedinthefollowinglist.Thesynchronizationprocedureisprecededbyameasureofroundtriptime(RTT)fromLCUsandtheirGCUsinordertorecovercablelengthsmismatches.

1. DCSwillputallGCUsinsynchronizationmode,byprogrammingoneofitsregistersviaIPBUS.

2. Fromthatmomenton,GCUswillignoredataandtriggerscomingfromADUs3. GCUswillmonitortheirlocalcopyofGTC,waitingforclockcycleinwhichtheirGTC

willmatchapresetvalue4. AtthatexacttimeallGCUswillissueatechnicaltriggeronthededicatedpairtotheir

LCUs5. LCUsrecordthetriggertimeoffsetsamongalltheirsubordinatedGCUs:those

offsetsreflecttheoffsetsofGCUslocalGTCs.6. AssumingthatLCUsshareacorrectGTC,theycaninformtheDCSofthemeasured

counteroffsetsinGCUs7. DCSwilladjustGTCineachGCUbyaddingthecorrectoffsetviaIPBUS8. DCSwillinstructGCUstoissueanothertechnicaltriggerwhentheirGTCsmatcha

newpresetvalueinordertoverifythecorrectnessofthesynchronizationprocedure9. DCSwillinstructGCUstoexitsynchronizationmode.

Thesynchronizationprocedure(excepttheRTTsmeasurements)canberepeatedperiodically(e.g.onceperhour)inordertoverifytheconsistenceofglobaltime.ItisworthnotingthatthedeadtimegeneratedbytheprocedurecanbekeptverylowifthepresetvalueatwhichGCUsissueatriggerisclosetotheactualGTC.ThehighestcontributiontodeadtimewilllikelycomefromDCSoperationsviaIPBUS,butweexpectthatthefullproceduremaylastintheorderofhundredsofms.4.FPGAreprogrammingFPGA reprogramming is an important feature for bug fixing, feature enhancements, faultfixingandmaintenance.Dueto inaccessibilityofGCUsafter installation,thetask iscriticalbecause the firmwaremaygetcorruptedduring the reprogrammingphase itself.A safetymechanismmustbeputinplacefordisasterrecovery.Awatchdogmechanism,periodicallytoggled,wouldallowtochoosebetweentwofirmwarerevisions(aproductionversionanda“golden”version)storedinaflashmemory.Inpresenceof external toggling, the FPGA would be presented and programmed with a productionfirmware at power-up, while in absence of external toggle, the firmware selected forprogrammingwouldbethe“golden”version.ThetogglemechanismcouldbesoftwaretriggeredviaIPbusorhardwaresupported,basedonthepresenceofanexternalsignal,e.g.theexternallysuppliedclock,asshowninfig.6.Inthe latter case,whenanexternal clock isnot fed toGCU, theprogrammingmemorygets

INFNandUniversityofPadova-GCUProposal

8

selectedwiththe“golden”firmware,theFPGArunsonaninternalclockand,afterapowercycle,theFPGAwillbeconfiguredwithaknown-to-be-workingfirmware.

Figure6.In case of bug fixes, firmware enhancements, tests, or GCU board qualification, the“production”firmwarecanbeoverwrittentakingcontroloftheflashmemorydatabusontheFPGA. With suitable support of IPbus mapped registers and FIFOs, the programmingprocedurecanbeperformedwithsimplesoftwarescripts.5.I/OTheInput/outputsignalsoftheGCUcanthereforebesummarizedasinthetablebelow.ApartfromHighVoltagebiasandcontrolsignalswhoserunisshort(e.g.,fewcm),allothersignalsaretobeconsidereddifferentialpairsfornoiseimmunityoverlongruns.

INFNandUniversityofPadova-GCUProposal

9

Name Type DirectionIPbustx Ethernet OutputIPbusrx Ethernet InputClock Cablereceiver InputTrigger Cabledriver OutputHV_i2c_SDA i2copto-isolateddata BidirectionalHV_i2c_SCL I2copto-isolatedclock BidirectionalHV_power_bias 48Vbias SharedHV_power_ref 48Vreference Shared6.ControlsTheadoptionofIPbuswillallowtransparentmanipulationofFPGAsregistersacrossEthernet,thuseasingthetaskofdistributedcontrol.TheinterfacetothePMThighvoltageboardcouldexploitsimpleserialbusses like I2CorSPI,conceivedforchiporperipheralcontrol.Thesebussesuseareducednumberofsignalsforcommunicationandalthoughdataexchangeisusuallysynchronoustoabusclock,pulsetimingcanextendintherangeofmsandbeyond,thus allowing direct software manipulation of bus signals instead of using dedicatedhardware.Thistechniqueisknownas“bitbanging”andcaneasilybeimplementedviaIPbus.7.References[1]-R.Frazier,G.Iles,D.Newbold,A.Rose:“SoftwareandfirmwareforcontrollingCMStriggerandreadouthardwareviagigabitEthernet”,TIPP2011.[2]–TheCMSCollaborationatLHC:http://cms.web.cern.ch[3]-http://www.dell.com/us/business/p/power-over-ethernet-switches[4]–P.Moreiraetal.“TheGBTproject”,TopicalWorkshoponElectronicsforParticlePhysics,Paris,France,21-25Sep2009,pp.342-346[5]–M.Bellatoetal,“GlobalTriggerandReadoutSystemfortheAGATAexperiment”,Real-TimeConference,200715thIEEE-NPSS[6]–TheIceCubecollaboration:“TheIceCubeDataAcquisitionSystem:SignalCapture,Digitization,andTimestamping,ARXIV,2008.

INFNandUniversityofPadova-GCUProposal

10

8.RevisionHistory

Date Version Revision11/12/2015 1.0 Initialrelease