1350 Precursor talk - MeerKAT Lessons Learned (Thomas Abbott

www.ska.ac.za

SKA Engineering Meeting October 2016

PRESENTER: Thomas AbbottFROM THE INTEGRATION OF MEERKAT ARRAY RELEASE 1

LESSONS LEARNED

www.ska.ac.za

Agenda

• StoryofMeerKATAR1integration

• MeerKATAIVplan

• Tools

• LessonsLearned

www.ska.ac.za

ThestoryofMeerKATintegration

• IntegrationstartedinJune2015• AR1systemwasintegratedandworkingby~May2016• ScaledupandbugsfixedforourAR1releaseinJune2016

www.ska.ac.za

2015– Earlyintegrationishard

January2015:ReceptorTestSystemisworking.RTSisasmallcorrelator,aprevious-generationtelescopemanager,andtworeceptors.

9June: WeconnectupacompletecopyoftheMeerKATsignalpathintheintegrationlab.It’sexpectedtoworkinafewweeks.

June: LabIntegrationeffort:twice-a-weeksessionsstart

July: LabIntegrationeffort,firstfileisrecordedfromasignalgenerator

August: Labintegrationcontinues

Sept: Labintegrationcontinues.Systemnowstartsupandrunsobservations

Oct: Labintegrationcontinues.Labverificationstarts.

www.ska.ac.za

July2015

Firstfilerecordedinlab:(cross-correlationofthesignalgeneratornoiseonly)

www.ska.ac.za

Connectingreceptors

July-October2015:Back-endisinstalledonsite.

Nov2015:FirstreceptorisreleasedfromRTS.Integrationofsitecorrelatorwithreceptor.

LateDec2015:FourreceptorsconnectedtoAR1toattempttoformafirstimagebeforetheendoftheyear.Systemisnotstableenough,butweget:Firstfringes Phase closure

Jan2016:BacktotworeceptorsasRTScontinuesonthesecondpair

www.ska.ac.za

MarchandApril2016

April:Firstbeamformer observationandofflinereductionofapulsar

March:FourreceptorsconnectedFirstroughimageformed

www.ska.ac.za

20May2016

Firstimagecreatedwitha12-hourobservationon4antennas–threeinthecoreandone4kmaway.ComparisontoearliersurveysshowsthehighresolutionofMeerKAT.

20May:8-antennacorrelatorrunsforthefirsttime

www.ska.ac.za

5June2016

16-antennacorrelatorrunsforthefirsttime,11receptors

www.ska.ac.za

10June2016

10June:16th receptorconnectedacceptedandintegrated(butwehaveonefailure,soweneeda17th receptor)

The17th receptorisconnectedaweeklater

www.ska.ac.za

16June2016

Disaster!Abuginthedelaytrackingsystemsurfaces,wheresomedelaysarenotcorrectlyappliedbythecorrelator.

Saturday18June,9am,anewversionofthecorrelatorhasbeencompiledandtested.Theweekend’sobservingstarts...

www.ska.ac.za

17June2016

Successfulpulsarobservations

www.ska.ac.za

19June2016

SuccessfulcontinuumobservationsofourDEEP2fieldReducedoverthefollowingweektoformtheimage

www.ska.ac.za

20June2016

Successfulspectrallineobservationofamasercalibrator

www.ska.ac.za

16July2016

Firstlightpressrelease

www.ska.ac.za

www.ska.ac.za

Agenda

• StoryofMeerKATintegration

• MeerKATAIVplan

• Tools

• LessonsLearned

www.ska.ac.za

MeerKATIntegrationandVerification

• Arrayreleases• I&Vplan• Verification• Tools

www.ska.ac.za

Arrayreleases

DefinedintheMeerKATIntegrationandVerificationPlanStagedreleaseoffunctionalityandscale:1. Receptorconstructionschedule2. ScaleofCBFandSDP3. FunctionalityofCAM4. FunctionalityreleasesforCBFandSP

www.ska.ac.za

Arrayreleases

AR1

16receptors

Correlator4k &32k

Offlineimaging

1beam

AR2

32receptors

Correlator4k &32k

Offlineimaging

1beam

AR3

64receptors

Correlator4k &32k

Pipelineimaging

1beam

Laterreleases

64receptors

Correlator1k,4k &32k

Pipelineimaging

4beams

Narrowbandmodes

Transient modes

VLBI

www.ska.ac.za

AR1IntegrationandVerificationplan

www.ska.ac.za

AR1Verification

AR1hasthreesystem-levelverificationeventsdefined:– VE1:Labevent,testingmostbasicfunctionality– VE2:Firstsiteverification.EssentiallythesamefeaturesastestedinVE1,butnowonthesitesystemwithrealreceptorsandskysignals.

– VE3:CompleteAR1verificationonsitesystem,withatleast6receptors.

Subsystemseachqualifiedaccordingtoowndevelopmentplan

www.ska.ac.za

www.ska.ac.za

Agenda

• StoryofMeerKATAR1integration

• MeerKATAIVplan

• Tools

• LessonsLearned

www.ska.ac.za

Tools

• ReceptorTestSystem• IntegrationLabinCapeTown• ManagementTools

www.ska.ac.za

ReceptorTestSystem

Firstantennasaredeliveredandintegratedbeforethetelescopeback-endisready.

InshortyouneedaTelescopetotestaReceptorEnterReceptorTestSystems(RTS)

TotesttheemergentpropertiesofaReceptorweneed:• Controlinterface(CAM)• Networktransportofdataandcontrol

(LAN/Fibre)• Timeandfrequencyreference(TFR)• Correlator(CBF)• Signalprocessingandreduction(SDP)• SpecialisedTestreceivers(KuRX)

www.ska.ac.za

ReceptorTestSystem

www.ska.ac.za

ReceptorTestSystem

Testsystem(RTS)needsqualificationaswell• Driftscan

• KuHolography

www.ska.ac.za

ReceptorTestSystem

L-bandtesting• Singledish:

– Nominalinput(widebandgain)– Tsys andTnoise diode– Tippingcurve– Gainstability– Linearity– Spectralbaseline

• Interferometric:– Reflectorefficiencies– Phasestability– Beamshapeandsidelobe levels– Polarisation

Ku-bandtesting• Singledish

– Gaincurve– Pointing

• Interferometric– Aperturephaseefficiency– Gaincurve– Errorbeam

www.ska.ac.za

IntegrationLab

AsimpleIntegrationLaballowedearlyintegrationtobedonelongbeforethesystemonsitewasready.

Ourlabconsistedof:• 2x40Gbps inter-officeLAN• Signalgeneratorandnoisesource• SimulatedAntennaPositioner• 8-inputcorrelator:12Roaches.Correlatormastercontroller• Onedigitiserandpedestalswitch.Digitisermastercontroller• ControlandMonitoringsystem,one(virtualised)server• Scienceprocessoringestnode• RegularMeerKATsciencearchive• General-purposetimeandfrequencyreferencesystem

www.ska.ac.za

IntegrationLab

www.ska.ac.za

Managementtools

VITECHCOREformodellingofthesystem.RequirementsspecificationsandverificationplansareprinteddirectlyfromCORE

WeuseJIRAextensivelyinallteamsformanagingweek-to-weektasksandreportingbugsandfeaturerequests

Primaveraprojectplanningsoftwareforhighlevelprojectplansandreporting

www.ska.ac.za

www.ska.ac.za

Agenda

• StoryofMeerKATAR1integration

• MeerKATAIVplan

• Tools

• LessonsLearned

www.ska.ac.za

LessonsLearned

• Prototyping• Labintegration• Earlyintegration• SubsystemInvolvement• Integrationteam• Systemstability– emergentbehaviour• Real-timecommunication• Cameras• RFI

www.ska.ac.za

PrototypeEverything

“Tomeasureistoknow”-- LordKelvinIndividualsubsystemsallbuiltmockups,models,orprototypes

• Don’tbelieveeverythingyoureadonaproduct’sspecificationsheet.Testitinyoursetup!

• CBFspenttwoyearswithsevennetworkswitchvendors.Allhadabsoluteconfidenceintheirproduct,butnonemanagedtomakeitoperateasadvertised,withoutsomerework.

• Atthesystemlevel,MeerKATprototypingstartedwithXDMandKAT-7.

www.ska.ac.za

LabIntegration

Beforethesitesystemwasavailable,theLabwasusedfor:• Nearlythreemonthsofintegrationwork

betweenCAM,CBFandSPsubsystems• Characterisationofdigitiser,CorrelatorPFB

leakageproblem• Substantialdevelopmentandtestingof

correlatorgateware• Discoveredatimeoffsetbetween

polarisationsindigitiser,nowimproved.• CorrelatorF-enginenon-linearity.• Beingusedtohuntadelayed-packetbug• IntegrationtestswithPulsar-timinguser

Duringsitetesting,theLabwasusedfor:• Releasetestingofnew

CBF,SPandCAMsoftware

Ongoingapplication• Evaluationofsub-

channelPFBeffects• Integrationeffortswith

USE– PulsarTiming,MeerTRAP

• DigitiserandCorrelatorgateware testing

www.ska.ac.za

Earlyintegration

Integrationismuchharderthanexpected• Allsubsystemsperformverycareful,thoroughverificationindependently.• Still,thereareinterfacingnigglesonlyexposedduringintegration.• Besttofindtheseearlywhileitcanbecorrectedwithoutimpacting

systemscheduleorbudget.• Asdescribedearlier,ittookusseveralmonthstointegratethreesystems

andgetthemtotalkandworknormally.

Thelabenvironmentenabledthisearlyintegration.

Subsystemsshouldbringareduced-functionalityversionoftheirsystemtotheintegrationlab,6monthsbeforethefull-functionalityrelease.

www.ska.ac.za

Subsystempeopleneedtobeinvolved• Integrationcannotbedonebytheintegrationteamalone.• Initially,subsystemexpertswilldothebulkoftheintegrationwork• Later,theintegrationteamwillbeabletotakeoversomefunctions

Integrationteam

SubsystemInvolvement

Subsystemexperts

Verification

Startofintegration

time

www.ska.ac.za

StrongIntegrationTeam

Astrongintegrationteamisrequiredforthelaterstagesoftheintegrationeffort.

Thefirsthalfcanrunalmostentirelyonpeoplefromsubsystems,andasthey’reresponsiblefortheproblems,they’rethebestpeopletosolvethem.

Canbemanagedbyasmallteamof

subsystemexperts

Requiresseveralspecialistswhodoonlyintegration

Involvetelescopeoperatorsearly.Thetelescoperequiresbabysittingallnight,thetechnicalteamleadscan’tkeepthisupformonths.

Laterwhensystembehaviourbecomesthemainproblem,theintegrationteamneedsseveralindependentexpertsaswellasthesubsystemteammembers.

www.ska.ac.za

Systemstability- emergentbehaviour

Overallsystemstabilityisanelusivegoal,muchmoredifficultthangettingittoworkonce.Ittakesanobsessionforqualityinallparts,notsimplygettingittowork.Toachieveasystemthatworks95%ofthetime,everypartmustworkat99%ofthetime.Stabilityproblemsweremostlyemergentbehaviour,neweffectsthatcroppedupduringintegration,orevenonceithadbeenworkingforawhile.

www.ska.ac.za

Real-timecommunication

TheintegrationteamusedIRCtocommunicatebetweenvariousfloors,officesandsite.Italsoworkedwellonacomputerpluggedintoanyreceptor

Communicationtositewasaproblem,withtheRFIrestrictions.Nocellphones orsatellitephones.WehaveafewIPphonesinsomereceptors.

www.ska.ac.za

Camerasforsitevisibility

Thehighelevationcamerawasveryusefulforobservingthingsonsite.Withoutbeingabletocallpeoplethere,itwasgoodtobeabletowatchandsee.

SurveillancecamerasfosterasenseofclosenesstotheactionintheKaroo.Considerhavingcamerasinthecontrolroom,andmajorintegrationsites,forotherstoseewho’sthereandwhat’shappening.

www.ska.ac.za

DoNotUnderestimateRFIchallenges

Thisisanultrasonicwindspeedanddirectiontransducer.

QualifiedforuseonKAT-7andinstalledforyearsonsite.

DuringRFIqualificationoftheMeerKATWSSwediscoverthatthissensorfailsthenewlimitsby20dB

www.ska.ac.za

Summary

• StoryofMeerKATAR1integration

• MeerKATAIVplan

• Tools

• LessonsLearned

• Questions

www.ska.ac.za

ThomasAbbottMeerKATSystemsEngineer

Email:tabbott@ska.ac.za

Contactinformation

SKASouthAfrica,aBusinessUnitoftheNationalResearchFoundation.WearebuildingtheSquareKilometreArrayradiotelescope(SKA),locatedinSouthAfricaandeightotherAfricancountries,withpartinAustralia.TheSKAwillbethelargestradiotelescopeeverbuiltandwillproducesciencethat

changesourunderstandingoftheuniverse