SCADA - User Manual Introduction and Purpose

SCADA-UserManualAuthor:ConnorWiniarczyk

IntroductionandPurposeSCADAisanacronymforSupervisoryControlAndDataAcquisition.Itisatermborrowedfromindustrialcontrolapplications,whereasingleserverisoftenusedtooverseelargeindustrialplantssuchasoilrefineriesandassemblylines.Ingeneral,SCADAsystemshavethreefunctions,theyacquiredatafromthenetworkofsensorsconnectedtotheplant,sendcontrolsignalstotheplant’sothersubsystems,andprovideaninterfaceforhumanstointeractwiththeplantbyviewingaggregateddataandissuingcommands.

TheLafayetteFSAEteamhasbeenworkingtodevelopaSCADAsystemthatcanbefullyintegratedintoitselectricvehicle,withthegoalofperformingalloftheabovethreefunctionsbothduringnormaloperationandthroughoutthevarioustestingandmaintenanceproceduresthatitwillundergo.Attimesinthepast,thissystemhasbeenreferredtoasVSCADA,shortforVehicleSCADA,todistinguishitfromtheindustrialsystemsdescribedabove,butforbrevitythisdocumentwillrefertoitsimplyasSCADA.

Thisdocumentisreferredtoasausermanual,butcouldperhapsbebetterdescribedasaninheritancemanual.Itincludesinformationabouthowtousethesoftware,butitalsoincludesabunchofotherinformationthatwouldn’tbeusefultoanenduser,butwouldbetosomeoneinheritingthisprojectwiththeintentionofexpandinguponorimprovingit.Itincludesinformationsuchastheoveralldesignandstructureoftheproject,whyitwasdesignedthewaythatitwas,whatIwasthinkingwhiledesigningit,andmypersonalthoughtsforhowitcanbeimprovedandexpandedon.Forthatreason,itismoreverbosethananordinaryusermanualmightbe,butIhopethatthereaderwillfindthisinformationuseful.

DesignOverview

CANNetwork

TheexistenceofaSCADAsystemimpliesanetwork.TheremustbecommunicationtakingplacebetweenSCADAandtheothersubsystemsforittosuccessfullyperformanyofitsresponsibilities.Thereareanumberofnetworkprotocolstochoosefrom,butthedefactostandardforautomotiveuseisCAN(ControllerAreaNetwork)andsothatiswhatisusedintheLafayetteFSAECar.

AgoodexplanationofCANcanbefoundhere:

https://www.csselectronics.com/screen/page/simple-intro-to-can-bus/language/en

Atthetimeofwriting,theCANNetworkcontainsbetween6and7nodes,dependingwhetheranexternalpcisneededtoconfigureandoperatethemotorcontroller.Theyare:

MotorControllerSCADABatteryPacks(1and2)TSIDashManExternalConfigurationPC

CANOpen

CANisaverylowlevelprotocol.Itdefinesawayforanodetobroadcastupto8bytesofdataandanID,butveryfewoftheotherthingsneededtoperformsophisticatednetworkoperations.Forthis,ahigherlevelprotocolneedstobedefinedontopofCAN,andcurrentlythereareseveralcompetingstandards.Thesehavebeendescribedastheequivalentofsomethinglikehttptothetcp/ipstack,addinganadditionallayerofabstractionforeasieruse.

Whileitisnotthemostpopular,theLafayetteFSAEteamhaschosentheCANOpenStandardforitsvehicleinordertomatchthatofthealreadypurchasedMotorController,whichincludesarichsetofCANOpenbasedtools.

CANOpencanalsobethoughtofasasubsetofCAN,meaningifanodeisaddedtothenetworkthatdoesnotcomplywiththeCANOpenstandard,thebehavioroftheothernodesisundefined.Forthisreason,itisveryimportantthateverynodeaddedtothenetwork,evenifitdoesnotintendtomakeuseofthefullsetofCANOpenfeatures,atleastcomplywithasubsetoftheprotocol.

Herearesomegoodresourcesforlearningmoreabouttheprotocol:

https://www.can-cia.org/canopen/

https://www.youtube.com/watch?v=DlbkWryzJqg

Asasupplementtotheseresources,abriefdescriptionofCANOpenisalsoprovidedhere:

CANOpencanbestbethoughtofasthesumofseveralcommunicationprotocols,coupledwithsomedefinedbehaviorforeachnode.Eachhasadifferentpurpose,name,andassociateacronym.Anodecanchoosetoimplementanysubsetoftheseprotocolsandbehaviors.

Theprotocolbeingusedisdeterminedbythefunctioncode,whichistransmittedintheidpartoftheCANframe.IntheCANOpenstandard,themessageidisformedbythesumofthefunctioncodeandthenodeid,thisistotakeadvantageoftheCANbehaviorofprioritizingmessageswithlowerids,soprotocolsthataremoreimportantaregivenlowerfunctioncodes.

NMT

NMTstandsforNetworkManagement.Itisamasterslaveprotocolusedtomanagethestateofthevariousnodesonthe

bus.AnynodewhichwishestointeractwiththeNMTprotocolasaslavemustimplementthefollowingfinitestatemachine.

Thisdeterminesthefunctionalstateofthenode.CANpacketscanbesentbythemastertomovethenodealongthisstatemachine,allowingittoperformactions,likesoftresets,emergencystops,andbootups.

SDOandOD

AnynodewhichwishestoexposeinternaldatatotheCANnetworkmustimplementadatastructureknowasanObjectDictionary,orOD.TheObjectDictionarymapseachpieceofdatatoanaddressconsistingofatwobyteindexfollowedbyaonebytesubindex.Certainaddressesarereservedforgeneraldatalikedevicenameanderrorregisters.ThemanufacturerofanodecanpublishinformationabouttheObjectDictionaryinafilecalledanElectronicDataSheet,orEDS,whichtakestheformofanINIfilethatisbothhumanandmachinereadable.

AnynodeonthenetworkcanaccessinformationfromtheObjectDictionaryofanothernodeusingtheServiceDataObject,orSDOprotocol.TheSDOpacketconsistsofabyteofmetadatafollowedbyathreebyteaddressand4bytesofdata.Thiscanbeusedtobothreadandwritedata,andcanbeusedtocontrolnodebehaviorinrealtime.Thisisthetechniqueusedinthedynoroomtospinthemotorandqueryitfordataliketemperatureandangularvelocity.

PDO

ProcessDataObjectsareaprotocolmeanttosupplementSDO’sbyprovidingadatatransfermethodwithahigherdatarateandlessoverhead.Theyaremeanttobethestandardforhighvolumeinternodedatatransferduringnominaloperationofthenetwork,andhavetheaddedbenefitofbeingmuchsimplertoimplementthanSDO.AnodethatchoosestoimplementonlythesmallestpossiblesubsetoftheCANOpenprotocolwillmostlikelyimplementaPDO.

PDO’sworkonaproducer/consumer,orbroadcast/subscribemodelofcommunication,whereoneormoreCANpackets

aresentatregularintervals,eachcontaining8bytesofdatawithastructureagreeduponbeforehand.Anynodeonthenetworkcansubscribetothesepacketsandupdatetheirbehavioraccordingly.Boththebroadcastandsubscribebehavior(calledtheTransmitPDOandReceivePDOrespectively)canbeconfiguredbydedicatedaddressesintheObjectDictionary.

Hardware

SCADArunsonaRaspberryPithatismountedinsideoftheCarManenclosure.Attachedtoitisa“hat”thatenablesCANcommunicationandprovidesapassthroughforthePi’sGPIOpins.Together,theseareconnectedtotheGLVboardviaasetof3cables:AmicroUSBcableforpower,DB9serialcableforCANhighandlowsignals,andahomemaderibboncableforcertainGPIOpins.AdditionalcablescanberuntoconnectthePitootherpartsofthecarand/orDynoRoom.Theseinclude:

RibbonCabletotheCarManDisplayHDMItototheDynoRoomMonitorUSBtoexternalkeyboardandmouseUSBtothedynamometer

author’snote

ThethreecablesconnectingtheSCADAPitotheGLVboardareabaddesigndecisionandshouldberemovedatthenextpossibleopportunity.WiththeexceptionoftheDB9,thesecablesandtheirconnectorsarenotratedforautomotiveuse,andcouldeasilybecomedisconnectedduringheavyvibrationsandrenderSCADAinoperable.Inaddition,theadjacencyoftheSCADAPiandtheGLVBoardwithintheCarManenclosuremakethesecablesappearcomicallylarge,forcingspacetobeusedforcableroutingthatcouldotherwisebetakenupbysomethingmoreuseful.ItwouldbeafarbettersolutiontointegratetheSCADAPiandtheCANhatintotheGLVboard.

RaspberryPioffersacomputemodule,whichisthecomputerastheordinaryRaspberryPi3,butwithsomeperipheralsremoved,andcondensedintoamuchsmallerformfactorwithasinglecardedgeconnectorasitsonlyinterface.IhighlyrecomendthatfutureteamsreplacetheexistingSCADAPiwithoneofthese,withexternalcomponentsliketheCANinterfaceandethernetjackintegratedintotheGLVboard.

https://www.raspberrypi.org/products/compute-module-3-plus/

Software

ThemostimportantthingtounderstandaboutthedesignofSCADAinitscurrentformisthatitisnotasingleprogram,butratherasystemofprogramswhicharemadetorunconcurrentlyandinterfacewitheachother.Theseprogramstryasmuchaspossibletoabidebytheunixphilosophyofsoftwaredevelopment,whichisinessence,towriteprogramsthatareextremelylimitedinscope,andthatinteractwellwitheachother,throughstandard,welldefinedinterfaces.ThisisopposedtothewaythatSCADAhasbeenwritteninthepast,whereasinglelarge,integratedprogramismadetoimplementallthefeaturesrequiredbythesystem.Therearesomedrawbackstothesmallandmodularapproach,whichwillbeaddressedlater,butalsoalotofadvantages.Theyincludethefollowing:

Featurescanusuallybeaddedwithouthavingtomodifyexistingcode,thiscanhelptomitigatethesquarelawofaddingfeaturestothings.

SCADAcanbewritteninmultipledifferentlanguages,sothebestlanguagecanalwaysbechosenforagivenpurpose,withouthavingtomakecompromises.

Whenthepurposeofaprogramisgeneralenough,existingopensourcesoftwarecanbeusedinsteadofwritinganinhousesolution.

Theindividualcomponentsofthesystemareeasytounderstandandreasonabout.

Whenproblemsarise,theyareeasytoisolateanddiagnose.

TheprogramsmakingupSCADAandtheirstructurearedescribedinthefollowingdiagram.Theyareorganizedintoapipeline,withdataflowingfromoneservicetothenextinawelldefinedway.Eachservicehasaspecificrollineither

organizingortransformingthedatasothatitcanbemadesenseofbytheuser.

CANbus User

Calibrator

Sorter

Clients

Logger

Redis Postgres

SCADADataAquisitionPipelineDataflowsfromtheCANinterfacetotheuserthroughaseriesofservicesandstoragemediums.Eachofwhichservesadistinctpurpose.

Whatfollowsisashortdescriptionofeachserviceanditsfunction.

Sorter

ThesorterlistenstotheCANbusforincomingmessages,uponreceivingamessage,itchecksitsIDandStructureagainstaconfigfiletogenerateanassociatedsetofkeyvaluepairs.ItthenwritesthesekeyvaluepairsintotheRediscacheandsendsamessagetothecalibratorthatnewdatahasbeenreceived.ThesorteristheonlyservicethatisawareofCANorCANOpen,meaningSCADAcanbemadetoworkwithadifferentnetworkingprotocol,ormorethanone,simplybyswitchingoutthesorterserviceoraddinganadditionalone,withoutaffectinganyservicesdownstream.

Calibrator

ThecalibratorisresponsiblefortranslatingtherawdatareadbythesorterfromtheCANbusintoamorehumanreadableform.Thisisusuallyacaseoflinearcalibrationofrawsensordata,suchaswiththeTSInode,ortherecombiningofdatathatwastransmittedasmultiplebytes,likewiththeThrottlevaluereportedbyMotorController.However,itcanbemadetosupportcalculationsofarbitrarycomplexity,suchasforexample,thecalculationofpowerfromagivenvoltageandcurrent,theaveragingofmultiplesensors,filtering,differentiation,andsoon.Allcalibratoroperationsaredefinedbytheuserasfunctionsintheuser_cal.pyfile.Oncedataiscalculated,itiswrittenbackintotheRedisserverasadifferentkey.

Logger

TheloggertakesthedatagatheredbythesorterandcalculatedbythecalibratorandlogsasubsetofitintothePostgresqldatabaseatregularintervals.Thesubsetofdatatobeloggedisdefinedinconfig.yamlfile.

Itsamplesthedataatregularintervals,andperformssomebasiclogictolimitthedatabasetoonlystoringusefuldata.Ingeneral,itittriestoonlylogdatathatischanging,soasnottofloodthedatabasewithconstantvalues.Theloggerwritesallloggeddatatoasinglesqltable,calleddata,withthefollowingcolumns:

idsensor_idvaluetimestamp

Inkeepingwiththeunixtradition,alldataisstoredinthesamewayastext.Programsreadingfromthedatabaseareresponsiblefortranslatingthedataintoitsexpectedtype.Acompaniontable,calledsensorsisincludedinthedatabasetostoremetadataabouteachsensor.Ithascolumns:

idredis_key(equivalenttosensor_id,oneoftheseshouldprobablyberenamed)display_namedatatypeunit

Redis

Forthemostpart,programscommunicatewitheachotherviaRedis.Redisisanopensource,inmemorydatabasethatstoresdataaskeyvaluepairs,italsofunctionsasabasiccommunicationsbusforsendingsimpleinter-servicemessages.

https://redis.io/

Postgres

Ifaprogramneedstowritenon-volatiledata,itdoessousinganSQLdatabaseviaaPostgresserver.

https://www.postgresql.org/

Concurrency

Concurrencyishandledbytheoperatingsystemviasystemd.Allprogramswhicharemeanttoberunasaservicehaveanassociated.servicefile,andcanbemanagedwiththesystemctlinterface.Allservicesaresinglethreadedandshouldconsistofonlyoneupdateloop,delaystatementsareinsertedintotheloopinordertoreleasethecpuforusebyotherservices.Agoodexampleofanupdateloopforaservicethatrespondstoredismessagesisincludedbelow:

https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units

Clients

BoththeRedisandPostgresserversexposeTCPports,andanysoftwarecapableofinteractingwiththeseTCPportscanbeconsideredaSCADAclient.TheseclientscanberuneitherlocallyontheSCADAPiorremotelyoveranetwork.TherearealreadyanumberofSCADAclientstochoosefrom,andthebestoneforanygivenapplicationwilldependlargelyoncircumstanceandpreference.

Onepotential3rdpartyclientthatcomesveryhighlyrecommendedisGrafana.Grafanaisanopensourcedataviewingandmonitoringtoolthatspecializesintheconstructionofawiderangeofdatavisualizationssuchasgraphs,gauges,statistics,andlogsbasedonanumberofpotentialdatasources.ItsupportsPostgresqloutoftheboxandisextremelyeasytosetup.Italsohasarichcommunityofplugindevelopers.

whileTrue:message=p.get_message()ifmessage:update()else:time.sleep(0.1)

https://grafana.com/

Otherclientsareindevelopmentaswell,includingacommandlinetoolforbasicmanagement,acursesbasedmonitoringtoolforeasytestingoverssh,aJavabasedtoolforgraphgeneration,andaTkinterbasedtoolforgraphicallivemonitoringandmanagement.

FutureGoals

Atthemoment,SCADAisastrictlypassivesysteminitsrelationshiptotheCANbus.TheservicesdescribedaboveimplementapuredataacquisitionsystemwhichlistenstotheCANbusandpresentstheincomingdatatotheuser,butdoesnothavetheabilitytotalkonthatbustoissuecommandsorconfigurethenetworkforcertainbehaviors.ThismeansthatSCADAis,bydefinition,incomplete.Thereneedstobeawaytoperformsupervisorycontrolfunctionsbytalkingbacktothenetwork.

BecausedataacquisitionisthemostimportantpartoftheFSAEteam’sneedforaSCADAsystem,theseweretheservicesthatwerebuiltfirst,butmoreworkisrequiredtoturnthisintoacompleteSCADAsystem.Inaddition,theSCADAPIneedstoperformsomesecondaryservicesthatarenotstrictlypartoftheSCADAsystem.AnexampleofamorecompleteSCADAdesignthatincludesnon-implementedfeaturesisshownbelow.

CANbus

User

Calibrator

Sorter

Clients

Logger

Redis Postgres

SCADABlockDiagramShowsanexampleofacompletedesignofSCADA,includingfeatureswhichhavenotyetbeenimplemented.Featureswhichareplannedbutnotimplementedattimeofwritingaredrawninred.

Watcher

InstructionParser

GLV-CANInterface

GLVHardware

SubsystemEmulators

Alistoftheadditionalservicesdepictedandadescriptionoftheirfunctionisincludedbelow:

InstructionParser

Theinstructionparserwouldserveasacounterparttothesorter.InsteadofreadingfromtheCANbus,itwouldwritetoit,andserveasthebridgethroughwhichtherestofSCADAcouldsendcontrolsignalstothecar’sothersubsystems.Inorder

tokeeptheotherSCADAservicesagnosticaboutexactCANmessagesyntax,atextbasedformatshouldbeagreeduponforrepresentingcontrolsignalsthatcansentovertheCANnetwork.TheinstructionparserwouldthenhavethesoleresponsibilityofparsingthisformatintoCANmessages.

AproposalforsuchaninstructionsetisincludedwithSCADAdocumentationandcanbereadhere:

https://raw.githubusercontent.com/Lafayette-FSAE/scada/master/documentation/sparkyscript.pdf

Watcher

Theinstructionparserwould,bydefault,receivemostofitsinstructionsfromclientprograms,whichwouldinturnreceiveinstructionsfromtheUser,whowouldmakedecisionsaboutwhichcommandstosendbyviewingtheincomingdata.Thisformsafeedbackloopfordoingcertainkindsmanualsystemcontrol.However,therewillmostlikelybeaneedforanadditional,automaticfeedbackloopwhichcantakeplaceatahigherfrequencyandwithoutuserintervention.

Intheory,aclientcouldbewrittentoperformsuchatask,butitwouldnotcontinuetorunafteritwasdisconnected.Instead,adedicatedserviceshouldbewritten,whichregularlypollstheRedisserverforthestateofthecar,andmakesdecisionsaboutwhatcommandstosendbasedonthisdata.Itcouldbeconfiguredasalistofcondition,instructionpairs,andwouldbeusefulforthingslikeopeningthesafetyloopduringemergencyconditions.

GLV-CANInterface

TheGLVboardcontainsanumberofsensorsandcontrolvectorsthatshouldideallybeexposedtotherestofthecarusingthestandardCANOpeninterface.Thiscould,intheory,bedonebyadedicatedmicrocontroller,likeisdonewiththecar’sothersubsystems,buttheproximityoftheSCADAPitothesesensorshasmeantthat,forbetterorforworse,ithasabsorbedtheseresponsibilities.

Nevertheless,theGLV-CANinterfaceshouldbethoughtofasdistinctfromSCADA,andwrittenasitsownprogramthatonlyinterfaceswithSCADAthroughtheCANbus.Thishasthedrawbackofrequiringsomeextrastepsinthecommunicationprocess,butitallowsothernodesonthenetworktoaccessdatafromtheGLVsystem.Inaddition,ithelpstokeepSCADAimplementationmore“pure”inthesensethatitdoesnotrequiretheneedforbuiltinexceptionsandspecialbehaviorbasedonthemethodthedatawasobtained.PreviousimplementationsofSCADAwereriddledwithsuchexceptions,andtheywereacommonsourceofcomplaints.

SubsystemEmulators

Intheabsenceofafullyintegratedsystem,testingtheCANinterfaceofagivensubsystemcanbeadifficulttask.Forexample,youmaywantoneofthebatteriestobehavedifferentlybasedonthestateoftheotherone,butonlyhaveconstructedasingleprototype.TheDashMansystemmaywanttotestthewayitrespondstoacertainTSIstate,butthelastTSIboardmayhavebeenfrieddotoawiringerror.Andofcourse,youmaybeforcedbyunforeseencircumstancestodevelopSCADAandallothersubsystemsremotelyandwhileforbiddentocomewithinsixfeetofthelab,thecar,oreachother.

Forsuchevents,the2020teamhasbegunworkonaseriesofSubsystemEmulators,whicharemeanttoberuneitherontheSCADAPioraseparateCANconnectedcomputer,andmimictheCANinterfaceofeachsubsystem.Eachemulatorshouldmimicitssubsystem’sObjectDictionary,SDOserver,andTransmitPDOmessages.Whilenotnecessary,additionalhardwarelikeLEDsandbuzzerswouldalsobefun.

ExtraIdeas

SDOs

TheinstructionparserwouldalsoberequiredforreadingSDOdatafromthenetwork.ThisisbecausetheSDOoperatesontheclientservercommunicationmodel,ratherthantheproducerconsumermodelusedbyPDOdata.

HowtoUse

Installation

Prerequisites

SCADAassumesitisbeingrunonadebianbasedlinuxdistribution.Thiswillalmostalwaysberaspbian,butithasalsobeentestedonapuredebianserver.Thisshouldalsoworkonsomethinglikeubuntu,butthishasnotbeentested.SCADAwillnotworkonaWindowsorMac,andmostlikelyneverwill.Thisisbydesign,asSCADAtakesadvantageofalargenumberofassumptionsaboutitsenvironment.

ItisrecommendedtosetupadedicateddevelopmentserverforSCADAusingaspareraspberrypiandideallysomethingresemblingtheGLVhardware.ThiscouldbeexpandedovertimeintoincludeamockCANbusforintegrationtestingwithothersubsystems.

BecausetheLafayettenetworkcanbedifficulttonavigate,itisrecommendedthatthisbebypassedusingeitheradedicatedphysicalnetworkorvpn.Thiscouldbesetupinsuchawayastopreventipaddresschangesandtoenableoffsitework.

InstallScript

Oneofthedrawbacksofthesmallandmodularapproachtosystemdevelopmentisthatinstallationbecomesabitmoretricky.Tohelpmitigatethis,aninstallscriptisincludedwithSCADAwhichismeanttohelpautomatetheprocess.

Ideally,afullinstallationofSCADAcouldbeperformedwiththefollowingshellcommands:

However,becauseSCADAisstillrelativelynewsoftware,theinstallscriptislikelytofailinsomeenvironments,requiringmanualintervention.Inaddition,updateswillneedtobemadetotheinstallscriptperiodicallyasnewfeaturesareaddedtoSCADA.Tothatend,anexplanationoftheinstallscriptandwhatitdoesisprovidedhere.

Thissectionusesapt-gettoinstall3rdpartyprogramsanddependencies.NewdependenciescanbeaddedtoSCADAbyappendingapt-getcallstothislist.

Next,thepythonpackagemanagerisusedtoinstallpythonspecificdependencies.

ThePi’sCANinterfaceiscreatedandturnedon.ThisiscurrentlyconfiguredtouseavirtualCANinterfacefordevelopmentandtesting,butcanbeswitchedtotherealCANbusbyreplacingallinstancesofvcanwithcan.

$gitclonehttps://github.com/Lafayette-FSAE/scada$cdscada$sudobashinstall

apt-getinstallpython3apt-getinstallpython3-pipapt-getinstallredis-serverapt-getinstallcan-utils

#installpythondependenciespip3installpython-canpip3installredispip3installblessedpip3installpsycopg2-binary

#makesurevirtualcanbusissetupfortestingmodprobevcaniplinkadddevvcan0typevcaniplinksetupvcan0

#makebinaryfilesexecutablechmod+xinstallchmod+xscadachmod+xsorter/sorter.py

Fileswhichneedtobeexecutablearemadesoexplicitlywiththechmodcommand.Thisincludesscriptswhicharerunasservices,thescadacliinterfaceandthisinstallscript.

Copyexecutablefilesintoaknowndirectory.Theexactdirectorychosenissortofarbitrary,solongasitmatchesthedirectorywritteninthe.servicefiles./usr/binisagoodchoicebecauseitisthestandardforuserinstalledbinaryfilesanditisalreadyinthePATHvariable.

Thisalsocopiesfilesintoaknowndirectory,thistimethenonexecutablefiles,andtothe/usr/etc/scadadirectory.Againthiswaschosenarbitrarily.Thisiswheretheserviceswilllookforthingslikecustompythonlibrariesandconfigfiles.

Finallycopythe.servicefilesintoaplacewheresystemdcanfindthem.Thisdirectoryisnotarbitrary,andmustbe/etc/systemd/system

VerificationandTroubleshooting

IfSCADAwasinstalledcorrectly,youshouldnowhaveacommandlinehelpertoolthatcanbeusedforbasicsystemmanagement.Verifythisbytypingscada--helpintothecommandlineandrunningit.Youshouldseeahelpmessagedescribingthepossiblesubcommandsthatcanbeused.

Thestatusofeachservicecanbeverifiedusingthecommandscadastatus,whichisequivalenttorunningsystemcmtlstatus<service>foreachserviceincludedinscada.Youshouldseeanentryforcalibrator.service,logger.service,andsorter.service,ifallisworkingtheyshouldallbegreenandhaveastatusofactive.Ifoneormoreserviceismissing,theywereprobablynotinstalledcorrectly,youcanlookfortheirservicefilesinthedirectory/etc/systemd/system.

Ifoneormoreoftheseservicesisinthefailedstate,thatusuallyindicatesthattherewasanerrorrunningtheirassociatedpythonscript.Theseerrorscanusuallybefoundbycheckingthelogs,whichareaccessedwiththecommandsudoscadalogs.

Servicescanalsobetroubleshootedbystoppingthemandrunningthescriptmanually.Thecommandsudoscadastop<service>willstopagivenservice,andsudoscadastart<service>canbeusedtorestartit.Anyservicecanberunmanuallybyinvokingitspythonscriptfiledirectly.Forexamplepython3~/scada/sorter/sorter.py

Ifallservicesarepresentandrunningwithouterrorsyoushouldbegintestingthesystemagainstsomerealcandata.Ifnoexternalcanbusisavailable,theincludedsubsystememulatorswillgenerallybetheeasiestway.Thesedonothaveassociated.servicefilesandmustberunmanually.Navigatetothe~/scada/emulatordirectoryandrunpython3main.py.Forconvenience,itisbesttorunthisinatmuxwindowsothatothertaskscanbeperformedwiththisprograminthebackground.

Thisisaverygoodguideabouthowtosetupandusetmux.https://www.hamvocke.com/blog/a-quick-and-easy-guide-to-tmux/

chmod+xcalibrator/calibrator.pychmod+xlogger/logger.py

#copybinaryfilesto/usr/bincpscada/usr/bin/scadacpsorter/sorter.py/usr/bin/scada_sorter.pycpcalibrator/calibrator.py/usr/bin/scada_calibrator.pycplogger/logger.py/usr/bin/scada_logger.py

#createaworkspaceandcopyimportantfilesintoitmkdir-p/usr/etc/scadarm-rf/usr/etc/scada/utilscp-rutils/usr/etc/scada/utilsrm-rf/usr/etc/scada/configcp-rconfig/usr/etc/scada/configcp./install/usr/etc/scada

#copyservicefilesforsystemdcpsorter/sorter.service/etc/systemd/systemcpcalibrator/calibrator.service/etc/systemd/systemcplogger/logger.service/etc/systemd/system

Oncetheemulatorisupandrunning,youshouldtesttoseeifitisgeneratingCANtraffic.Thebestwaytodothisisbyusingthecandumpcommandthatisbuiltintocan-utils.Typethecommandcandumpvcan0andrunit,youshouldseearegularstreamofcanpackets.IftheTSIemulatorisrunning,youwillseearegularpacketwiththecanidof183,thisisthesumofthefunctioncode0x180,indicatingthatitisaPDOpacket,andthenodeidof3,indicatingthatitiscomingfromtheTSI.IfyouareusingadifferentCANinterface,youwillneedtoswitchthevcan0argumenttothecorrectone.Thiswillbecan0inthelivesystem.

ThesorterandcalibratorcanbeverifiedbylookingattheRediscachedirectly.Typethecommandredis-cli--scantogetaprintoutofallactivekeys.Redisdatawrittenbythesorterandcalibratorwillexpireafteraperiodof10seconds,soalackofdatausuallyindicatesthatoneorbothofthemhavestopped.Thevalueofoneofthesefieldscanbereadusingthecommandredis-cliget<key>.Agoodtestistoquerythetsi:throttlekey,itshouldbechangingovertime.

Theredis-cliisaverypowerfultool,andiseffectivelyitsownSCADAclient.Moreinfoabouthowtousetheredis-clitoolcanbefoundhere:

https://redis.io/topics/rediscli

Thelaststepistoverifythattheloggeriswritingdataintothedatabasecorrectly.Theeasiestwaytodothisisbyqueryingthedatabasewiththepsqlcommand.

Runningthecommand

ThedatabasenameandusernamewillbetheoneschosenwhilesettingupPostgres.Thiswillpromptyouforapassword,whichisalsochosenduringthesetupprocess.Ifsuccessful,youwillbeenteredintoaREPLwhereyoucanexecuteSQLcommandsagainstthedatabase.

Typing\dtwillshowthelistoftables,andshouldincludeatleastadataandsensorstable.Runningthecommandselect*fromdata;willshowthecontentsofthedatatable.Confirmthattheloggerisworkingcorrectlybycheckingtheendofthetabletoseeifnewdataisbeingwritten.

Configuration

WhileSCADAconsistsofseveraldistinctprograms,forthesakeofconvenience,theyallreadfromthesamesetofconfigurationfiles.Atthemoment,therearetwo,bothofwhichcanbefoundintheconfigdirectory.

config.yamlisageneralpurposeconfigfilewhichhandlesthemajorityofoptions.user_cal.pyisapythonfilespecifictothecalibratorwhichdefinesthecalibrationfunctionsitwillusetomapdataasitcomesin.TogethertheymakeupthefullsetofconfigurationoptionsforSCADA.

Thefollowingsectionsexplaintheconfigurationoptionsinfurtherdetail.

BusInfo

ThebusinfosectiongivesthesorterinformationabouttheCANbus.IfyouarerunningSCADAonasystemwithavirtualCANinterface,setchanneltovcan0.Asitiscurrentlyconfigured,theraspberrypiintheDynoroomtalkstotherestofthenetworkonthecan0channelofsocketcanwithabitrateof125000

Data

Mostdataissentoverthebusin8byteCANpacketscalledProcessDataObjects,orPDOs,witheachbyterepresentingadifferentpieceoffromthatnode.Theprocess_datafieldstellsSCADAwheretoexpecteachpieceofdataineachpacket

psql-hlocalhost-p5432-d<databasename>-U<username>

bus_info:bustype:socketcanchannel:can0|vcan0|(oranyothersocketcanchannel)bitrate:125000

process_data:MOTOR:[STATUS,DUMMY2,TORQUE,DUMMY4,DUMMY5,DUMMY6,DUMMY7,DUMMY9]

Whilenotyetimplemented,SCADAshouldeventuallyimplementanSDOclientfortheabilitytopollthenetworkforadditionaldatanotincludedthedevicesPDOs.Forthatreason,aservicedatasectionisalsodefinedtogiveinstructionstothisclientaboutwhatdataitneedstopollandhowoften.

Theservice_datasectiondefinesalistofdatathatneedsbebemanuallypolledinthisway.Eachpieceofdataneedstohaveanode_id,index,subindex,andpoll_rate

NOTE:itisimportanttorememberthatnotallnodesonthebuswillsupportthis,buttheMotorControllerdefinitelywill.

Acompletedescriptionofalldatathatcanbeaccessedfromthemotorcontrollercanbefoundhere

FurtherreadingonServiceDataandProcessDatacanbefound

http://www.byteme.org.uk/canopenparent/canopen/sdo-service-data-objects-canopen/

and

http://www.byteme.org.uk/canopenparent/canopen/pdo-process-data-objects-canopen/

Calibration

Calibrationisconfiguredinaseparatepythonfilecalleduser_cal.pyinthecalibrationfolder.ACalibrationfunctionisdefinedusingthe@cal_functionfunctiondecorator,whichtakestwoarguments:

output:thenameofthedatabeinggeneratedarguments:alistofthedataneededasarguments

BecausethecalibratoroperatesonlyondatawithintheRediscache,outputandargumentsshouldbothcontainRediskeys.Thesecanbeanystringintheory,but,byconvention,consistofalllowercasecharacters,andareorganizedintohierarchiesviathe:character.(Forexample,allkeysthatstoredataabouttheTSItaketheformtsi:*)

Itisimportantthatthekeyswrittenintheargumentslistcorrespondtothosethosewritteninthedatasectionoftheconfig.yamlfile.Otherwise,thesorterandcalibratorwillnotagreeonwheretolookfordata.Itisalsoimportanttoensuretheoutputkeydoesnotconflictwithanyotherinputoroutputkeys.

Tohelpavoidkeycollisions,whenmultiplekeysareusedtorepresentthesamepieceofdata,buttransformedinsomeway,suchasinthecaseofalinearcalibrationorunitconversion,theyshouldbedistinguishedfromeachotherwiththeuseofasecond:followedbyamodifier.Onlythekeythatismeanttorepresentthedefaultformofthedatashouldomitthismodifier.Thegroupingofkeysinthiswayallowsforeasierqueryingoftheavailabledatabydownstreamprograms.Forexample,inthecaseoftheTSI,dataistransmittedasrawsensorvalues,andconvertedintousableunitsbycalibration

MOTOR-2:[MOTOR_TEMP,CONTROLLER_TEMP,DC_LINK_VOLTAGE,WARNING,CURRENT_DEMAND,TEST]MOTOR-3:[THROTTLE-byte0,THROTTLE-byte1,AUX,BRAKE,PHASEB_CURRENT,DUMMY5,DUMMY6,DUMMY7]MOTOR-4:[TORQUE_REGULATOR,FLUX_REGULATOR,VELOCITY]

TSI:[conditions,'mc_voltage:raw','voltage:raw','cooling_temp:raw','throttle:raw','flow_rate:raw','state:int','current:raw']

PACK1:[voltage,current,state_of_charge_01,state_of_charge_02,temp,'cells:temp:min','cells:temp:avg','cells:temp:max']

PACK2:[voltage,current,state_of_charge_01,state_of_charge_02,temp,'cells:temp:min','cells:temp:avg','cells:temp:max']

service_data:

Cell1Temp:node_id:2index:2011poll_rate:10

MotorTemp:node_id:1index:2010subindex:0poll_rate:0.5

functions.Thenoncalibrateddataisstoredwiththekeytsi:<data>:rawanditsconvertedformisstoredundertsi:<data>.

Examples

1.CelsiustoFahrenheit

2.VoltageandCurrenttoPower

3.MotorControllerVoltagefromSensorData

4.TSIstateenumeration

5.MotorControllerThrottle

Datageneratedbythesefunctionswillbewrittentothesamedatacachestructureastherest,andwillhaveanindexdeterminedbytheoutputargument,allowingittobeaccessedbyotherprogramsdownstream.

ViewingData

SCADAdatacanbeviewedbyanySCADAclient.Aclientisdefinedhereasanyprogrammeanttointerfacewiththe

#Convertsambienttempofpack1tofarenheitbecause#weliveinAmericagoddamnit@cal_function(output='pack1:temp:farenheit',arguments=['pack1:temp'])defpacktemp_farenheit(args):temp,*other=argstemp_faranheit=temp*(9/5)+32returntemp_faranheit

#CalculatestheTSpowerdrawinkW@cal_function(output='tsi:power',arguments=['tsi:voltage','tsi:current'])defts_power(args):voltage,current,*other=argspower=(voltage*current)/100returnpower

@cal_function(output='tsi:mc_voltage',arguments=['tsi:mc_voltage:raw'])defmc_voltage(args):mc_voltage_raw,*other=argsreturn(((mc_voltage_raw/255)*5)-1.28)*61

@cal_function(output='tsi:state',arguments=['tsi:state:int'])defstate(args):state_number,*other=args

ifstate_number==1:return'GLV-ON'elifstate_number==2:return'AIRS-CLOSED'elifstate_number==3:return'DRIVESETUP'elifstate_number==4:return'READYTODRIVESOUND'elifstate_number==5:return'DRIVE'else:return'STATEUNDEFINED'

@cal_function(output='motor:throttle',arguments=['motor:throttle:byte0','motor:throttle:byte1'])defthrottle(args):lsb,msb,*other=argsreturnmsb*256+lsb

SCADAservicesthroughtheTCPportsexposedbyRedisandPostgresql.Forreference,thesearebydefaultport6379and5432,respectively.Asseeninsomeoftheprevioussections,commandlinetoolsprovidedbybothRedisandPostgrescanbeusedascrudeSCADAclients,andareusefulforquickdebugging.However,mostpeoplewillwanttousesomethingmorevisual.

Inordertosupportataglancesystemmonitoringoverssh,acursesbasedclientisunderdevelopmentandincludedwithscada.Itcanberunwiththecommandscadatui.Unfortunately,itisnotyetconsideredcomplete.Thereisalotofpotentialforimprovement,andcouldprovequitepowerful,butlikealluserinterfaces,itismainlyamatterofpreference,andfutureteamsshoulddeveloptheclienttheyfindmostuseful.

Grafana

Grafanaisanopensourcetoolforgeneratinglivegraphsandotherdatavisualizationswithanemphasisonsystemmonitoringanddiagnostics.Itwasbuiltprimarilyforthewebdevelopmentcommunityasafrontendfortimeseriesdatabasesandmonitoringtoolslikeprometheusandgraphite,butitisflexibleenoughtobeusedforawiderangeofapplications,suchasSCADAsystems.

Thebestwaytogetasenseforwhatgrafanaishowtouseitistotryouttheirplaygroundserver,foundhere:

https://play.grafana.org

Youcanalsoreadmoreabouttheprojecthere:https://grafana.com/

orhere:https://github.com/grafana/grafana

SettingupGrafana

Grafanarunsasitsownserver,soitcanbeinstalledanywherewithaninternetconnection,andmostlikelyshouldn’tbeinstalledonthesameRaspberryPiasSCADA.ThereareanumberofwaysthataGrafanaservercanbeinstalled,includingthroughtheapt-getrepositoryorintoadockercontainerviatheofficialdockerimage:

https://grafana.com/docs/grafana/latest/installation/docker/

Onceinstalledlogintothegrafanaserverbynavigatingtotheipaddressofthedeviceyouinstalleditonwiththedefaultportof3000.

LogingIn

Theloginscreenwilllooksomethinglikethis.Enterthedefaultusernameandpasswordofadmin,admin.

AddingaDataSource

Clicktheconfigurationicononthelefthandsideofthescreen.Gotodatasources,andclicktheadddatasourcebuttonontheright.ChoosePostgreSQLfromthelist.

Fillouttheformwiththeappropriateinfo.ThehostshouldbetheipaddressofthecomputerSCADAisrunningon,followedbytheport5432.Therestwillhavebeeninfochosenwhensettingupthedatabase.Inmycase,Iusethedatabase“demo”withtheuserfsaeandpasswordcables.BesuretosetSSLModetodisable.

CreatingaPanel

Oncethedatabasehasbeenadded,youcanbegincreatingdashboards.Onthegrafanahomescreen,clicktheplusbuttononthelefthandsideofthescreenandselectcreatedashboard.Oncecreated,youcanaddtothedashboardanynumberofpanels.EachpanelcombinesanSQLquerywithanumberofadditionalsettingstocreateoneofseveralvisualizations.Someexamplesareincludedbelow:

DriveState

AdrivestateindicatorcanbemadefromtheSTATvisualization.ThefollowingSQLqueryisusedtogetthemostrecententryofthedrivestatesensorinthedatatable:

Inthesettingstab,setcalculationtolastandfieldstoallfields.TheresultingpanelshouldbeastringdescribingthemostrecentlyloggedDriveState.ThiscanbeverifiedbyrestartingtheTSIemulatororboard,andtakingitthroughthestartupsequence.

SELECTvalueFROMdataWHEREsensor_id='tsi:state'ORDERBYtimestampdescLIMIT1

ThrottleValue

Thegaugepanelisagreatwaytovisualizenumericaldata,likethrottlevalueorpackstateofcharge.ThefollowingSQLisusedtoqueryforthemostrecentlyloggedthrottledata

Notethatwehavetoexplicitlycastthevaluetoafloathere,becauseotherwiseitwillbetreatedasastring.Again,calculationshouldbesettolast.

SELECTtimestampastime,sensor_idasmetric,cast(valueasfloat)asvalueFROMdataWHERE$__timeFilter(timestamp)andsensor_id='tsi:throttle'

ThrottleHistory

Thesamequerycanalsobeusedtographahistoryofthrottlevaluesovertime,simplybychoosingthegraphvisualizerinsteadofthegauge.Herewecanseethatthethrottlevalueisfollowingasinusoid,whichmatcheswhatweexpectbasedonthemockbehavioroftheTSIboard.

FinalDashboard

Theresultofallthesepanelswillbeadashboardthatlookssomethinglikethis.Wenowcanlogintothegrafanaserverandseealivedisplayofthecar’sdrivestate,itscurrentthrottleinputasavoltage,andthehistoryofthatinputoveranygiventimerange.AstheteamaddsmoresensorsandCANnodes,itcanseamlesslyaddadditionalpanelsforviewingtheirdatainwhicheverwayismostappropriate.

author’snote

Grafanaisawesome.Itisflexible,easytouse,andcanbeintegratedintotheexistingSCADAsystemforfreeandatnocosttoperformanceorcomplexity.FranklyIregretnotdiscoveringitsooner,sinceitwouldhaveprovidedaneasyanswertomostofthequestionswehadwhileSCADAwasbeingdeveloped.Ihavepersonallybeeninvolvedinmany,manydiscussionsabouttherequirementsforaSCADAgraphicalclient,andtomyknowledge,thismeetseveryoneofthemandthensome.SomeofthefeatureswhichIdidnotdescribeaboveincludetimedgraphannotations,automaticalerting,exportingdatatoCSVfiles,embeddingpanelsinotherHTMLfilesandmore.Becauseitisopensource,anyfeaturethatisnotalreadyincludedwithGrafanawillmostlikelyexistasaplugin,andifnot,couldbeeasilywritten.

TheteamshouldstronglyconsideradoptingGrafanaasitsdefaultSCADAclientforalldatavisualizationandmonitoringpurposes.Aservercouldbesetupinthedynoroomandprovideliveinsightoftestsbeingperformedtoanynumberofcomputersonthenetwork.Itiseasyenoughtousethateachsubsystemcouldhaveitsown,dedicateddashboard,writtenbytheteaminchargeofit,anddisplayingonlythedataneededforagivenusecaseortest.