Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
SCADA-UserManualAuthor:ConnorWiniarczyk
IntroductionandPurposeSCADAisanacronymforSupervisoryControlAndDataAcquisition.Itisatermborrowedfromindustrialcontrolapplications,whereasingleserverisoftenusedtooverseelargeindustrialplantssuchasoilrefineriesandassemblylines.Ingeneral,SCADAsystemshavethreefunctions,theyacquiredatafromthenetworkofsensorsconnectedtotheplant,sendcontrolsignalstotheplant’sothersubsystems,andprovideaninterfaceforhumanstointeractwiththeplantbyviewingaggregateddataandissuingcommands.
TheLafayetteFSAEteamhasbeenworkingtodevelopaSCADAsystemthatcanbefullyintegratedintoitselectricvehicle,withthegoalofperformingalloftheabovethreefunctionsbothduringnormaloperationandthroughoutthevarioustestingandmaintenanceproceduresthatitwillundergo.Attimesinthepast,thissystemhasbeenreferredtoasVSCADA,shortforVehicleSCADA,todistinguishitfromtheindustrialsystemsdescribedabove,butforbrevitythisdocumentwillrefertoitsimplyasSCADA.
Thisdocumentisreferredtoasausermanual,butcouldperhapsbebetterdescribedasaninheritancemanual.Itincludesinformationabouthowtousethesoftware,butitalsoincludesabunchofotherinformationthatwouldn’tbeusefultoanenduser,butwouldbetosomeoneinheritingthisprojectwiththeintentionofexpandinguponorimprovingit.Itincludesinformationsuchastheoveralldesignandstructureoftheproject,whyitwasdesignedthewaythatitwas,whatIwasthinkingwhiledesigningit,andmypersonalthoughtsforhowitcanbeimprovedandexpandedon.Forthatreason,itismoreverbosethananordinaryusermanualmightbe,butIhopethatthereaderwillfindthisinformationuseful.
DesignOverview
CANNetwork
TheexistenceofaSCADAsystemimpliesanetwork.TheremustbecommunicationtakingplacebetweenSCADAandtheothersubsystemsforittosuccessfullyperformanyofitsresponsibilities.Thereareanumberofnetworkprotocolstochoosefrom,butthedefactostandardforautomotiveuseisCAN(ControllerAreaNetwork)andsothatiswhatisusedintheLafayetteFSAECar.
AgoodexplanationofCANcanbefoundhere:
https://www.csselectronics.com/screen/page/simple-intro-to-can-bus/language/en
Atthetimeofwriting,theCANNetworkcontainsbetween6and7nodes,dependingwhetheranexternalpcisneededtoconfigureandoperatethemotorcontroller.Theyare:
MotorControllerSCADABatteryPacks(1and2)TSIDashManExternalConfigurationPC
CANOpen
CANisaverylowlevelprotocol.Itdefinesawayforanodetobroadcastupto8bytesofdataandanID,butveryfewoftheotherthingsneededtoperformsophisticatednetworkoperations.Forthis,ahigherlevelprotocolneedstobedefinedontopofCAN,andcurrentlythereareseveralcompetingstandards.Thesehavebeendescribedastheequivalentofsomethinglikehttptothetcp/ipstack,addinganadditionallayerofabstractionforeasieruse.
Whileitisnotthemostpopular,theLafayetteFSAEteamhaschosentheCANOpenStandardforitsvehicleinordertomatchthatofthealreadypurchasedMotorController,whichincludesarichsetofCANOpenbasedtools.
CANOpencanalsobethoughtofasasubsetofCAN,meaningifanodeisaddedtothenetworkthatdoesnotcomplywiththeCANOpenstandard,thebehavioroftheothernodesisundefined.Forthisreason,itisveryimportantthateverynodeaddedtothenetwork,evenifitdoesnotintendtomakeuseofthefullsetofCANOpenfeatures,atleastcomplywithasubsetoftheprotocol.
Herearesomegoodresourcesforlearningmoreabouttheprotocol:
https://www.can-cia.org/canopen/
https://www.youtube.com/watch?v=DlbkWryzJqg
Asasupplementtotheseresources,abriefdescriptionofCANOpenisalsoprovidedhere:
CANOpencanbestbethoughtofasthesumofseveralcommunicationprotocols,coupledwithsomedefinedbehaviorforeachnode.Eachhasadifferentpurpose,name,andassociateacronym.Anodecanchoosetoimplementanysubsetoftheseprotocolsandbehaviors.
Theprotocolbeingusedisdeterminedbythefunctioncode,whichistransmittedintheidpartoftheCANframe.IntheCANOpenstandard,themessageidisformedbythesumofthefunctioncodeandthenodeid,thisistotakeadvantageoftheCANbehaviorofprioritizingmessageswithlowerids,soprotocolsthataremoreimportantaregivenlowerfunctioncodes.
NMT
NMTstandsforNetworkManagement.Itisamasterslaveprotocolusedtomanagethestateofthevariousnodesonthe
bus.AnynodewhichwishestointeractwiththeNMTprotocolasaslavemustimplementthefollowingfinitestatemachine.
Thisdeterminesthefunctionalstateofthenode.CANpacketscanbesentbythemastertomovethenodealongthisstatemachine,allowingittoperformactions,likesoftresets,emergencystops,andbootups.
SDOandOD
AnynodewhichwishestoexposeinternaldatatotheCANnetworkmustimplementadatastructureknowasanObjectDictionary,orOD.TheObjectDictionarymapseachpieceofdatatoanaddressconsistingofatwobyteindexfollowedbyaonebytesubindex.Certainaddressesarereservedforgeneraldatalikedevicenameanderrorregisters.ThemanufacturerofanodecanpublishinformationabouttheObjectDictionaryinafilecalledanElectronicDataSheet,orEDS,whichtakestheformofanINIfilethatisbothhumanandmachinereadable.
AnynodeonthenetworkcanaccessinformationfromtheObjectDictionaryofanothernodeusingtheServiceDataObject,orSDOprotocol.TheSDOpacketconsistsofabyteofmetadatafollowedbyathreebyteaddressand4bytesofdata.Thiscanbeusedtobothreadandwritedata,andcanbeusedtocontrolnodebehaviorinrealtime.Thisisthetechniqueusedinthedynoroomtospinthemotorandqueryitfordataliketemperatureandangularvelocity.
PDO
ProcessDataObjectsareaprotocolmeanttosupplementSDO’sbyprovidingadatatransfermethodwithahigherdatarateandlessoverhead.Theyaremeanttobethestandardforhighvolumeinternodedatatransferduringnominaloperationofthenetwork,andhavetheaddedbenefitofbeingmuchsimplertoimplementthanSDO.AnodethatchoosestoimplementonlythesmallestpossiblesubsetoftheCANOpenprotocolwillmostlikelyimplementaPDO.
PDO’sworkonaproducer/consumer,orbroadcast/subscribemodelofcommunication,whereoneormoreCANpackets
aresentatregularintervals,eachcontaining8bytesofdatawithastructureagreeduponbeforehand.Anynodeonthenetworkcansubscribetothesepacketsandupdatetheirbehavioraccordingly.Boththebroadcastandsubscribebehavior(calledtheTransmitPDOandReceivePDOrespectively)canbeconfiguredbydedicatedaddressesintheObjectDictionary.
Hardware
SCADArunsonaRaspberryPithatismountedinsideoftheCarManenclosure.Attachedtoitisa“hat”thatenablesCANcommunicationandprovidesapassthroughforthePi’sGPIOpins.Together,theseareconnectedtotheGLVboardviaasetof3cables:AmicroUSBcableforpower,DB9serialcableforCANhighandlowsignals,andahomemaderibboncableforcertainGPIOpins.AdditionalcablescanberuntoconnectthePitootherpartsofthecarand/orDynoRoom.Theseinclude:
RibbonCabletotheCarManDisplayHDMItototheDynoRoomMonitorUSBtoexternalkeyboardandmouseUSBtothedynamometer
author’snote
ThethreecablesconnectingtheSCADAPitotheGLVboardareabaddesigndecisionandshouldberemovedatthenextpossibleopportunity.WiththeexceptionoftheDB9,thesecablesandtheirconnectorsarenotratedforautomotiveuse,andcouldeasilybecomedisconnectedduringheavyvibrationsandrenderSCADAinoperable.Inaddition,theadjacencyoftheSCADAPiandtheGLVBoardwithintheCarManenclosuremakethesecablesappearcomicallylarge,forcingspacetobeusedforcableroutingthatcouldotherwisebetakenupbysomethingmoreuseful.ItwouldbeafarbettersolutiontointegratetheSCADAPiandtheCANhatintotheGLVboard.
RaspberryPioffersacomputemodule,whichisthecomputerastheordinaryRaspberryPi3,butwithsomeperipheralsremoved,andcondensedintoamuchsmallerformfactorwithasinglecardedgeconnectorasitsonlyinterface.IhighlyrecomendthatfutureteamsreplacetheexistingSCADAPiwithoneofthese,withexternalcomponentsliketheCANinterfaceandethernetjackintegratedintotheGLVboard.
https://www.raspberrypi.org/products/compute-module-3-plus/
Software
ThemostimportantthingtounderstandaboutthedesignofSCADAinitscurrentformisthatitisnotasingleprogram,butratherasystemofprogramswhicharemadetorunconcurrentlyandinterfacewitheachother.Theseprogramstryasmuchaspossibletoabidebytheunixphilosophyofsoftwaredevelopment,whichisinessence,towriteprogramsthatareextremelylimitedinscope,andthatinteractwellwitheachother,throughstandard,welldefinedinterfaces.ThisisopposedtothewaythatSCADAhasbeenwritteninthepast,whereasinglelarge,integratedprogramismadetoimplementallthefeaturesrequiredbythesystem.Therearesomedrawbackstothesmallandmodularapproach,whichwillbeaddressedlater,butalsoalotofadvantages.Theyincludethefollowing:
Featurescanusuallybeaddedwithouthavingtomodifyexistingcode,thiscanhelptomitigatethesquarelawofaddingfeaturestothings.
SCADAcanbewritteninmultipledifferentlanguages,sothebestlanguagecanalwaysbechosenforagivenpurpose,withouthavingtomakecompromises.
Whenthepurposeofaprogramisgeneralenough,existingopensourcesoftwarecanbeusedinsteadofwritinganinhousesolution.
Theindividualcomponentsofthesystemareeasytounderstandandreasonabout.
Whenproblemsarise,theyareeasytoisolateanddiagnose.
TheprogramsmakingupSCADAandtheirstructurearedescribedinthefollowingdiagram.Theyareorganizedintoapipeline,withdataflowingfromoneservicetothenextinawelldefinedway.Eachservicehasaspecificrollineither
organizingortransformingthedatasothatitcanbemadesenseofbytheuser.
CANbus User
Calibrator
Sorter
Clients
Logger
Redis Postgres
SCADADataAquisitionPipelineDataflowsfromtheCANinterfacetotheuserthroughaseriesofservicesandstoragemediums.Eachofwhichservesadistinctpurpose.
Whatfollowsisashortdescriptionofeachserviceanditsfunction.
Sorter
ThesorterlistenstotheCANbusforincomingmessages,uponreceivingamessage,itchecksitsIDandStructureagainstaconfigfiletogenerateanassociatedsetofkeyvaluepairs.ItthenwritesthesekeyvaluepairsintotheRediscacheandsendsamessagetothecalibratorthatnewdatahasbeenreceived.ThesorteristheonlyservicethatisawareofCANorCANOpen,meaningSCADAcanbemadetoworkwithadifferentnetworkingprotocol,ormorethanone,simplybyswitchingoutthesorterserviceoraddinganadditionalone,withoutaffectinganyservicesdownstream.
Calibrator
ThecalibratorisresponsiblefortranslatingtherawdatareadbythesorterfromtheCANbusintoamorehumanreadableform.Thisisusuallyacaseoflinearcalibrationofrawsensordata,suchaswiththeTSInode,ortherecombiningofdatathatwastransmittedasmultiplebytes,likewiththeThrottlevaluereportedbyMotorController.However,itcanbemadetosupportcalculationsofarbitrarycomplexity,suchasforexample,thecalculationofpowerfromagivenvoltageandcurrent,theaveragingofmultiplesensors,filtering,differentiation,andsoon.Allcalibratoroperationsaredefinedbytheuserasfunctionsintheuser_cal.pyfile.Oncedataiscalculated,itiswrittenbackintotheRedisserverasadifferentkey.
Logger
TheloggertakesthedatagatheredbythesorterandcalculatedbythecalibratorandlogsasubsetofitintothePostgresqldatabaseatregularintervals.Thesubsetofdatatobeloggedisdefinedinconfig.yamlfile.
Itsamplesthedataatregularintervals,andperformssomebasiclogictolimitthedatabasetoonlystoringusefuldata.Ingeneral,itittriestoonlylogdatathatischanging,soasnottofloodthedatabasewithconstantvalues.Theloggerwritesallloggeddatatoasinglesqltable,calleddata,withthefollowingcolumns:
idsensor_idvaluetimestamp
Inkeepingwiththeunixtradition,alldataisstoredinthesamewayastext.Programsreadingfromthedatabaseareresponsiblefortranslatingthedataintoitsexpectedtype.Acompaniontable,calledsensorsisincludedinthedatabasetostoremetadataabouteachsensor.Ithascolumns:
idredis_key(equivalenttosensor_id,oneoftheseshouldprobablyberenamed)display_namedatatypeunit
Redis
Forthemostpart,programscommunicatewitheachotherviaRedis.Redisisanopensource,inmemorydatabasethatstoresdataaskeyvaluepairs,italsofunctionsasabasiccommunicationsbusforsendingsimpleinter-servicemessages.
https://redis.io/
Postgres
Ifaprogramneedstowritenon-volatiledata,itdoessousinganSQLdatabaseviaaPostgresserver.
https://www.postgresql.org/
Concurrency
Concurrencyishandledbytheoperatingsystemviasystemd.Allprogramswhicharemeanttoberunasaservicehaveanassociated.servicefile,andcanbemanagedwiththesystemctlinterface.Allservicesaresinglethreadedandshouldconsistofonlyoneupdateloop,delaystatementsareinsertedintotheloopinordertoreleasethecpuforusebyotherservices.Agoodexampleofanupdateloopforaservicethatrespondstoredismessagesisincludedbelow:
https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units
Clients
BoththeRedisandPostgresserversexposeTCPports,andanysoftwarecapableofinteractingwiththeseTCPportscanbeconsideredaSCADAclient.TheseclientscanberuneitherlocallyontheSCADAPiorremotelyoveranetwork.TherearealreadyanumberofSCADAclientstochoosefrom,andthebestoneforanygivenapplicationwilldependlargelyoncircumstanceandpreference.
Onepotential3rdpartyclientthatcomesveryhighlyrecommendedisGrafana.Grafanaisanopensourcedataviewingandmonitoringtoolthatspecializesintheconstructionofawiderangeofdatavisualizationssuchasgraphs,gauges,statistics,andlogsbasedonanumberofpotentialdatasources.ItsupportsPostgresqloutoftheboxandisextremelyeasytosetup.Italsohasarichcommunityofplugindevelopers.
whileTrue:message=p.get_message()ifmessage:update()else:time.sleep(0.1)
https://grafana.com/
Otherclientsareindevelopmentaswell,includingacommandlinetoolforbasicmanagement,acursesbasedmonitoringtoolforeasytestingoverssh,aJavabasedtoolforgraphgeneration,andaTkinterbasedtoolforgraphicallivemonitoringandmanagement.
FutureGoals
Atthemoment,SCADAisastrictlypassivesysteminitsrelationshiptotheCANbus.TheservicesdescribedaboveimplementapuredataacquisitionsystemwhichlistenstotheCANbusandpresentstheincomingdatatotheuser,butdoesnothavetheabilitytotalkonthatbustoissuecommandsorconfigurethenetworkforcertainbehaviors.ThismeansthatSCADAis,bydefinition,incomplete.Thereneedstobeawaytoperformsupervisorycontrolfunctionsbytalkingbacktothenetwork.
BecausedataacquisitionisthemostimportantpartoftheFSAEteam’sneedforaSCADAsystem,theseweretheservicesthatwerebuiltfirst,butmoreworkisrequiredtoturnthisintoacompleteSCADAsystem.Inaddition,theSCADAPIneedstoperformsomesecondaryservicesthatarenotstrictlypartoftheSCADAsystem.AnexampleofamorecompleteSCADAdesignthatincludesnon-implementedfeaturesisshownbelow.
CANbus
User
Calibrator
Sorter
Clients
Logger
Redis Postgres
SCADABlockDiagramShowsanexampleofacompletedesignofSCADA,includingfeatureswhichhavenotyetbeenimplemented.Featureswhichareplannedbutnotimplementedattimeofwritingaredrawninred.
Watcher
InstructionParser
GLV-CANInterface
GLVHardware
SubsystemEmulators
Alistoftheadditionalservicesdepictedandadescriptionoftheirfunctionisincludedbelow:
InstructionParser
Theinstructionparserwouldserveasacounterparttothesorter.InsteadofreadingfromtheCANbus,itwouldwritetoit,andserveasthebridgethroughwhichtherestofSCADAcouldsendcontrolsignalstothecar’sothersubsystems.Inorder
tokeeptheotherSCADAservicesagnosticaboutexactCANmessagesyntax,atextbasedformatshouldbeagreeduponforrepresentingcontrolsignalsthatcansentovertheCANnetwork.TheinstructionparserwouldthenhavethesoleresponsibilityofparsingthisformatintoCANmessages.
AproposalforsuchaninstructionsetisincludedwithSCADAdocumentationandcanbereadhere:
https://raw.githubusercontent.com/Lafayette-FSAE/scada/master/documentation/sparkyscript.pdf
Watcher
Theinstructionparserwould,bydefault,receivemostofitsinstructionsfromclientprograms,whichwouldinturnreceiveinstructionsfromtheUser,whowouldmakedecisionsaboutwhichcommandstosendbyviewingtheincomingdata.Thisformsafeedbackloopfordoingcertainkindsmanualsystemcontrol.However,therewillmostlikelybeaneedforanadditional,automaticfeedbackloopwhichcantakeplaceatahigherfrequencyandwithoutuserintervention.
Intheory,aclientcouldbewrittentoperformsuchatask,butitwouldnotcontinuetorunafteritwasdisconnected.Instead,adedicatedserviceshouldbewritten,whichregularlypollstheRedisserverforthestateofthecar,andmakesdecisionsaboutwhatcommandstosendbasedonthisdata.Itcouldbeconfiguredasalistofcondition,instructionpairs,andwouldbeusefulforthingslikeopeningthesafetyloopduringemergencyconditions.
GLV-CANInterface
TheGLVboardcontainsanumberofsensorsandcontrolvectorsthatshouldideallybeexposedtotherestofthecarusingthestandardCANOpeninterface.Thiscould,intheory,bedonebyadedicatedmicrocontroller,likeisdonewiththecar’sothersubsystems,buttheproximityoftheSCADAPitothesesensorshasmeantthat,forbetterorforworse,ithasabsorbedtheseresponsibilities.
Nevertheless,theGLV-CANinterfaceshouldbethoughtofasdistinctfromSCADA,andwrittenasitsownprogramthatonlyinterfaceswithSCADAthroughtheCANbus.Thishasthedrawbackofrequiringsomeextrastepsinthecommunicationprocess,butitallowsothernodesonthenetworktoaccessdatafromtheGLVsystem.Inaddition,ithelpstokeepSCADAimplementationmore“pure”inthesensethatitdoesnotrequiretheneedforbuiltinexceptionsandspecialbehaviorbasedonthemethodthedatawasobtained.PreviousimplementationsofSCADAwereriddledwithsuchexceptions,andtheywereacommonsourceofcomplaints.
SubsystemEmulators
Intheabsenceofafullyintegratedsystem,testingtheCANinterfaceofagivensubsystemcanbeadifficulttask.Forexample,youmaywantoneofthebatteriestobehavedifferentlybasedonthestateoftheotherone,butonlyhaveconstructedasingleprototype.TheDashMansystemmaywanttotestthewayitrespondstoacertainTSIstate,butthelastTSIboardmayhavebeenfrieddotoawiringerror.Andofcourse,youmaybeforcedbyunforeseencircumstancestodevelopSCADAandallothersubsystemsremotelyandwhileforbiddentocomewithinsixfeetofthelab,thecar,oreachother.
Forsuchevents,the2020teamhasbegunworkonaseriesofSubsystemEmulators,whicharemeanttoberuneitherontheSCADAPioraseparateCANconnectedcomputer,andmimictheCANinterfaceofeachsubsystem.Eachemulatorshouldmimicitssubsystem’sObjectDictionary,SDOserver,andTransmitPDOmessages.Whilenotnecessary,additionalhardwarelikeLEDsandbuzzerswouldalsobefun.
ExtraIdeas
SDOs
TheinstructionparserwouldalsoberequiredforreadingSDOdatafromthenetwork.ThisisbecausetheSDOoperatesontheclientservercommunicationmodel,ratherthantheproducerconsumermodelusedbyPDOdata.
HowtoUse
Installation
Prerequisites
SCADAassumesitisbeingrunonadebianbasedlinuxdistribution.Thiswillalmostalwaysberaspbian,butithasalsobeentestedonapuredebianserver.Thisshouldalsoworkonsomethinglikeubuntu,butthishasnotbeentested.SCADAwillnotworkonaWindowsorMac,andmostlikelyneverwill.Thisisbydesign,asSCADAtakesadvantageofalargenumberofassumptionsaboutitsenvironment.
ItisrecommendedtosetupadedicateddevelopmentserverforSCADAusingaspareraspberrypiandideallysomethingresemblingtheGLVhardware.ThiscouldbeexpandedovertimeintoincludeamockCANbusforintegrationtestingwithothersubsystems.
BecausetheLafayettenetworkcanbedifficulttonavigate,itisrecommendedthatthisbebypassedusingeitheradedicatedphysicalnetworkorvpn.Thiscouldbesetupinsuchawayastopreventipaddresschangesandtoenableoffsitework.
InstallScript
Oneofthedrawbacksofthesmallandmodularapproachtosystemdevelopmentisthatinstallationbecomesabitmoretricky.Tohelpmitigatethis,aninstallscriptisincludedwithSCADAwhichismeanttohelpautomatetheprocess.
Ideally,afullinstallationofSCADAcouldbeperformedwiththefollowingshellcommands:
However,becauseSCADAisstillrelativelynewsoftware,theinstallscriptislikelytofailinsomeenvironments,requiringmanualintervention.Inaddition,updateswillneedtobemadetotheinstallscriptperiodicallyasnewfeaturesareaddedtoSCADA.Tothatend,anexplanationoftheinstallscriptandwhatitdoesisprovidedhere.
Thissectionusesapt-gettoinstall3rdpartyprogramsanddependencies.NewdependenciescanbeaddedtoSCADAbyappendingapt-getcallstothislist.
Next,thepythonpackagemanagerisusedtoinstallpythonspecificdependencies.
ThePi’sCANinterfaceiscreatedandturnedon.ThisiscurrentlyconfiguredtouseavirtualCANinterfacefordevelopmentandtesting,butcanbeswitchedtotherealCANbusbyreplacingallinstancesofvcanwithcan.
$gitclonehttps://github.com/Lafayette-FSAE/scada$cdscada$sudobashinstall
apt-getinstallpython3apt-getinstallpython3-pipapt-getinstallredis-serverapt-getinstallcan-utils
#installpythondependenciespip3installpython-canpip3installredispip3installblessedpip3installpsycopg2-binary
#makesurevirtualcanbusissetupfortestingmodprobevcaniplinkadddevvcan0typevcaniplinksetupvcan0
#makebinaryfilesexecutablechmod+xinstallchmod+xscadachmod+xsorter/sorter.py
Fileswhichneedtobeexecutablearemadesoexplicitlywiththechmodcommand.Thisincludesscriptswhicharerunasservices,thescadacliinterfaceandthisinstallscript.
Copyexecutablefilesintoaknowndirectory.Theexactdirectorychosenissortofarbitrary,solongasitmatchesthedirectorywritteninthe.servicefiles./usr/binisagoodchoicebecauseitisthestandardforuserinstalledbinaryfilesanditisalreadyinthePATHvariable.
Thisalsocopiesfilesintoaknowndirectory,thistimethenonexecutablefiles,andtothe/usr/etc/scadadirectory.Againthiswaschosenarbitrarily.Thisiswheretheserviceswilllookforthingslikecustompythonlibrariesandconfigfiles.
Finallycopythe.servicefilesintoaplacewheresystemdcanfindthem.Thisdirectoryisnotarbitrary,andmustbe/etc/systemd/system
VerificationandTroubleshooting
IfSCADAwasinstalledcorrectly,youshouldnowhaveacommandlinehelpertoolthatcanbeusedforbasicsystemmanagement.Verifythisbytypingscada--helpintothecommandlineandrunningit.Youshouldseeahelpmessagedescribingthepossiblesubcommandsthatcanbeused.
Thestatusofeachservicecanbeverifiedusingthecommandscadastatus,whichisequivalenttorunningsystemcmtlstatus<service>foreachserviceincludedinscada.Youshouldseeanentryforcalibrator.service,logger.service,andsorter.service,ifallisworkingtheyshouldallbegreenandhaveastatusofactive.Ifoneormoreserviceismissing,theywereprobablynotinstalledcorrectly,youcanlookfortheirservicefilesinthedirectory/etc/systemd/system.
Ifoneormoreoftheseservicesisinthefailedstate,thatusuallyindicatesthattherewasanerrorrunningtheirassociatedpythonscript.Theseerrorscanusuallybefoundbycheckingthelogs,whichareaccessedwiththecommandsudoscadalogs.
Servicescanalsobetroubleshootedbystoppingthemandrunningthescriptmanually.Thecommandsudoscadastop<service>willstopagivenservice,andsudoscadastart<service>canbeusedtorestartit.Anyservicecanberunmanuallybyinvokingitspythonscriptfiledirectly.Forexamplepython3~/scada/sorter/sorter.py
Ifallservicesarepresentandrunningwithouterrorsyoushouldbegintestingthesystemagainstsomerealcandata.Ifnoexternalcanbusisavailable,theincludedsubsystememulatorswillgenerallybetheeasiestway.Thesedonothaveassociated.servicefilesandmustberunmanually.Navigatetothe~/scada/emulatordirectoryandrunpython3main.py.Forconvenience,itisbesttorunthisinatmuxwindowsothatothertaskscanbeperformedwiththisprograminthebackground.
Thisisaverygoodguideabouthowtosetupandusetmux.https://www.hamvocke.com/blog/a-quick-and-easy-guide-to-tmux/
chmod+xcalibrator/calibrator.pychmod+xlogger/logger.py
#copybinaryfilesto/usr/bincpscada/usr/bin/scadacpsorter/sorter.py/usr/bin/scada_sorter.pycpcalibrator/calibrator.py/usr/bin/scada_calibrator.pycplogger/logger.py/usr/bin/scada_logger.py
#createaworkspaceandcopyimportantfilesintoitmkdir-p/usr/etc/scadarm-rf/usr/etc/scada/utilscp-rutils/usr/etc/scada/utilsrm-rf/usr/etc/scada/configcp-rconfig/usr/etc/scada/configcp./install/usr/etc/scada
#copyservicefilesforsystemdcpsorter/sorter.service/etc/systemd/systemcpcalibrator/calibrator.service/etc/systemd/systemcplogger/logger.service/etc/systemd/system
Oncetheemulatorisupandrunning,youshouldtesttoseeifitisgeneratingCANtraffic.Thebestwaytodothisisbyusingthecandumpcommandthatisbuiltintocan-utils.Typethecommandcandumpvcan0andrunit,youshouldseearegularstreamofcanpackets.IftheTSIemulatorisrunning,youwillseearegularpacketwiththecanidof183,thisisthesumofthefunctioncode0x180,indicatingthatitisaPDOpacket,andthenodeidof3,indicatingthatitiscomingfromtheTSI.IfyouareusingadifferentCANinterface,youwillneedtoswitchthevcan0argumenttothecorrectone.Thiswillbecan0inthelivesystem.
ThesorterandcalibratorcanbeverifiedbylookingattheRediscachedirectly.Typethecommandredis-cli--scantogetaprintoutofallactivekeys.Redisdatawrittenbythesorterandcalibratorwillexpireafteraperiodof10seconds,soalackofdatausuallyindicatesthatoneorbothofthemhavestopped.Thevalueofoneofthesefieldscanbereadusingthecommandredis-cliget<key>.Agoodtestistoquerythetsi:throttlekey,itshouldbechangingovertime.
Theredis-cliisaverypowerfultool,andiseffectivelyitsownSCADAclient.Moreinfoabouthowtousetheredis-clitoolcanbefoundhere:
https://redis.io/topics/rediscli
Thelaststepistoverifythattheloggeriswritingdataintothedatabasecorrectly.Theeasiestwaytodothisisbyqueryingthedatabasewiththepsqlcommand.
Runningthecommand
ThedatabasenameandusernamewillbetheoneschosenwhilesettingupPostgres.Thiswillpromptyouforapassword,whichisalsochosenduringthesetupprocess.Ifsuccessful,youwillbeenteredintoaREPLwhereyoucanexecuteSQLcommandsagainstthedatabase.
Typing\dtwillshowthelistoftables,andshouldincludeatleastadataandsensorstable.Runningthecommandselect*fromdata;willshowthecontentsofthedatatable.Confirmthattheloggerisworkingcorrectlybycheckingtheendofthetabletoseeifnewdataisbeingwritten.
Configuration
WhileSCADAconsistsofseveraldistinctprograms,forthesakeofconvenience,theyallreadfromthesamesetofconfigurationfiles.Atthemoment,therearetwo,bothofwhichcanbefoundintheconfigdirectory.
config.yamlisageneralpurposeconfigfilewhichhandlesthemajorityofoptions.user_cal.pyisapythonfilespecifictothecalibratorwhichdefinesthecalibrationfunctionsitwillusetomapdataasitcomesin.TogethertheymakeupthefullsetofconfigurationoptionsforSCADA.
Thefollowingsectionsexplaintheconfigurationoptionsinfurtherdetail.
BusInfo
ThebusinfosectiongivesthesorterinformationabouttheCANbus.IfyouarerunningSCADAonasystemwithavirtualCANinterface,setchanneltovcan0.Asitiscurrentlyconfigured,theraspberrypiintheDynoroomtalkstotherestofthenetworkonthecan0channelofsocketcanwithabitrateof125000
Data
Mostdataissentoverthebusin8byteCANpacketscalledProcessDataObjects,orPDOs,witheachbyterepresentingadifferentpieceoffromthatnode.Theprocess_datafieldstellsSCADAwheretoexpecteachpieceofdataineachpacket
psql-hlocalhost-p5432-d<databasename>-U<username>
bus_info:bustype:socketcanchannel:can0|vcan0|(oranyothersocketcanchannel)bitrate:125000
process_data:MOTOR:[STATUS,DUMMY2,TORQUE,DUMMY4,DUMMY5,DUMMY6,DUMMY7,DUMMY9]
Whilenotyetimplemented,SCADAshouldeventuallyimplementanSDOclientfortheabilitytopollthenetworkforadditionaldatanotincludedthedevicesPDOs.Forthatreason,aservicedatasectionisalsodefinedtogiveinstructionstothisclientaboutwhatdataitneedstopollandhowoften.
Theservice_datasectiondefinesalistofdatathatneedsbebemanuallypolledinthisway.Eachpieceofdataneedstohaveanode_id,index,subindex,andpoll_rate
NOTE:itisimportanttorememberthatnotallnodesonthebuswillsupportthis,buttheMotorControllerdefinitelywill.
Acompletedescriptionofalldatathatcanbeaccessedfromthemotorcontrollercanbefoundhere
FurtherreadingonServiceDataandProcessDatacanbefound
http://www.byteme.org.uk/canopenparent/canopen/sdo-service-data-objects-canopen/
and
http://www.byteme.org.uk/canopenparent/canopen/pdo-process-data-objects-canopen/
Calibration
Calibrationisconfiguredinaseparatepythonfilecalleduser_cal.pyinthecalibrationfolder.ACalibrationfunctionisdefinedusingthe@cal_functionfunctiondecorator,whichtakestwoarguments:
output:thenameofthedatabeinggeneratedarguments:alistofthedataneededasarguments
BecausethecalibratoroperatesonlyondatawithintheRediscache,outputandargumentsshouldbothcontainRediskeys.Thesecanbeanystringintheory,but,byconvention,consistofalllowercasecharacters,andareorganizedintohierarchiesviathe:character.(Forexample,allkeysthatstoredataabouttheTSItaketheformtsi:*)
Itisimportantthatthekeyswrittenintheargumentslistcorrespondtothosethosewritteninthedatasectionoftheconfig.yamlfile.Otherwise,thesorterandcalibratorwillnotagreeonwheretolookfordata.Itisalsoimportanttoensuretheoutputkeydoesnotconflictwithanyotherinputoroutputkeys.
Tohelpavoidkeycollisions,whenmultiplekeysareusedtorepresentthesamepieceofdata,buttransformedinsomeway,suchasinthecaseofalinearcalibrationorunitconversion,theyshouldbedistinguishedfromeachotherwiththeuseofasecond:followedbyamodifier.Onlythekeythatismeanttorepresentthedefaultformofthedatashouldomitthismodifier.Thegroupingofkeysinthiswayallowsforeasierqueryingoftheavailabledatabydownstreamprograms.Forexample,inthecaseoftheTSI,dataistransmittedasrawsensorvalues,andconvertedintousableunitsbycalibration
MOTOR-2:[MOTOR_TEMP,CONTROLLER_TEMP,DC_LINK_VOLTAGE,WARNING,CURRENT_DEMAND,TEST]MOTOR-3:[THROTTLE-byte0,THROTTLE-byte1,AUX,BRAKE,PHASEB_CURRENT,DUMMY5,DUMMY6,DUMMY7]MOTOR-4:[TORQUE_REGULATOR,FLUX_REGULATOR,VELOCITY]
TSI:[conditions,'mc_voltage:raw','voltage:raw','cooling_temp:raw','throttle:raw','flow_rate:raw','state:int','current:raw']
PACK1:[voltage,current,state_of_charge_01,state_of_charge_02,temp,'cells:temp:min','cells:temp:avg','cells:temp:max']
PACK2:[voltage,current,state_of_charge_01,state_of_charge_02,temp,'cells:temp:min','cells:temp:avg','cells:temp:max']
service_data:
Cell1Temp:node_id:2index:2011poll_rate:10
MotorTemp:node_id:1index:2010subindex:0poll_rate:0.5
functions.Thenoncalibrateddataisstoredwiththekeytsi:<data>:rawanditsconvertedformisstoredundertsi:<data>.
Examples
1.CelsiustoFahrenheit
2.VoltageandCurrenttoPower
3.MotorControllerVoltagefromSensorData
4.TSIstateenumeration
5.MotorControllerThrottle
Datageneratedbythesefunctionswillbewrittentothesamedatacachestructureastherest,andwillhaveanindexdeterminedbytheoutputargument,allowingittobeaccessedbyotherprogramsdownstream.
ViewingData
SCADAdatacanbeviewedbyanySCADAclient.Aclientisdefinedhereasanyprogrammeanttointerfacewiththe
#Convertsambienttempofpack1tofarenheitbecause#weliveinAmericagoddamnit@cal_function(output='pack1:temp:farenheit',arguments=['pack1:temp'])defpacktemp_farenheit(args):temp,*other=argstemp_faranheit=temp*(9/5)+32returntemp_faranheit
#CalculatestheTSpowerdrawinkW@cal_function(output='tsi:power',arguments=['tsi:voltage','tsi:current'])defts_power(args):voltage,current,*other=argspower=(voltage*current)/100returnpower
@cal_function(output='tsi:mc_voltage',arguments=['tsi:mc_voltage:raw'])defmc_voltage(args):mc_voltage_raw,*other=argsreturn(((mc_voltage_raw/255)*5)-1.28)*61
@cal_function(output='tsi:state',arguments=['tsi:state:int'])defstate(args):state_number,*other=args
ifstate_number==1:return'GLV-ON'elifstate_number==2:return'AIRS-CLOSED'elifstate_number==3:return'DRIVESETUP'elifstate_number==4:return'READYTODRIVESOUND'elifstate_number==5:return'DRIVE'else:return'STATEUNDEFINED'
@cal_function(output='motor:throttle',arguments=['motor:throttle:byte0','motor:throttle:byte1'])defthrottle(args):lsb,msb,*other=argsreturnmsb*256+lsb
SCADAservicesthroughtheTCPportsexposedbyRedisandPostgresql.Forreference,thesearebydefaultport6379and5432,respectively.Asseeninsomeoftheprevioussections,commandlinetoolsprovidedbybothRedisandPostgrescanbeusedascrudeSCADAclients,andareusefulforquickdebugging.However,mostpeoplewillwanttousesomethingmorevisual.
Inordertosupportataglancesystemmonitoringoverssh,acursesbasedclientisunderdevelopmentandincludedwithscada.Itcanberunwiththecommandscadatui.Unfortunately,itisnotyetconsideredcomplete.Thereisalotofpotentialforimprovement,andcouldprovequitepowerful,butlikealluserinterfaces,itismainlyamatterofpreference,andfutureteamsshoulddeveloptheclienttheyfindmostuseful.
Grafana
Grafanaisanopensourcetoolforgeneratinglivegraphsandotherdatavisualizationswithanemphasisonsystemmonitoringanddiagnostics.Itwasbuiltprimarilyforthewebdevelopmentcommunityasafrontendfortimeseriesdatabasesandmonitoringtoolslikeprometheusandgraphite,butitisflexibleenoughtobeusedforawiderangeofapplications,suchasSCADAsystems.
Thebestwaytogetasenseforwhatgrafanaishowtouseitistotryouttheirplaygroundserver,foundhere:
https://play.grafana.org
Youcanalsoreadmoreabouttheprojecthere:https://grafana.com/
orhere:https://github.com/grafana/grafana
SettingupGrafana
Grafanarunsasitsownserver,soitcanbeinstalledanywherewithaninternetconnection,andmostlikelyshouldn’tbeinstalledonthesameRaspberryPiasSCADA.ThereareanumberofwaysthataGrafanaservercanbeinstalled,includingthroughtheapt-getrepositoryorintoadockercontainerviatheofficialdockerimage:
https://grafana.com/docs/grafana/latest/installation/docker/
Onceinstalledlogintothegrafanaserverbynavigatingtotheipaddressofthedeviceyouinstalleditonwiththedefaultportof3000.
LogingIn
Theloginscreenwilllooksomethinglikethis.Enterthedefaultusernameandpasswordofadmin,admin.
AddingaDataSource
Clicktheconfigurationicononthelefthandsideofthescreen.Gotodatasources,andclicktheadddatasourcebuttonontheright.ChoosePostgreSQLfromthelist.
Fillouttheformwiththeappropriateinfo.ThehostshouldbetheipaddressofthecomputerSCADAisrunningon,followedbytheport5432.Therestwillhavebeeninfochosenwhensettingupthedatabase.Inmycase,Iusethedatabase“demo”withtheuserfsaeandpasswordcables.BesuretosetSSLModetodisable.
CreatingaPanel
Oncethedatabasehasbeenadded,youcanbegincreatingdashboards.Onthegrafanahomescreen,clicktheplusbuttononthelefthandsideofthescreenandselectcreatedashboard.Oncecreated,youcanaddtothedashboardanynumberofpanels.EachpanelcombinesanSQLquerywithanumberofadditionalsettingstocreateoneofseveralvisualizations.Someexamplesareincludedbelow:
DriveState
AdrivestateindicatorcanbemadefromtheSTATvisualization.ThefollowingSQLqueryisusedtogetthemostrecententryofthedrivestatesensorinthedatatable:
Inthesettingstab,setcalculationtolastandfieldstoallfields.TheresultingpanelshouldbeastringdescribingthemostrecentlyloggedDriveState.ThiscanbeverifiedbyrestartingtheTSIemulatororboard,andtakingitthroughthestartupsequence.
SELECTvalueFROMdataWHEREsensor_id='tsi:state'ORDERBYtimestampdescLIMIT1
ThrottleValue
Thegaugepanelisagreatwaytovisualizenumericaldata,likethrottlevalueorpackstateofcharge.ThefollowingSQLisusedtoqueryforthemostrecentlyloggedthrottledata
Notethatwehavetoexplicitlycastthevaluetoafloathere,becauseotherwiseitwillbetreatedasastring.Again,calculationshouldbesettolast.
SELECTtimestampastime,sensor_idasmetric,cast(valueasfloat)asvalueFROMdataWHERE$__timeFilter(timestamp)andsensor_id='tsi:throttle'
ThrottleHistory
Thesamequerycanalsobeusedtographahistoryofthrottlevaluesovertime,simplybychoosingthegraphvisualizerinsteadofthegauge.Herewecanseethatthethrottlevalueisfollowingasinusoid,whichmatcheswhatweexpectbasedonthemockbehavioroftheTSIboard.
FinalDashboard
Theresultofallthesepanelswillbeadashboardthatlookssomethinglikethis.Wenowcanlogintothegrafanaserverandseealivedisplayofthecar’sdrivestate,itscurrentthrottleinputasavoltage,andthehistoryofthatinputoveranygiventimerange.AstheteamaddsmoresensorsandCANnodes,itcanseamlesslyaddadditionalpanelsforviewingtheirdatainwhicheverwayismostappropriate.
author’snote
Grafanaisawesome.Itisflexible,easytouse,andcanbeintegratedintotheexistingSCADAsystemforfreeandatnocosttoperformanceorcomplexity.FranklyIregretnotdiscoveringitsooner,sinceitwouldhaveprovidedaneasyanswertomostofthequestionswehadwhileSCADAwasbeingdeveloped.Ihavepersonallybeeninvolvedinmany,manydiscussionsabouttherequirementsforaSCADAgraphicalclient,andtomyknowledge,thismeetseveryoneofthemandthensome.SomeofthefeatureswhichIdidnotdescribeaboveincludetimedgraphannotations,automaticalerting,exportingdatatoCSVfiles,embeddingpanelsinotherHTMLfilesandmore.Becauseitisopensource,anyfeaturethatisnotalreadyincludedwithGrafanawillmostlikelyexistasaplugin,andifnot,couldbeeasilywritten.
TheteamshouldstronglyconsideradoptingGrafanaasitsdefaultSCADAclientforalldatavisualizationandmonitoringpurposes.Aservercouldbesetupinthedynoroomandprovideliveinsightoftestsbeingperformedtoanynumberofcomputersonthenetwork.Itiseasyenoughtousethateachsubsystemcouldhaveitsown,dedicateddashboard,writtenbytheteaminchargeofit,anddisplayingonlythedataneededforagivenusecaseortest.