ESCUELA TÉCNICA SUPERIOR DE INGENIERÍA (ICAI)
Investigation of Smart Grids scenarios by the use of co-simulation tools with a
European low voltage test feeder and an optimal power flow solver
Autor: Ezequiel Fredes Sáez Director: Anna Magdalena Kosek
Madrid Octubre 2016
AUTORIZACIÓN PARA LA DIGITALIZACIÓN, DEPÓSITO Y DIVULGACIÓN EN RED DE PROYECTOS FIN DE GRADO, FIN DE MÁSTER, TESINAS O MEMORIAS DE BACHILLERATO
1º. Declaración de la autoría y acreditación de la misma. El autor D. Ezequiel Fredes Sáez DECLARA ser el titular de los derechos de propiedad intelectual de la obra: Investigation of Smart Grids scenarios by the use of co-simulation tools with a European low voltage test feeder and an optimal power flow solver, que ésta es una obra original, y que ostenta la condición de autor en el sentido que otorga la Ley de Propiedad Intelectual. 2º. Objeto y fines de la cesión. Con el fin de dar la máxima difusión a la obra citada a través del Repositorio institucional de la Universidad, el autor CEDE a la Universidad Pontificia Comillas, de forma gratuita y no exclusiva, por el máximo plazo legal y con ámbito universal, los derechos de digitalización, de archivo, de reproducción, de distribución y de comunicación pública, incluido el derecho de puesta a disposición electrónica, tal y como se describen en la Ley de Propiedad Intelectual. El derecho de transformación se cede a los únicos efectos de lo dispuesto en la letra a) del apartado siguiente. 3º. Condiciones de la cesión y acceso Sin perjuicio de la titularidad de la obra, que sigue correspondiendo a su autor, la cesión de derechos contemplada en esta licencia habilita para:
a) Transformarla con el fin de adaptarla a cualquier tecnología que permita incorporarla a internet y hacerla accesible; incorporar metadatos para realizar el registro de la obra e incorporar “marcas de agua” o cualquier otro sistema de seguridad o de protección.
b) Reproducirla en un soporte digital para su incorporación a una base de datos electrónica, incluyendo el derecho de reproducir y almacenar la obra en servidores, a los efectos de garantizar su seguridad, conservación y preservar el formato.
c) Comunicarla, por defecto, a través de un archivo institucional abierto, accesible de modo libre y gratuito a través de internet.
d) Cualquier otra forma de acceso (restringido, embargado, cerrado) deberá solicitarse expresamente y obedecer a causas justificadas.
e) Asignar por defecto a estos trabajos una licencia Creative Commons. f) Asignar por defecto a estos trabajos un HANDLE (URL persistente).
4º. Derechos del autor. El autor, en tanto que titular de una obra tiene derecho a:
a) Que la Universidad identifique claramente su nombre como autor de la misma b) Comunicar y dar publicidad a la obra en la versión que ceda y en otras posteriores a través
de cualquier medio. c) Solicitar la retirada de la obra del repositorio por causa justificada. d) Recibir notificación fehaciente de cualquier reclamación que puedan formular terceras
personas en relación con la obra y, en particular, de reclamaciones relativas a los derechos de propiedad intelectual sobre ella.
5º. Deberes del autor. El autor se compromete a:
a) Garantizar que el compromiso que adquiere mediante el presente escrito no infringe ningún derecho de terceros, ya sean de propiedad industrial, intelectual o cualquier otro.
b) Garantizar que el contenido de las obras no atenta contra los derechos al honor, a la intimidad y a la imagen de terceros.
c) Asumir toda reclamación o responsabilidad, incluyendo las indemnizaciones por daños, que pudieran ejercitarse contra la Universidad por terceros que vieran infringidos sus derechos e
MasterThesisEzequielFredesSáez
2
TableofContentsAbstract....................................................................................................................6
English..............................................................................................................................6
Español.............................................................................................................................7
1.-Introduction..........................................................................................................8
2.-LVNetworkDescription.......................................................................................11
2.1.-Nodes.......................................................................................................................12
2.2.-Lines.........................................................................................................................12
2.3.-Linetypes.................................................................................................................13
2.4.-Transformertypes....................................................................................................14
3.-SystemDescription..............................................................................................15
3.1.-IntroductiontoCo-simulationwithMosaik..............................................................15
3.2.-ConfigurationofMosaikandPyPower......................................................................19
3.2.1.-Co-SimulationSetupImplementedinMosaikOverview.........................................19
3.2.2.-Var.py......................................................................................................................21
3.2.3.-Excel.py....................................................................................................................21
3.2.4.-LVTest.py.................................................................................................................27
3.2.5.-Output.py................................................................................................................31
4.-EuropeanLowVoltageTestFeederValidation.....................................................33
5.-Application..........................................................................................................36
5.1.-EnergyTheft.............................................................................................................37
5.2.-EnergyTheftTool.....................................................................................................38
5.2.1.-Powerbalance.........................................................................................................41
5.2.2.-NeuralNetwork.......................................................................................................44
6.-Experiments........................................................................................................46
6.1.-ChangesintheTool..................................................................................................46
6.1.1.-Var.py......................................................................................................................49
6.1.2.-Excel.py....................................................................................................................49
6.1.3.-LVTest.py.................................................................................................................50
6.1.4.-Output.py................................................................................................................54
6.1.5.-NN.py.......................................................................................................................56
MasterThesisEzequielFredesSáez
3
6.1.6.-Comparator.xlsx......................................................................................................59
6.2.-ExperimentsScenarios.............................................................................................60
6.2.1.-Experiment0Thefts................................................................................................60
6.2.2.-Experiment5Thefts................................................................................................60
6.2.3.-Experiment10Thefts..............................................................................................61
6.2.4.-Experiment15Thefts..............................................................................................62
6.3.-ExperimentsResults.................................................................................................62
6.3.1.-NetworkLevel.........................................................................................................63
6.3.2.-SubsystemLevel......................................................................................................63
6.3.3.-NeuralNetworkLevel..............................................................................................64
6.4.-ExperimentConclusions...........................................................................................65
7.-Conclusion...........................................................................................................66
8.-References..........................................................................................................69
9.-Appendix.............................................................................................................71
9.1.-Var.py......................................................................................................................71
9.2.-ExcelValidation.py...................................................................................................71
9.3.-Excel.py....................................................................................................................74
9.4.-LVTest.py.................................................................................................................77
9.5.-Output.py.................................................................................................................79
9.6.-NN.py.......................................................................................................................81
MasterThesisEzequielFredesSáez
4
TableofFiguresFigure1:NetworkDescription.[2].................................................................................11
Figure2:ScreenshotofNodes.[Grid.xlsx].....................................................................12
Figure3:ScreenshotofLines.[Grid.xlsx].......................................................................13
Figure4:ScreenshotofLineTypes.[Grid.xlsx]...............................................................13
Figure5:ScreenshotofTransformerTypes.[Grid.xlsx].................................................14
Figure6:Co-simulationStructure..................................................................................15
Figure7:DemoOutputofMosaik..................................................................................18
Figure8:OverviewoftheProgram................................................................................20
Figure9:Variables.[Var.py]...........................................................................................21
Figure10:ColumnExtractionFunction.[Excel.py].........................................................22
Figure11:ExcelInsertFunction.[Excel.py]....................................................................24
Figure12:CheckandCopyFunction.[Excel.py].............................................................26
Figure13:RearrangeCSVFunction.[Excel.py]...............................................................26
Figure14:LoadCreationFunction.[LVTest.py].............................................................27
Figure15:ConnectEntitiesFunction.[LVTest.py]..........................................................27
Figure16:MosaikVisualization......................................................................................28
Figure17:WebVisualizationJavaScriptChanges.[mosaik.js].......................................29
Figure18:WebVisualizationCSSChanges.[main.css]...................................................29
Figure19:SettingVisualizationConfiguration.[LVTest.py]...........................................30
Figure20:ImprovedWebVisualization.........................................................................30
Figure21:OutputPrintingFunction.[Output.py]..........................................................31
Figure22:OutputPlotExample.....................................................................................32
Figure23:RealOutputFeeder.......................................................................................33
Figure24:PercentageDifferencebetweenFeederandCo-simulation.........................34
Figure25:AccumulatedErrorinPercentage.................................................................34
Figure26:BoxChart.......................................................................................................35
Figure27:PaperMethodSummary.[11].......................................................................39
Figure28:FlowDiagramtoevaluateEnergyTheft........................................................40
Figure29:Q-GIS.............................................................................................................43
Figure30:SystemsDesign..............................................................................................43
MasterThesisEzequielFredesSáez
5
Figure31:ArchitectureofNeuralNetwork.[14]...........................................................44
Figure32:OverviewoftheProgramwithChanges........................................................47
Figure33:LayoutDefinition.[Var.py]............................................................................49
Figure34:ChangesinRearange_csvFunction.[Excel.py]..............................................50
Figure35:HousesandTheftsLoadProfile.[Excel.py]....................................................50
Figure36:CreatingVariableNumberofLoadsandThefts.[LVTest.py].........................51
Figure37:ConnectingLoadsandThefts.[LVTest.py]....................................................52
Figure38:WebVisualizationChanges...........................................................................53
Figure39:ZoomedWebVisualization............................................................................53
Figure40:Hdf5ReaderandWriter.[Output.py]...........................................................54
Figure41:LinesReaderandWriter.[Output.py]...........................................................54
Figure42:NodesReaderandWriter.[Output.py].........................................................55
Figure43:TheftsReaderandWriter.[Output.py].........................................................55
Figure44:DataReadingFunction.[NN.py]....................................................................56
Figure45:NeuralNetworkDataPreparation.[NN.py]..................................................57
Figure46:NormalizeFunction.[NN.py].........................................................................57
Figure47:NeuralNetworkPrototype.[NN.py]..............................................................58
Figure48:NeuralNetworkDeveloped.[NN.py]............................................................58
Figure49:SubstationComparator.[Comparator.xlsx]..................................................59
Figure50:SubsystemsComparator.[Comparator.xlsx].................................................59
Figure51:NNComparator.[Comparator.xlsx]...............................................................60
Figure52:Experiment5Thefts.[Comparator.xlsx]........................................................61
Figure53:Experiment10Thefts.[Comparator.xlsx]......................................................61
Figure54:Experiment15Thefts.[Comparator.xlsx]......................................................62
MasterThesisEzequielFredesSáez
6
Abstract
English
According to the Danish energy strategy, Denmark's power consumption has to be
independentofsourcesofenergyreliantonfossilfuelsby2050.Inordertoachievethis
goal, renewable energy is the key component to provide energy and reduce the
greenhouseemissions.Forthesakeofmaintainingthequalityandreliabilitythatthe
powersystemhastoday,newwaysofoperatingpowersystemswillbenecessary.Future
SmartGridsscenariosincludingnewcomponentsandcommunicationtechnologycannot
be investigated in the current power system. New tools need to be developed to
representthecomplexityofSmartGridandallowenergysystemsstakeholderstodefine
andexecuterealisticscenarios.
Mosaik allows the co-simulation of different simulation models and simulators and
opensthepossibilitytohavelargescaleSmartGridscenarios.
Theobjectiveof theproject is tounderstand theEuropean lowvoltage test feeder in
order to implement it into the program PYPOWER, that is the optimal power flow
simulator.ThisimplementationwillbeusedtogetherwithMosaiktoperformsimulations
oflargeSmartGridscenarios.
ThefeederimplementationwillbetestedinseveralSmartGridsscenariostocheckthe
robustness of the tool. Finally, the prepared co-simulation set-up will be used to
investigatedifferentSmartGridsapplication,suchasevaluationoftheenergytheft.
Thepossibilitytohaveadvancedandpreciseco-simulationtoolswillhelpenergysystem
stakeholderstodevelopsolutionsintheSmartGridsmuchfasterandaccurate
MasterThesisEzequielFredesSáez
7
Español
Deacuerdoa laestrategiaenergéticadanesa,elconsumodepotenciaenDinamarca
tienequeserindependientedefuentesdeenergíaprovenientesdecombustiblesfósiles
paraelaño2050.Conobjetodeconseguirlo,lasfuentesdeenergíarenovablessonel
componenteclaveparaproporcionarenergíayreducirlosnivelesdeemisióndegases
deefectoinvernadero.Conlaideademantenerlacalidadyfiabilidadquelossistemas
depotenciahoyendía,nuevosmétodosdeoperacióndelossistemasdepotenciason
necesarios. Escenarios de Smart Grids en el futuro, incluyendo nuevos componentes
tecnológicosysistemasdecomunicación,nopuedenserinvestigadosconlossistemas
depotenciaactuales.Nuevasherramientasnecesitanserinvestigadaspararepresentar
lacomplexidaddelasSmartGridsypermitiralosinteresadosensistemasdeenergía
definiryejecutarescenariosrealistas.
Mosaik es una herramienta que permite la co-simulación de diferentes modelos y
simuladoresya suvezabre laposibilidadde tenerescenariosdeSmartGridsagran
escala.ElobjetivodelproyectoesentenderelEuropeanLowVoltageTestFeederpara
implementarloenelprogramaPyPower;queesunsimuladordeflujosdecarga.Esta
implementaciónserealizarájuntoconMosaikpararealizarsimulacionesenescenarios
deSmartGridsagranescala.
La implementacióndelFeeder será testadaenvariosescenariosdeSmartGridspara
comprobarlafiabilidaddelaherramienta.Finalmente,laherramientadeco-simulación
desarrolladaseemplearáendistintasaplicacionesdeSmartGrids,comoporejemplola
evaluacióndelrobodeenergía.Laposibilidaddetenerherramientasdeco-simulación
avanzadasyprecisasconseguiráque los interesadosensistemasdeenergíaeléctrica
puedandesarrollarsolucionesenlasSmartGridsdemaneramásrápidayprecisa.
MasterThesisEzequielFredesSáez
8
1.-IntroductionThetermSmartGridsisdaybydaybecomingmoreandmorepopular.Theinstallation
ofmoresophisticatedmeasurementdevicesinpowersystem,bringsmoreinformation,
especiallyinrealtime,fromthegrid;informationthatwasnotquicklyavailableinthe
past.Asnewdataandresourcesareavailable,newmethodsforevaluatingthegridcan
beusedtoanswerdemandinggridchallengesmuchfasterandmoreefficient.
This project is developed as part of the European project called European Research
InfrastructuresupportingSmartGridSystemsTechnologyDevelopment,Validationand
RollOut,orERIGridproject.ThetargetoftheERIGridproject,asstatedintheproposal
oftheprojectistosupportthetechnologydevelopmentaswellastherolloutofSmart
Grid approaches, solutions and concepts in Europe taking a holistic, cyber-physical
systemsapproachintoaccount.ItintegratesthemajorEuropeanresearchcenterswith
a considerable, outstanding Smart Grid research infrastructure to jointly develop
commonmethods,concepts,andprocedures.[1]
Theobjectiveofthismasterthesisistoprovideatoolthatisabletoautomaticallyload
andsimulatelargesmartgridsscenarios,withtheuseofco-simulationandapplythe
results of the simulation to an energy theft scenario. The reason for choosing co-
simulation tool is the ability to join together different simulators, that have been
developed and tested in their domains, and very easily join them together without
needingtocreateasinglesimulatorthatrepresentsmanydomains.
Thisprojecthasseveralgoalsthatareneededtobedonetofulfilltheobjective.Butin
order to achieve the goals, some requirements have to be meet first. These
requirementsare:
• R1:LoadtheEuropeanLowVoltageTestFeederfromtheoriginalfiles.
• R2: Exchange original loads and/or production units from the European Low
VoltageTestFeederwithotherloadsand/orproductionunitsatanybusinthe
EuropeanLowVoltageTestFeeder.
• R3:ReconfigureVisualizationtodisplaylargedistributionnetworks.
MasterThesisEzequielFredesSáez
9
• R4:Randomlydistribute loadsand/orproduction intobusesatEuropeanLow
VoltageTestFeeder.
These requirements will be developed within the goals. The goals consist on the
milestonesrequiredforthefulfillmentoftheobjectiveoftheproject.Thegoalsmayas
wellbedividedintosubgoals.Foreachofthegoals,itwillbestatedalsowhereinthe
projectitisachieved.
§ G1:ModifyMosaikexampledemocodeinordertofulfillrequirements(R1,R2
andR3).[Achievedinchapter3]
o G1.1: Modify PyPower configuration file and prepare helper scripts.
[Achievedinchapter3.2]
o G1.2: Obtain load and/or production profiles from an open database.
[Achievedinchapter3.2.3]
o G1.3:Prepareconfigurationscriptstodistributeloadand/orproduction
profiles on thebuses of the implemented European LowVoltage Test
Feedernetwork.[Achievedinchapter3.2.3and3.2.4]
o G1.4:ModifyvisualizationfilestodisplayallEuropeanLowVoltageTest
Feederinthevisualizationbrowser.[Achievedinchapter3.2.5]
§ G2: Validate the obtained results against the results provided in the
documentation.[Achievedinchapter4]
§ G3: Select a smart grid scenario to be implemented in the tool. [Achieved in
chapter2]
§ G4:Developanenergytheftdiscoverymethod.[Achievedinchapter5]
o G4.1:Performliteraturereviewonenergytheftdiscoverymethodsand
selectamethodtobeimplemented.[Achievedinchapter5.1]
o G4.2:Modifytheselectedenergytheftdiscoverymethod.[Achievedin
chapter5.2]
§ G5:Preparethetooltoexperimentwith.[Achievedinchapter6.1]
o G5.1: Prepare models and datasets for the investigated method.
[Achievedinchapter6.5]
§ G6:Performseveraltestcasesontheselectedscenarioandgathertestresults.
[Achievedinchapter6.2]
MasterThesisEzequielFredesSáez
10
§ G7:Interpretthetestresultsanddrawconclusions.[Achievedinchapter6.3]
§ G8: Provide recommendations for usewith the developed tool. [Achieved in
chapter7]
§ G9:Publishthedevelopedsoftwareinasharedrepository.[Achievedinchapter
7]
MasterThesisEzequielFredesSáez
11
2.-LVNetworkDescriptionInthischapteritisdescribedthewholelowvoltagegridthatisusedduringthewhole
project.Inordertobetterexplainthedifferentpartsofthegrid,itisshowntheoutput
ofthetooltosimulatethegrid;thisisanExcelfilecalledGrid.xlsx;thecreationofthis
Excelfileisexplainedinchapter3.Thebestwaytodescribethenetworkisgoingthrough
eachofthesheetsintheGrid.xlsxExcelfilecreatedforthePyPowersimulator.
Inordertohaveatoolthatisabletocreatescenariosandmodelsasstandardaspossible
ithasbeenconsideredallover theproject that the informationof thegrid isalways
giveninthesameformatastheonefromtheEuropeanTestFeeder;therefore,thetool
configuredismorethanenoughtodevelopthesimulationsneeded.Asaresult,during
this project, and in the different experiments performed, the characteristics of the
physical network are to be left unchanged. The European Low Voltage Test Feeder
consistsofaradialnetworkasseeninFigure1withanominalfrequencyof50Hz.Inthe
northwesterspointofthenetwork,thereisasubstationwithatransformerthatlinks
themediumvoltagetothelowvoltage.Thetransformeris11/0.416KVtransformation
rate.
Figure1:NetworkDescription.[2]
MasterThesisEzequielFredesSáez
12
2.1.-Nodes
Thefirstofthesheetsisthenodessheet.Inthissheet4differentcolumnsmaybefound.
Thefirstcolumnisthenameofeachnode.Forsimplicitypurposes,PyPowerconsiders
thatthereferencenodeisthehighsideofthetransformerandthesecondarysideis
consideredasthefirstnode.Asinthegridisnotmeanttohavegeneratorsin,allthe
nodes are considered to be PQbuseswith the exceptionof the primary side of the
transformerthatistheReferencenode.Themeaningofthereferencenodeisthatitis
thenodewiththeangleof0º,sotherestofthenodeshavedifferentanglesbutwith
comparisonto thereferenceone.The fact that thenodesarePQmeans thatall the
nodes are considered as consumers, none of it has network stability or generation
purposes.Finally,thethirdcolumnisusedforhavingthebasevoltageofeachofthe
nodes.Asitisobvious,thereferencenodehasthebasevoltageofthehighsideofthe
primarysideofthetransformerandtherestofnodeshavethevoltageofthesecondary
sideofthetransformer.Inthegridthatisusedduringtheproject,thereareatotalof
906nodesplustheprimarysideofthetransformerasareference.Thebasevoltages
thatareusedare11kVand416Vrespectively.
AscreenshotofthenodesistheexcelfileisseeninFigure2:
Figure2:ScreenshotofNodes.[Grid.xlsx]
2.2.-Lines
Thenextworksheetisdedicatedtothelinesinthegrid.Thefirstcolumnisusedforthe
nameofthelineonly;thenthesecondandthethirdcolumnareusedtodefinewhichis
thenodeoforiginandwhichoneisthedestinynode.Aseachofthebranchesmayhave
adifferentkindofwireconnectingthenodes,inthefourthcolumnitcanbeseenwhich
kindofcableisconnectingthenodes.
MasterThesisEzequielFredesSáez
13
Thenextcolumnisusedforhavingthelengthoftheline,expressedinkm.Thecolumn
namedOnlineisusedtodefinewhetherthelineisconnectedornot.Inthisscenarioall
thelinesarealwaysconnected.Thereisonelastcolumnthatonlyappliesforthefirst
row that is called Taps; this value is used to define which transforming rate the
transformer is using. There is a total of 905 lines plus the transformer, which for
simplicitypurposesisconsideredasaline.
AscreenshotofthelinesistheexcelfileseeninFigure3:
Figure3:ScreenshotofLines.[Grid.xlsx]
2.3.-Linetypes
Linetypesworksheetisusedtodefineallthedifferentkindsofwiresthatcouldbeused
inthenetwork.Inthisproject10differentkindofwiresareconsidered.Foreachofthe
wires there are four columns to describe the characteristics of the wires, this is,
resistance,inductanceandcapacitance.Plus,anextracolumnforthemaximumcurrent
thatcangothroughthewire.Thefirstofthecolumnsisusedforthenamingofthekind
ofwirethatisusedbeforeintheLinesworksheet.
AscreenshotofthelinetypesistheexcelfileisseeninFigure4:
Figure4:ScreenshotofLineTypes.[Grid.xlsx]
MasterThesisEzequielFredesSáez
14
2.4.-Transformertypes
This lastworksheet isusedtodescribethedifferenttransformersthatmaybe inthe
network.Asinthisprojectonlyonetransformerisconsidered,onlyonetransformeris
described. There is a column for the name of the transformer, another one for the
nominalpower,themaximumcurrentsontheprimaryandsecondaryside,thepower
loss and the impedances of the transformer and one last column to define which
transformationsratethetransformercoulduse.
AscreenshotofthetransformertypesintheexcelfileisseeninFigure5:
Figure5:ScreenshotofTransformerTypes.[Grid.xlsx]
MasterThesisEzequielFredesSáez
15
3.-SystemDescription
3.1.-IntroductiontoCo-simulationwithMosaik
Co-simulation is the termusedwhendifferent subsystems (so called simulators) are
modeled, and simulated, in a distributed way. This means that, in co-simulation
software, each of the programs are independent from the others. Having different
simulatorsworkingatthesame,insteadofjustonlyone,timehasmainly2advantages.
Ononehand, it represents lesscomputationalefforthavingtheprocessesseparated
andthenexchangingthenecessaryinformationbetweenthem.Ontheotherhand,and
mostimportant,itallowstoreuseexistingmodelsandsimulatorsofcomplexscenarios
that have been developed, tested and verified; instead of having to develop these
simulatorsinsideasinglemultidomainsimulationenvironment.Forexample,anuclear
plantmayhaveaverycomplexsimulatorthatworksproperly;withco-simulationone
wouldonlyneedtodefinewhichdatahastobeexchangedbetweenanuclearplantand
therestoftheinvestigatedpowersystem.
Thisindependencyavoidshavingcouplingproblemswhilestillthedifferentsubsystems
areabletoexchangedatabetweeneachother.[3]
Asimplewayofdepictingtheideaofco-simulationmaybefoundFigure6:
Figure6:Co-simulationStructure.
Synchronizer
Model1 Compiler1
Model2 Compiler2
Model3 Compiler3
MasterThesisEzequielFredesSáez
16
Inthisproject,thesynchronizerthatisusedisMosaik.Theroleofthesynchronizeristo
exchange the data and configure and execute simulators. It is very important to
understandthefundamentbehindtheprogram.Mosaikgrantsthepossibilitytohave
severalsimulatorsandsimulationmodelsworkingtogether,andasaresult,havingthe
possibilitytoanalyzelargeSmartGridscenarios.Thisprogramallowstheusertocreate
andrunco-simulationsetupsworkingatthesametime.
The whole co-simulation system is made of different models that can even have
different compilers and they communicate between each other through the
synchronizer,thatinMosaikitiscalledOrchestrator.ThewayMosaikco-simulatesand
exchangedatabetweenthedifferentsimulatorsisthankstoadiscreteeventsimulator
library.Eachofthedifferentsubmodelsisrunningwiththeirownsimulatorinstanceor
processandcommunicatewiththeothermodelswhenacertaineventoccurs.Thisway
ofinteractingallowsaswelltohaveeventsthatareyielded,orinotherwordsstopped,
until a certain event happens. There is also a possibility of having asynchronous
simulations. [4]ThewayMosaikcoordinatestheexecutionofthesimulatorsandthe
data exchanged between them is thanks to anOrchestrator. TheOrchestrator is an
automatedcoordinatorandmanagerofcomputersystemsandservices.Itcanbeseen
asanadvancedautomationsystemandcontrolmethod.[5]
MosaikisanApplicationProgrammingInterface(API)whichallowstosettheframework
for Smart Grid co-simulation. Mosaik is written in Python language. Python is an
interpreter (like C++ or Java) object oriented with a high-level and dynamic
programming language [6]. Mosaik is an open source software that provides a
demonstrationscenariofortheuserstobeginworkingwithandunderstandingthetool.
In this thesis, thedemo,provided inMosaikwebpage,hasbeenusedas thestarting
point for the implementationof the chosenpower system scenario.As a result, the
demo provided with slight modifications is used in this chapter to explain the tool
capabilities. Inside the demo, there are four simulators that are used with some
modificationsthroughoutthewholeproject.Inthischapter,itisexplainedthewayeach
simulatorworks and howMosaik joins them together. Before describing them, it is
necessarytomakeadifferencebetweensimulatorsandMosaikadditionaltools.
MasterThesisEzequielFredesSáez
17
Simulatorsareusedtosimulateoremulatearealsystemwheretheadditionaltoolsonly
aidtheco-simulationsetup,withforexamplevisualization,datalogging.Thesimulators
are in a domain that is interesting in the co-simulation system (power, ICT, energy,
mechanical,social,etc.)thatcontainsobjectsarepartofthemulti-domainscenario.
Thenameofeachsimulatorandashortdescriptionofeachofthemislistedbelow:
• Mosaik_csv:Thisisasimulatorthatisusedtosimulatecommaseparatedvalues
(CSV) files. This simulator is able to open, a csv file, read it and store the
informationindifferentvariables,basedoncolumnnames.Thissimulatorcan
beusedtosimulateanyrecordedsystembehavior;forexample,timeseriesofa
homepowerconsumption.
• Mosaik_PyPower: This simulator is an adaptation of the optimal power flow
solvernamedPyPower.Thissimulatorisinchargeofcalculatingthepowerflow
andthevoltagedropinthenodesofthesystem.
Thenameofeachadditionaltoolandashortdescriptionofeachofthemislistedbelow:
• Mosaik-hdf5:Itisnecessarytostoretheinformationproducedinthesimulation.
Thistoolgathersallthedatafromthesystemandsaveitwithaspecificstructure
intoahdf5file.Thetoolcanbeconfiguredtostoreonlyselectedtimeseries
• Mosaik-web: This tool provides the web visualization of the scenario. It is
possibletoselectthewaythecomponentsintheset-uparedisplayedintheweb
browser.Itispossibletochangethelocationinthescreen,thecolor,theshape
andmanymoreattributes.
Both simulators and tools are connected via Mosaik simulator API. In the Mosaik
ecosystem, tools to recordordisplaydataare treatedas simulators.Thesoftware is
usedinthefollowingwaythroughoutthedemo.Firstthesimulatorsareinitializedin
Mosaik.ThentheCSVsimulatorisusedinordertostoretheloadprofilesintovariables
thatareusedinthesystem.Atthesametime,thePyPowersimulatorisruntocalculate
thepowerflowbetweenthebranches.
MasterThesisEzequielFredesSáez
18
Theinformationofthelocationofthebranches,theirlongitudeandtheirimpedancesis
loadedfromaJSONfilethathastobegiventothePyPowerprogramasaconfiguration
file. Manipulating this file provides the user with the possibility to change the grid
configuration.AlltheinformationloadedfromtheCSVandJSONfilesandtheoutputof
thePyPowersimulationaresavedwith theuseof thehdf5simulator.This simulator
takesallthedataandsaveitstructurallyintoahdf5file.Finally,thewebsimulatorallows
theuser to see the topology that is beingused and also real time values calculated
throughthepowerflowsimulator.
Figure7showstheoutputoftheco-simulationviathewebvisualization.
Figure7:DemoOutputofMosaik.
Asitisseeninthefigure;thereisanumberofnodesthatcanbevisualized.Oneofthe
nodesisdeterminedasthereferencebus(RefBus)andtherestaredefinedasPQBus.
TheconnectionsbetweenthedifferentPQBusesandwiththeRefBusarethephysical
lines inthenetwork.Allthisnodesandconnectionsareintroducedandsimulatedby
PyPower. Component nodes, like for example Batt and PV, represent the power
consumptionandproductionthatisconnectedtoaPQBus.TheBattandPVnodesare
connectedtothePQBusbyICTconnections;onlydataisintroducedinthePQBus.This
BattandPVconsumptionarebeingsimulatedwiththeCSVsimulator.Theconnection
betweenthetwosimulatorsisdonethankstoMosaik.Inordertosimulateanimpactof
aPVproductionorthepowerflowofthePQBus,thedatabetweenPVsimulatorand
loadflowcalculatorneedtobeexchanged.Mosaikfacilitatesthisexchangeandre-runs
calculationsforeverysimulationstep.
MasterThesisEzequielFredesSáez
19
3.2.-ConfigurationofMosaikandPyPower
Inordertosimulateapowersystemnetwork,PyPowerneedstheinformationofthe
characteristicsofthegridandtheloadsinaveryspecificformatandstructure.However,
theinformationandcharacteristicsofthesystemarepresentedintheEuropeanTest
Feederinanothercompletelydifferentformatandstructure.Asaresult,itisnecessary
toreadalltheinformationthatiscontainedinthesource,modifyitandafterwardssave
itinthecorrespondingformatforMosaiktoconfigurePyPowercorrectly.Inordertobe
abletocreateandsimulateascenariosomestepshavetobefollowed.First,ithastobe
developedafilewithalltheinformationfromthegrid.Secondonevariablewithallthe
informationoftheloadshastobedefined;andofcourse,oneloadprofilefileforeach
oftheloads.Afterwards,theco-simulationsetupisreadyforexecution.
3.2.1.-Co-SimulationSetupImplementedinMosaikOverview
Beforeexplaining indetailhoweachof thedifferentpartsof theprogramworkand
communicatewiththeotherparts,itisinterestingtographicallyexplaintheoverview
oftheco-simulationimplementationsetup.Asitisexplainedthroughtheprojectthere
aremainly4pythonscriptsthatweredevelopedforthepurposeofthisthesis.Theyare
Var.py,Excel.py,LVTest.pyandOutput.py.Thesescriptsarepresentedinchapter3.2.2
to3.2.5.
Thesefilesaredividedinto2differentcategories;configurationfilesandhelperfiles.
Theconfiguration filesaremeant tobeaccessibleby theuser.Theusermaychange
themtosimulatedifferentscenariosorobtaindifferentoutputs.Var.pyandOutput.py
are theconfiguration files.Thehelper filesare filesmade toease theuseof theco-
simulationsetupaswellasfacilitatingfuturemodifications.Theyarenotmeanttobe
changedduringthescenarioreconfigurationoncecreated.Excel.pyandLVTest.pyare
thehelperfiles.
Theideaoftheconfigurationfilesisthatalltheinformationthathastobegiventothe
program is provided in Var.py script; and the information of the simulation that is
neededoraskedbytheuserasdescribedinthescenariocanbeselectedonOutput.py.
Therestoffilesarenotmeanttobetouchedorused,sothatthesystemcanusethem
internallytoperformthesimulation.
MasterThesisEzequielFredesSáez
20
This way the risk of changing anything that is necessary is minimized and also the
applicationwould turn to bemoreuser friendly, avoiding large source code that an
unexperiencedusermaynotunderstand.
InFigure8 there isanoverviewofhowtheco-simulationsetup implementationhas
beendeveloped:
Figure8:OverviewoftheProgram.
Inthefigure,therectanglesarethepythonscripts(ingreentheconfigurationscripts
andinredthehelperscripts);theellipsesaretheinputtothescriptsandthetriangles
aretheoutputsofit.Thearrowsmeanflowofcommunicationbetweenthefilesand
theyhavealsodrawnthedirectionofthedataflow.
Theprocessisdescribedasfollows;theuseronlyneedstorunthescriptinOutput.py;
thisscriptautomaticallycallsLVTest.py.InLVTest.pyfirstofall,Excel.pyandVar.pyare
run.Excel.pyreadsallthevariablesfromVar.pyandtheinformationfromtheEuropean
Low Voltage Test Feeder (ELVTF) and formats it to the proper structure that the
simulator needs. The grid information is saved inGrid.xlsx. Afterwards, in LVTest.py
Mosaik iscalledandtheco-simulation issettledup.After thesimulation isdonethe
resultsofthesimulationarestoredintoseveraldifferentdocumentsbyOutput.pyfor
furtheranalysis.
MasterThesisEzequielFredesSáez
21
3.2.2.-Var.py
Thisscriptisusedtodefinewhichcolumnshavetobetakenoutofthecsvfilesgivenby
theEuropeanLowVoltageTestFeeder.Itisimportanttosituatethesevariableshere
becauseitmightbepossiblethatwithfutureupgradesofthetool,theusermightbe
willingtoaddnewcomponentsorcharacteristicstothetool.Forexample,itcouldbe
possible to add the geographical coordinates of the nodes; that can be passed to
visualizationoranyothersimulationtool.Thisinformationisgiveninthecsvfilesbutat
themoment is not used. It also has three very important variables in the proposed
scenario;thestartdate,theenddateandthenamethatisusedfortheExcelfilewith
allthegridinformation.
ThewaythevariablesaredefinedinthescriptcanbeseeninFigure9:
Figure9:Variables.[Var.py]
3.2.3.-Excel.py
Withthepurposeofavoidinglargescriptsandanykindofcouplingissues,anewscript
calledExcel.pywascreated.Thisscripthastheobjectiveofcreatingandformattingall
thenecessarydatatobeprocessedbythePyPowertool.Tobeginwith,thedifferentcsv
fileswiththeinformationofthegridandloadshavetobereadandstoredintovariables.
Duringthisproject, it isassumedthatthistool isusedalways. It isassumedthatthe
structureoftheEuropeanLowVoltageTestFeederfileisnotmodified.
AsthefileformatthefeederisCommaSeparatedValues(csv)files,anewlibraryforthe
scripthastobeimported.Thislibraryisthecsvlibrary.
MasterThesisEzequielFredesSáez
22
Toautomatizetheprocessofextractingtheinformationfromthecsvfiles,afunction
hasbeencreatedtoallowtheusertoextracttheinformationneededpercolumnsfrom
the csv file and storage the data into a variable. To storage the information into
variables,aspecialstructureofvariablehastobeused,theyarearrays.Tobeableto
createarraysandworkwith themafterwards,another libraryhas tobe importedas
well.Thisisthenumpylibrary.
Thesourcecodementionedbeforetoextracttheinformationfromthecsvandstorage
themintoavariableisshowninFigure10:
Figure10:ColumnExtractionFunction.[Excel.py]
Inthisfunctionthereare3inputs,csv_namethatisthenamewiththecorresponding
paththecsvinquestionhas;columnsthatisavariablethathasthenameoftheheaders
ofthecolumnsthattheuserwantstoextract;andfinallyrepthatisthenumberofrows
thathavetobedeletedfromthecsvbeforetreatingthedata.
Thenameandpathofthecsvfileisnotgoingtobeaccessiblefortheuser.Itisnecessary
tohavethecsvfileswiththeoriginalnamefromthefeederwebpageandinsideafolder
intheprojectcalledLVData.Thereasonfordoingthis,isthatthiswayitisavoidhaving
missingdata,ordatachangedbytheuser.
MasterThesisEzequielFredesSáez
23
Thevariablecolumns,withthenameofthedifferentcolumnstheuserwantstoextract
fromthecsvandstorageintovariables,isavariablethatisdefinedbytheuserinanother
script,Var.py.Itisclearthatitwouldbenecessaryfortheusertoatleastopenthecsv
filesandunderstandthemalittlebitiftheuseriswillingtoextractanothercolumnthat
theonesdefinedbydefault.
Oncethisfunctionhasbeencreated,itisexecutedinallthesevencsvfilesprovidedin
the European LowVoltage Test Feeder to extract the information. This csv files are
Buscoords.csv, LineCodes.csv, Lines.csv, Loads.csv, LoadShapes.csv, Source.csv,
Transformer.csv.Itisnotworthytogoovereachofthefilesprovidedinthefeederand
explainingtheirinformation.However,itisnecessarytounderstandwhichinformation
isprovided to the simulator, thishasbeenalreadydescribed in chapter2, asall the
informationisstorageintheExcelfile.
Nowthatalltheinformationhasbeenreadfromthecsvfiles,ithastobesavedtoafile
forthesimulatortocreatethegrid.Thesimulatorthatreadandrunthegridshapeis
PyPower;thissimulatorneedstohaveintroducedtheinformationfromthegrid.The
wayofinputtingthegridinformationintothesimulationcanbedoneusingonlytwo
differentkindoffiles,JSONandExcel,bothformattedinthecorrectway.
Originally,inthedemoversionofMosaik,thisfilewasaJSONfile;butitwasdecidedto
savetheinformationintoExcelformat.
Thereasonbehindthisdecisionweremainlytwo.First,havinganExcelfileallowsthe
usertoautomaticallyunderstandthestructureoftheinformationthatisrepresented.
Second,ifsmallchangesaretobemade,theExcelfileismoreaccessibletotheuser.
ThenamethisExcel filehas isGrid.xlsx;andthestructurefromthefile isalwaysthe
same,forthePyPowersimulatortoreadthedata.
MasterThesisEzequielFredesSáez
24
Functionexc_insert,aspresentedinFigure11createstheGrid.xlsxfile.
Figure11:ExcelInsertFunction.[Excel.py]
Basic Python packages are not able to create and write Excel files, but there is an
externallibrarythatcanbeusedtodoso.Thislibraryiscalledxlsxwriter.Thevaluesare
introduced in Excel one by one, but itmakes theworkmuch easier if a function is
created,thatintroducesthedatajustcellpercell,rowperrow,columnpercolumnor
asamatrixforlargeamountofdata.Itisalsousefultoinserttheheadersofthedifferent
columnsinExcel;butthisisjustfortheuserinformationasthefirstrowoftheexcelfile
isnotreadbythePyPowersimulator.
Beforegettingintoexplainingthedifferentwaystheinformationmaybeintroducedin
theExcelfile,itisexplainedwhatisthepurposeofthedifferentvariablesthatenterin
thefunction.Varisthevariableusedastheinputdatathatneedstobeintroducedin
theExcelfile.
MasterThesisEzequielFredesSáez
25
Itcanbefromanysizeorformat,butdependingonthewayitwantstobeintroduceda
distinct“if”caseisused.Variablesrowandcolarethestartingrowandcolumnwhere
itisnecessarytointroducethedata.
The ending row or column is not necessary to be given as an input because it is
automaticallycalculatedwiththelengthofthevariable.Then,variablekindisusedto
specifythewaydatahastobeintroducedintheExcelfile,orinotherwords,whichcase
hasbeenselected.Thereissomeinformationthatisnotpresentinthecsvfilesfromthe
feeder;whenthissituationhappens, the informationhastobe introducedmanually,
withthatpurposeonmindthevariablenumeroisused.
Finally,thedatareadfromthecsvfilesisinaspecificformat,thatisastringformat;for
thesystemtoworkcorrectly;thecellsthatrepresentvalues(thisisforexamplelengths,
voltagesorcurrents)havetobeintroducedwithanumberformat,thatiscalledfloat.
Thelastvariablecalledftlisusedtodeterminewhichcolumnisthefirstcolumnfrom
withtheExcelwritertriestoconvertthedatafromstringelementstofloat.
TherearefivedifferentwaysofintroducingtheinformationintotheExcelfile.Itcould
beintroducedjustonedatainonecell;thisiscontemplatedinthecaseOne.
Also,itcouldbeintroducedasamatrixofelements,thisistheeasiestwaytodoit,but
theinformationhastobeall introducedinthesamevariable;thisiscontemplatedin
caseMatrix.RowisusedtointroducethedatarowperrowandHeaderworksexactly
thesamebutitisusedonlyfortheheadersofthedifferentcolumns.Atlast,Multipleis
usedtointroducedthevaluespecifiedtheamountoftimesofthevariableintroduced.
Duetothewaytheinformationisgiveninthecsvfilesfromthefeeder,onevariable
containsthenames,locationsandshapesoftheloadsandanothervariablecontainsfor
eachshape,the loadprofilethathastobeused.Anotherfunctionwasdevelopedto
matchthistwovariableandendwiththevariableLoads,thathasforeachshape
MasterThesisEzequielFredesSáez
26
This function isnotuniqueforthispurposeandcanbeusedforanymatchbetween
variablesthatisnecessary;itisrepresentedinFigure12:
Figure12:CheckandCopyFunction.[Excel.py]
Finally,the loadshapeshavetobereformattedtothestructurethattheMosaik_csv
simulatorusestoreadcsvfilesandtranslatethemintoloads.
This function is created, having inmind the format the European Low Voltage Test
Feedercsvfilesare,butwhenadifferentscenarioisrunonthelowvoltagegrid,other
loadprofilesareusedandveryprobablywithdifferentformat;therefore,thefunction
wouldneedtobereadjustedinthefuture.
ThemostimportantinformationisthattheMosaik_csvsimulatorneedsacsvfilewith
thefirstrowasthenameofthevariable;thesecondrowwiththenameoftheattribute
thevariablehas,inthiscase,itispowerinW.
Fromthethirdrowon,dataisstructuredintwocolumns,thefirstcolumnisthedate
withthefollowingformat:YYYY-MM-DDHH:mm:ss;andthesecondcolumnisthevalue
oftheattribute.
ThesourcecodeisshownintheFigure13:
Figure13:RearrangeCSVFunction.[Excel.py]
MasterThesisEzequielFredesSáez
27
Thewaythisfunctionworksisthatfirstitopens,readsandstoresthedatainavariable
fromeachcsvshapefile; thenthe informationfromthevariable is introduced inthe
properformatintoanothercsvfilewithexactlythesamenameastheoriginal,butina
differentfolder.
3.2.4.-LVTest.py
Allthefilesandvariablesarereadytobeusedbythedifferentsimulatorsprovidedby
theMosaik demo, only some little readjustments have to be done. As a result, it is
explainedindetailhowMosaikworksandthedifferentinformationthathastobegiven
tothesimulators.WhenusingMosaik;firstallthedifferentsimulatorsmustbedefined
andinitialized.ThePyPowersimulatorhastoreadtheExcelfilewiththeinformation
fromthegridtobeabletocreateit.Oncethisisdone,theprogramisreadytocreate
the different loads that are connected in the grid. As eachof the loadsmayhave a
differentloadshapeorprofile,theMosaik_csvsimulatorhastobeinitializedforeachof
theloads.Thisisnotsystematicandtherehastobecreatedasmallfunctiontocreate
theloadswiththeirownprofileandstorageallthatinformationinsideavariable.
ThewayithasbeendoneisshowninFigure14:
Figure14:LoadCreationFunction.[LVTest.py]
Thevariableloadresultsinavariablewiththeinformationofthelocationandtheshape
ofeachoftheloadsinthenetwork.Aseachoftheloadsisconnectedinaspecificnode,
acertainfunctiontoconnectproperlytheloadshastobedeveloped.
Thecodecreatedtofirstidentifythenodesandthenconnecteachloadtotheproper
PQBusnodeisinFigure15:
Figure15:ConnectEntitiesFunction.[LVTest.py]
MasterThesisEzequielFredesSáez
28
Therestofthecodeisverysimilartotheoneprovidedinthedemo.Theonlythingthat
maybenecessarytomodifyistherangewithinthenodesareconsideredtobeinlimits.
TheoutputofthesimulationcanbeseeninFigure16:
Figure16:MosaikVisualization.
Asitcanbeseen,duetothehugenumberofnodesinthesimulation,itisdifficultto
visualizethelocationofthenodes.Tryingtoimprovethissituation,thecharacteristics
of the visualization have been changed in the Web Visualization Simulator. The
advanced configurations have been changed in the JavaScript that contained all the
functionsthatintroducedtheinformationintothebrowser.
Thesizeofthevisualizationwindowwaschangedtoautomaticallyfitinthewindowsize.
Alsothesizeofthenodesandthelengthbetweenlinkswasreducedtotrytofitthem
asmuchaspossibleinthewindow.
Thelocationofthesitepackagethatismodifiedforimprovingthevisualizationislocated
in the following path: /Users/EzequielFredes/.virtualenvs/mosaik/lib/python3.5/site-
packages/mosaik_web/html/media
MasterThesisEzequielFredesSáez
29
TheparametersthathasbeenchangedareseeninFigure17:
Figure17:WebVisualizationJavaScriptChanges.[mosaik.js]
Thecolorofthedifferentnodesinthesystem,this isReferenceNode,PQBusesand
Loadswaschangedtohaveamorevisualdifferenceandthenamesinthelegendwas
setcorrespondently.Allofthishadtobedonefromanotherfilecalledmain.css.Thisfile
containstheformatinwhichthevisualizationisshown.Wheneveranewkindofnode
wantstobevisualized,itisnecessarytocreatethenodeinthecssfileandatthesame
timedeclareitintheLVTest.py.Thenodesthathasbeennecessaryfortheprojectand
thecolorsselectedareshowninFigure18
Figure18:WebVisualizationCSSChanges.[main.css]
MasterThesisEzequielFredesSáez
30
Finally,itcanbeseenthatthenodesarefilledinwithsomecolors;thesecolorscanbe
green, yellowor red.According to the instantvalueofanattribute, thecolorof the
insidepartofthenodechangesfromgreen(iftheattributeiswithinrange)toored(if
theattributeisoutofrange).
Originally, the values and attributes of interest are decided in the LVTest.py. So the
intention is not being able to change it at themoment. They are referenced to the
maximumvaluesofpowerallowedinthegrid.
In the future, the script may be modified to be able to change the values in the
configurationfiles.Rightnow,thewaythattheyaresetiswiththecodefromFigure19
inthescript:
Figure19:SettingVisualizationConfiguration.[LVTest.py]
Withallthemodificationsthathavebeenmade,thewebvisualizationhasbeenslightly
improved;inFigure20thenewwebvisualizationcanbeseen:
Figure20:ImprovedWebVisualization.
MasterThesisEzequielFredesSáez
31
3.2.5.-Output.py
Output.pyisascriptcreatedwiththeonlypurposeofautomatizingtherunningofthe
co-simulationandatthesametimeperformvisualanalysisoftheoutputofthetool.
Theobjectiveofthisfileistorepresenttheoutputoftheprogram,thisisthecurvesof
voltageandpowerinthedifferentnodesofinterestsandafterwardsapplythetoolused.
First,ashortcommandiswrittenwiththeideathattheapplicationis launchedfrom
here; so from now on, it is only necessary to run this script to run the whole co-
simulation.
Then, introducingawhile function, the systemwaitsuntil the simulation is done, to
accessthedatabaseandprintthenecessaryinformation.
Inordertoaccessthedatabase,duetothefactthatthefileformatishdf5anewlibrary
hastobeimportedtobeabletogetthedata.Thislibraryiscalledh5py.Aswell,inorder
toplotthedifferentcurves,anotherlibraryhastobeimported.Thenameofthisnew
libraryismatplotlib.
Theoutputof theprogramisperformed intwoways.First,aplot isdonetovisually
watchtheoutputofthedesiredvariable.Thisfigureisautomaticallysavedinthedesired
folder.Afterwards,thenumericalvaluesaresavedinanewexcelfilewiththepurpose
offuturetreatmentandevaluation.
Thecodeusedineachofthevariablestoprinttheplotandsavetheinformationinan
excelfileisinFigure21:
Figure21:OutputPrintingFunction.[Output.py]
MasterThesisEzequielFredesSáez
32
Thiscodejustshowediscopiedforeverysinglevariablethattheuseriswillingtoprint.
TheoutputplotfromthecodecanbeseeninFigure22:
Figure22:OutputPlotExample.
Inchapter4,thetooloutputisvalidatedagainstthevaluesprovidedintheEuropean
LowVoltageTestFeeder.
MasterThesisEzequielFredesSáez
33
4.-EuropeanLowVoltageTestFeederValidationIn this chapter it is verified whether the created co-simulation setup reflect the
EuropeanLowVoltageTestFeedergrid.Thiskindofanalysishastobemadepriorany
kindofexperimentation,toassuretheprecisionofthetool. Inordertoevaluatethe
validityofthetool,thefirstthingthatneedstobedoneiscomparetheresultsobtained
inthetestsimulationwiththedataprovidedinthedocumentationoftheEuropeanTest
Feeder;thiswayitispossibletoevaluatetheaccuracyofthesolution.Beforeanykind
ofanalysisismade,ithastobesaidthatintheEuropeanLowVoltageTestFeeder,the
informationanddataisprovidedphaseperphase,howeverinthetool,onlythethree
phaseelectricpowergridisbeingconsidered.Allresultsfromthesimulationarebeing
storedintoanExcelfile.ResultsofthevalidationarestoredinValidation.xlsxfile.This
excelfilecontainstheinstantpowercalculationofthe2simulatorsusedintheEuropean
LowVoltageTestFeederandalsotheoutputoftheMosaikco-simulationsetupcreated
inthisproject.Thevariablethatiscomparedisthepoweratthesecondarysideofthe
transformerinthesubstation.
First,therealoutputgivenintheEuropeanLowVoltageTestFeederisshowedinFigure
23.ItcanbeeasilyobservedthesimilaritywithFigure22fromchapter3.2.5.
Figure23:RealOutputFeeder.
MasterThesisEzequielFredesSáez
34
Withthisinformation,thedifferencebetweentheFeederpowerandthepowergiven
bytheMosaikco-simulationiscompared.Thisdifferencerepresentedinpercentageis
showedinFigure24:
Figure24:PercentageDifferencebetweenFeederandCo-simulation.
TheaccumulatederrorwascalculatedandcanbeseeninFigure25:
Figure25:AccumulatedErrorinPercentage.
MasterThesisEzequielFredesSáez
35
Theoverallerroroftheenergywasof0.014%.Forthepurposeofthisinvestigation,the
errorisacceptable
OtherimportantcalculationsaretheRootMeanSquareError,whichisof117W.Alow
valuecomparedtothemeanvalueoftheinstantpowerisabove20kW.Thereisalsoa
boxchartoftheerror,inFigure26,thatshowsthatmostofthevaluesareclosetothe
medianwithsomeoutliers.
Figure26:BoxChart.
-200 0 200 400 600 800 1000 1200 1400
BoxChart
MasterThesisEzequielFredesSáez
36
5.-ApplicationThereisabroadrangeofpossibilitiesthatco-simulationwithMosaikbringsinregards
toSmartGridsscenarios.Usersareabletoloadandsimulatehugescenariosandthanks
totheco-simulationpossibility,itisabletoconnectdifferentsimulationinstancesinto
thenetwork;thiscouldbeasimulationofapowerplan,orasimulatorofaveryspecial
consumer.Themost straightforwardapplication that theprogramoffers is real time
monitoringofthegrid,wheretheusercancheckthestatusofanynodeandthehistory
ofit.Thetoolcouldbeusedforadifferentkindofapplications;forexample,tofindand
estimate weak points in the grid where some sort of compensators should be
introduced.Thetoolcouldalsobeusedtoforecastthefutureenergyneedsandenergy
balance.
Inthisproject,variousoptionswerestudied,suchasnetworkrelianceorchallengesof
networkhacking.Ithasbeenselectedthestudyofenergytheftastheapplication.At
themoment,thereislittleadvancedmeteringpracticestoevaluateandstudyenergy
theft.Withtheongoinginstallationofsmartmetersandadditionalmeasurementsinthe
distributiongrid,advancedmethodscanbeusedtoidentifyenergytheft.Thepossibility
ofremotelyidentifyingenergytheftsisaninterestingtopictobestudiedintheproject.
Someauthors,suchasRongJiangetal[9],StephenMcLaughlingetal[10]andSergio
Salinasetal [8]; focus the studyof theenergy theft innetworkattacks in the smart
meters.Thetechniqueusedbythemismorerelatedtocomputersciencesprocedures.
Thetechniquesconsistofclassificationproceduresand/oramathematicalmodelbased
on“honestycoefficients”.Thetoolthathasbeendevelopedinthismasterthesisismore
orientedtopowersystems,soothertechniquesarestudied.
With use of prepared co-simulation setup, it is possible to automatically generate a
networklayoutandvisualizeit,andatthesametime,introducenewpowernodesinto
the system. During the first phase of this project, only loads or consumers where
introduced,but it ispossibletointroduceotherkindsofnodessuchasgeneratorsor
compensators.Afterallthisdataandinformationareautomaticallyintroducedinthe
simulator,itispossibletoobservetheoperationofthepowersystem.
MasterThesisEzequielFredesSáez
37
Alltheoutputfromthesimulatorissavedintoadatabasethatafterwardsissavedand
couldbeused for future studies. Finally, studying theenergy theftspossibilitieswas
chosenasthemostappropriateapplicationwiththegivingbackground.Theapproach
choseninthisworkistousehistoricalmeasurementstoidentifyenergytheft.
Energytheftrepresentapercentageofthelosesthatarepresentedintoday’spower
systems.Asthisactivityatthemomentdoesnotrepresentarisktothestabilityofthe
grid, thepurposeof the tool is tohave real timemonitoringof thegridbutwithout
evaluating any kind of energy theft. Afterwards, with the information from the day
before,thetoolisruntoevaluatewhetheralltheconsumersinthenetworkwerehonest
orifthereisanykindofenergytheftintheareaandsubsequentlyidentifythetheft.
Thenextsectionintroducesenergytheft,explainshowitoccursandfinallydescribes
differentprocedurestodetectit.
5.1.-EnergyTheft
Energy theft consists of the illegal, unsupervised and unauthorized use of electricity
fromthegrid.Themainreasonswhyenergytheftisanissueare:
• Safety:Unsupervisedusecancauseelectrocutionsorfires:
• Economical:Thisenergystolenismoneynotpaidtothegridadministrator.
Themostcommontechniquesforenergytheftare[7]:
1. Hiddenconnectioninfrontofthemeter.
2. Physicaltamperingwiththeterminals.
3. Physicaltamperingwithananalogmeter’smechanism,suchasamagneticfield.
4. Tamperingwithadigitalmetersoftware.
There are different ways and techniques the energy companies have nowadays to
evaluateenergy theft. In the future,with thepossibilityofhaving smartmeters, the
electricalcompanieswouldbeabletodomoreadvancedevaluationoftheenergytheft.
MasterThesisEzequielFredesSáez
38
However,thecompaniespossessingandusingtheinformationoftheloadprofileofthe
smartmeters,posesaconcernregardingprivacyofthecitizens.
Thecompanycouldmakeuseofthisprofilesandforwardittothirdpartiesthatcould
haveeconomicinterestsonit[8].
One common technique used to find energy thefts is using the so called honesty
coefficients[9].Withthesecoefficientsitcouldbeevaluatedovertimeifaconsumeris
moreapproachedtobeanenergytheft.Theadvantageofthistechniqueisthatisnot
necessary tohave themeasurementsof thesmartmeter inorder to identifyenergy
theft;thisway,privacyoftheconsumersisassured.
Another way is observing all the different variables at the same time and all the
possibilitiesofenergytheftatonce[10].Thissystemavoidshavingahighnumberof
falsepositivesbuttheamountofinformationnecessarytocalculateitmightbedifficult
tohandleandanalyze.
InthisworkwewillinvestigateandmodifyamethodproposedbyP.Kadureketal[11]
andevaluateitwithuseofdevelopedco-simulationandrealresidentialconsumption
date.
5.2.-EnergyTheftTool
TheprocedureproposedbyP.Kadureketal[11]consistsoftwodifferentsteps:
1. Detection of illegal use of electricity. In order to determinewhether there is
some energy theft in the system, an evaluation procedure is done at the
substation level. If thereexistsanenergyunbalanceat thesubstation level,a
sequencetoidentifythelocationofthetheftisinitialized.
2. Theftlocalizationprocedure.Inthisstep,theauthorassumesthatthevaluesof
thecurrentmeasuredbythesmartmetersiscorrectduetothetheftprocedure
that is evaluated. However, the values of the voltages could not match the
measuredcurrent.Therefore,withthemeasuredcurrentfromthesmartmeters,
theestimatedvoltagesatthelocationarecalculated.Ifthereexistadifference
betweenthesetwovalues,itisidentifiedtheenergytheft(s).
MasterThesisEzequielFredesSáez
39
ThismethodissummarizedinFigure27:
Figure27:PaperMethodSummary.[11]
MasterThesisEzequielFredesSáez
40
Themethodthat isusedinthismasterthesis ishavinganenergybalancestepanda
secondarystepofhistoricaldataofthedifferentloads.Thismethodissupposedtobe
runonceadaywiththedatafromthedaybeforeandthehistoricaldataoftwoyears.
Itisnotaverycompletetechniquebutitcannotbestudiedforwardinthisprojectdue
tothelackofinformationfromothervariablesthatcanbestudiedinenergytheft.The
tool is divided in two steps. First it is trained to see if it is able to detect whether
somebody is stealingenergy.Afterwards, it isprogrammedtosee if it isable to find
wheretheenergythefttakesplace.Thesetwostepstogetherconsistofsevendifferent
phasesthataredescribedbelow.
Aflowdiagramcanbedrawntoexplaintheproposedmethod,showninFigure28:
Figure28:FlowDiagramtoevaluateEnergyTheft.
Asaresult,theonlywayofenergytheftthatisevaluatedisphysicaltamperingofthe
metersattheconsumers’dwelling.
MasterThesisEzequielFredesSáez
41
5.2.1.-Powerbalance
Themethodproposedinthisprojectistohavetheloadsmeasuredineachofthesmart
metersandloadflowiscalculatedfromthedata.Theresultfromtheco-simulationis
comparedtotheactualpowerthatismeasuredinthesubstation.
Phase1ofthetoolconsistonreceivingallthemeasurementsfromthesmartmeters.
Theco-simulationisperformedinPhase2andisstructuredthefollowingway;firstthe
toolisgivenallthemeasurementsfromthesmartmetersofalltheconsumersinthe
networkplusthemeasurementsatthesubstationlevel.Oncethetoolhasbeengiven
alltheinformation,itintroducesallthemeasurementsintothesimulationasconsumers
intheexactplacetheyarelocated,andthenasimulationiscarriedout.
Oncethesimulationisrun,theprogramhasallthepowerflowthroughthedifferent
linesinthesystemandasaresult,thepowerflowingfromthesubstationintothegrid.
Thispowerflowisthemostinterestingfortheapplicationintheveryfirststep.Thetool
hastocomparetheenergyconsumedduringthedaythathasbeencalculatedthrough
thetool,withtheonethathasbeengivenasdataasitwasmeasuredinthesmartmeter
connectedthere.Ifthesetwomeasurementsaresimilarwithinathresholdvalue,itis
determinedthatthereisnoenergytheftinthenetwork.However,ifthedifferencein
energies is higher than the threshold value, it is further studied if there is someone
stealingenergyfromthegrid.ThiscomparisonisdoneduringPhase3andthethreshold
valuehasthenameofδ1.
Thevalueofthethresholdneedstobeselectedcarefully,asithasaninfluenceonthe
accuracyofthedevelopedalgorithm.Butthequestionremainthere,howcanaproper
thresholdvaluebeselected?Theapproachfollowedtoselectthebestthresholdvalue
isexplainednow.First,a“standard”gridhad tobeselected; itwasdecided thanan
amountof75consumers isanaveragenumberof consumers inanetwork this size.
Then, a theftwas introduced in the simulationatdifferent locations in thegrid; the
networkconsistsof906nodes,so itwasconnectedatnodenumber1,300,600and
900. For each of the co-simulations, the percentage difference between the energy
consumedandthetheoreticalenergythathadtobeconsumedismade.
MasterThesisEzequielFredesSáez
42
From the results in all the simulations, only the one with the worst percentage
differenceisbechosenasareferenceforstudyingenergytheft.Theresultthathasbeen
seen,isthatapercentagedifferenceof1%istheonethatbestsuitstheapplication.If
thepowerbalancethresholdisreached,sometechnicalornon-technicallosesoccurred.
Technical losescan includefaults,changes inpowersystemtopology,measurements
error. Non-technical loses include connection or disconnection of loads and/or
productionthatarenotreflectedinthemodel;oneexampleofnon-technicallosesis
energytheft.Thedistinctionbetweenthesetwodifferentkindoflosesisperformedin
Phase6and7
InsidePhase3thereisaniterativemethod.Thepowergridisdividedintosubgridswith
measurementsonthepointofthecommoncoupling.
In the iterative process the same energy balance is performed as before, but on
subsystems.Thesubsystemisdividedintwolevelsof2subsystems,soitcomparesthe
energybalance in the2 subsystemsandevaluatewhether there is imbalance in the
subsystem.Inthecasethatinthesubsystemthereisanimbalanceaswell,2moresub
subsystemsarestudied.Thiswaytheareawheretheenergytheftistakingplacecanbe
evaluated more precisely. In order to perform this kind of monitoring on real life
equipment,metermeasurementsareneeded,notonlyintheconsumernodes(smart
meters),butalsoonthedifferentlinesthatareevaluatedascriticalorimportant.The
reasonwhythesystemsaredividedintwobytwoisthatthepowerlinescanonlyhave
a divisionof twoat a time.Adivision in three is not allowed. The additional design
decisionisachoiceoflinesthatareconsideredcritical.Ithasbeendecidedthatthebest
wayofdefiningthesubsystemsandsubsubsystemsisfirstbytheline’sgeographical
locationandnextdistributingthenodesasevenlyaspossibleintosubsubsystems.
In the Mosaik visualization, geographical location is not depicted; however, in the
EuropeanLowVoltageTestFeedercontainsthelocation.
Dueto timeconstraintsandthe fact thatunderstandingandmanipulating JavaScript
filesisnotthepurposeofthisproject,anotherprogramwasusedtovisualizethenodes
andthepositionofthelines.ThisprogramiscalledQ-GIS.Thewaythisprogramworks
isthat,twolayershavetobeinsertedintheprogram.
MasterThesisEzequielFredesSáez
43
Oneforthepositionofthenodesandoneforthestartingnodeandtheendingnodeof
eachofthelines.Thiswaynowitispossibletoseewhichofthelinesiseachone.
TheendingresultthatcanbeseenintheprogramisinFigure29:
Figure29:Q-GIS.
ThefinalshapeofthenetworkconsideringthesubsystemsisshowninFigure30:
Figure30:SystemsDesign.
Therearetwomainsubsystemsselected.Subsystem1beginsinLINE141.Thesecond
subsystemisbiggerandbeginsinLINE110.
MasterThesisEzequielFredesSáez
44
Both subsystems are divided into two more sub subsystems; however, the first
subsystem is smaller than the second subsystem, so in order to graphically see the
difference,inFigure30onlythesubsubsystemsfromthesecondsubsystemareshown.
Insubsystem2therearetwosubsubsystems.Subsubsystem1beginsinLINE302;sub
subsystem2beginsinLINE296.
5.2.2.-NeuralNetwork
Thesecondpartofthetool,findinphase4and5.consistsofaNeuralNetworkthat
evaluates whether there has been a significant change in the consumption of the
houses.InordertotraintheNeuralNetworkatrainingdataandvalidationsetofdata
havetobeavailable.BeforeexplaininghowtheNeuralNetworkhasbeendesigned,itis
interestingtoexplainwhataNeuralNetworkisandhowitworks.AnartificialNeural
Networkisasystemthat is inspiredbythebiologicalneuralnetworksthatareinthe
animals’brains[12].Theyareusedtoprojectfunctionswhicharedependentofacertain
numberofinputs.Thewaythemodelprojectsorapproximatesthedifferentoutputsis
donewithdifferentkindsoflearningalgorithms.Thelearningalgorithmusedinthistool
isthefeedforward.Thisisthesimplestofallthelearningalgorithmsasthedataonly
movesforwardanddoesnotmakeanykindofloops.Intheneuralnetwork,itispossible
toselectthenumberofneuronsthatarenecessaryandthenumberofhiddenlayers
thattheuserwishestohave.Figure31isanexampleofthearchitectureofaNeural
Network.
Figure31:ArchitectureofNeuralNetwork.[14]
MasterThesisEzequielFredesSáez
45
Inthisproject,anartificialNeuralNetworkisusedtomodeltheindividualhomeenergy
consumption.TheinputsthataregiventotheNeuralNetworkarethemonth,theday
inthemonthandthetemperature.Theoutputthatisexpectedfromthesystemisthe
energythatshallbeconsumedintheday.
During the project it has been investigated different artificial Neural Network
architectures;inordertoobtainthemostaccurateresultsaspossibleandareasonable
computationaltime.
Alltheinputdataisintroducedinayearlybasis;therefore,theideaistohaveonewhole
yearofdatainordertotraintheNeuralNetwork,anotheryearinordertovalidateit,
andfinallyuseitinthedayofthecomingyearwheretheenergycalculationisneeded.
TheNeuralNetworkcheckisdonethesamewaythepowerbalancecheckisdone,by
selectingthethresholdvalueδ2.Thisisappliedinthephase5.Themethodchecksthe
percentage difference between the given value in the smart meter and the value
estimatedbytheNeuralNetwork.DuetothefactthattheNeuralNetwork is justan
approximationof the reality and that thedifference that is introducedwhena theft
comes intoplay is50%oftheoriginalprofile.Thethresholdtodetermine ifa load is
stealingenergyhasbeensettoaδ2of50%.Alargethresholdvaluehasbeenchosen,
butwithmoreaccuratemodelsofthebehavioroftheloadsitcouldbereduced.
MasterThesisEzequielFredesSáez
46
6.-ExperimentsDuring theprojecta seriesofexperimentshavebeenperformedwith theuseofco-
simulationtoolandproposedenergytheftmethod.
Prioranyexperimentation,areconfigurationofthetoolisneededinordertoperform
the needed experiments. The scenario that is implemented in the experimentation
consistof75housesconnectedinthenetworkobtainedintheEuropeanLowVoltage
TestFeeder.Thesehousesareconnectedevenlyinthesystem.
The75housesarecreatedwith18differentloadprofiles(obtainedfrompecanstreet)
withanaverageenergyconsumptionof23kWheach.Theoverallenergyconsumedin
thesystemisaround1750kWh.
Inthisnetworkconfigurationthetheftsaregoingtobeplaced.Theftsarecreatedwith
the same 18 load profiles as the houses; with the difference that the energy
consumptionisonlyhalfoftheoriginalloadprofile.Furthermore,whenatheftappears
in the configuration, what is expected to be seen is that the energy consumption
measured is less than the one calculated. Therefore, the thefts have to reduce the
energyconsumptionmeasuredinthesmartmeters,inresult,theyareintroducedinthe
systemasnegativevalueoftheloadprofile.Thismeansthatthetheftsactintheco-
simulationasifaproducerisconnectedtothenode.
Inthissectionthesubchaptersaredividedthefollowingway:
• Chapter6.1:Overviewofthereconfigurationinthetool.
• Chapters6.1.1to6.1.6:Detailedexplanationofthemodificationsinthetool.
• Chapter 6.2: Presentation of the results in the experiments and further
calculations.
6.1.-ChangesintheTool
Thesourceofdatathatwaschosenforthisprojectwaspecanstreet[13];therefore,the
tool had to be reconfigured to fit the new data format. All the code that has been
explainedandshownbeforeisthecodethatwouldbeusedinordertoworkwith,as
theideaisthatalltheinformationcomesfromtheEuropeanLowVoltageTestFeeder
orequivalent format.All thedataregardingthepowerconsumptionandtheoutside
temperatureisprovidedbypecanstreet[13]maintainingthesamegridlayout.
MasterThesisEzequielFredesSáez
47
Firstithastobeexplainedhowthedifferentscriptshavebeenrewrittentoperformthe
proposedscenario.
Theoverviewofthedifferentchangesthatareperformed,willbeexplainedwiththe
sameFigure8fromchapter3.2.1thatwasusedtoexplainthedevelopmentofthetool
highlightingthechanges.
ThenewoverviewisshowninFigure32:
Figure32:OverviewoftheProgramwithChanges.
Thesimulationoverviewisverysimilarastheonepresentedinchapter3.Insideofthe
scriptssomechangeshavebeenmade(thatareexplained in thischapter).Themain
differenceisthatnow2newfileshavebeencreated.NN.pyisapythonscriptthathas
to be run separately from the Mosaik tool and calculates the expected energy
consumption of the nodes in the desired day. Comparator.xlsx is an Excel file that
gathersalltheresultsfromthedifferentsimulatorsandvisuallydefineanddiscoverthe
theftsintheregion.Intheoverviewtherunprocessisdefinedfor2differentscripts,
Output.pyandNN.py.Bothscriptsarenotconnectedsotheyhavetoberunbeforeany
resultscanbeseenbuttheorderoftheprocessdoesnotaffecttheresult.
MasterThesisEzequielFredesSáez
48
Inordertoperformtheexperiment,scenariodesignermustbeabletointroducethefts
inthenetwork.Inchapter5itwasalreadydefinedthatthetheftdiscoverywasgoingto
be done comparing the theoretical energy consumption with the real energy
consumption.Thetheoreticalenergyconsumptioniscalculatedatthesubstationlevel
withthesimulationofthemeasurementsinthesmartmetersandinthehouseswith
theNeuralNetwork.Therealenergyconsumptioniswhatactuallyhasbeenreadbythe
smartmeters.
Itwasalsodiscussedinchapter5thattheenergytheftmethodthatisbeingstudiedis
physicaltamperingofthemeters.
In conclusion, the difference that should be expected in the measurements and
calculations is that therealenergyconsumption is lowerthanthetheoreticalenergy
consumption.Therefore,inordertosimulateatheft,theloadprofileusedisthesame
asintheloadsbutdecreasedby50%.Thistheftprofileisacting,notasaloadbutasa
producer;sotheeffect isthesameoneas ifaphotovoltaicpanelorabatterywould
havebeenconnectedinthenode.
Inordertoruntheproposedscenario,itisneededtoloadtheprofilesofthehousesin
MosaikandPyPower.Todoso,thefirstExcel.pythatwasusedissavedasacopyand
renamedafterExcel_Validation.py;thereasonforthischangeisthattheoriginalscript
wasusedtovalidatetheresultsfromthefeeder.Now,smallchangesaremadeinthe
Excel.pyfiletointroducethenewloadprofilesintothesystem.
The new configuration consists of 18 different houses profiles, that can generate
whatevernumberofhousesasneeded,connectedevenlytothegrid.Theideaforthe
powerbalancecheck is that firstasimulationwiththe loadprofiles fromthehouses
(theoreticallymeasured in the smartmeters) is performed. Then, the total power is
comparedwiththeactualpowermeasuredinthesmartmeter.However,therealpower
flowing from the substation is not a variable that is available. Therefore, a new
simulationisperformedwithavariablenumberofenergytheftstoobtainthepower
flowfromthesubstation.
MasterThesisEzequielFredesSáez
49
Inordertohavethedatafromthetheoreticalpowerflowandbeingabletocompareit
withtherealpowerflow,thedatabase(hdf5file)fromthefirstsimulationiscopiedand
savedwithanothername,LVTheory.hdf5,andthevaluesfromthisfilearecompared
withtheonesobtainedintheLVTest.hdf5.
6.1.1.-Var.py
Thisscriptisoneoftheleastaffected,theonlythingthatisneededtoaddisthenumber
ofloadsandtheamountoftheftsthatareinthescenario.
Forthepurposeofthisinvestigation,severaltestswithdifferentamountoftheftsneed
tobeperformed
ThewaythattheyaredefinedisshowninFigure33:
Figure33:LayoutDefinition.[Var.py]
6.1.2.-Excel.py
Duetothefactthatthesamegriddistributionisused,thereisnoneedtochangeallthe
coderegardinginthisfile.TheloadswerecreatedfromtheEuropeanLowVoltageTest
Feeder,however,intheexperimentationpart,itwasdecidedtouseanyresidentialload.
Therefore,newloadsimulatorshavetobedeveloped,andthis isdonewiththedata
downloadedfrompecanstreet.
Inordertoincludenewloadsintheco-simulation,csvsimulatorisinitializedwithdata
downloadedfrompecanstreetdatabase.
However,aslightmodificationhastobemade.Aswitchcasehasbeenintroducedinthe
functiontoallowittodifferentiatebetweenloadsandthefts.Whentheprofileistobe
savedasaload,itisjustsavedthewayitis.Ontheotherhand,whentheprofileisa
theftprofile, the instantpowerthat issaved intheprofile isonlyhalfof theoriginal
power,plusitischangedtoanegativevalue.Whichmeansthattheenergyconsumption
atthenodeswheretheyareconnectedisgoingtobelower.
MasterThesisEzequielFredesSáez
50
ThesourcecodeusedisshowninFigure34:
Figure34:ChangesinRearange_csvFunction.[Excel.py]
Thereare18differentprofilesareavailableandthetheftshavethesameloadprofiles
asthehouses.Thefunctionabovehastobecalledforeachhouseandforeachtheft,as
showninFigure35:
Figure35:HousesandTheftsLoadProfile.[Excel.py]
TherestoftheExcel.pyfileremainsunchanged.
6.1.3.-LVTest.py
Thisscriptisthemostimportantforthereconfiguration,becauseitiswhereMosaikand
simulatorsareinitialized.Themaindifficultyduringtheexperimentsistosetcorrectly
theexactnumberofhousesandtheftsthatmustbestudied.Inordertodoso,different
situationshavetobeanalyzed.
Forexample:
• Howitshouldworkwhentheusersetsnoloads?
MasterThesisEzequielFredesSáez
51
• How should itworkwhen it sets less loads than the amount of profiles that
exists?
• Howtosettheloadswhenhavingabiggernumberthantheamountofprofiles
thatexists?
Thisfunctiondevelopedtosolvethesequestionsisexecutedwiththesourcecodefrom
Figure36:
Figure36:CreatingVariableNumberofLoadsandThefts.[LVTest.py]
Thesolutiontocreatemorehousesthanprofilesistocreateahighernumberofloads
withthesameprofileandthenconnectthemindifferentplaces.Thissolutionapplies
exactlythesamewaytothethefts.However,insomesituationsitisdifficulttocreate
exactly the number of loads the user defines. So two “if” statements have been
developedwhencreatinganumberofloadswiththesameprofile.
MasterThesisEzequielFredesSáez
52
Inthefirststatement,thenumberofloadsissimplytheentirepartofthedivisionofthe
numberofhousesdividedbythenumberofprofilesplusone.Ifatsomestagecreating
thisnumberofloadswouldoverpassthenumberofdesiredloads,lessloadsshouldbe
created.Originallytheideawastocreatethemonebyone.Butforbiggerscenarios,it
isneededtocreatemorethanonebutlessthaninthecalculationbefore.Thenumber
ofloadscreatedisthenumberofremainingloadstobecreateddividedbythenumber
ofprofilesnotusedsofar.
Nowtheonlythingmissingisconnectingthehousesandtheloads.Asintheprojectonly
mechanicaltamperingofthesmartmetersisconsideredasanenergytheftpossibility,
thetheftsareonlyconnectedinthesamenodesastheloads;therefore,itisnotpossible
tocreatemoretheftsthanloads.Inordertohavethetheftsasevenaspossible,they
areconnectedatevendistancesbetweeneachother.
ThisisdonewiththesourcecodefromFigure37:
Figure37:ConnectingLoadsandThefts.[LVTest.py]
Inthisscript,oneofthesimulatorsthatisinitializedisthewebvisualizationsimulator.
Forexperimentationpurposes,theftshavebeenintroducedinthevisualizationasanew
kind of node. This way it is possible to graphically check that the thefts have been
created.
MasterThesisEzequielFredesSáez
53
Therefore, this nodewas introduced in theweb visualization andnow thewebpage
lookslikeinFigure38:
Figure38:WebVisualizationChanges.
Asitisdifficulttoseethethefts,azoomfigurehasbeentakenaswellinFigure39:
Figure39:ZoomedWebVisualization.
MasterThesisEzequielFredesSáez
54
6.1.4.-Output.py
Theoutput.py script isusedtoeasetheanalysisof theexperiments results.Thetest
resultsandthetheoreticalresultshavebeencomparedusingExcel;sotheinformation
fromthetheoreticalpowerflowandtheactualpowerflowcanbefoundintheexcel
file;introducedthroughpythonandafterwardsitautomaticallyupdatesthevaluesand
givestheuserthedifferenceinpowerconsumption.Inordertodoso,thescriptread
thedatabase(hdf5file),andthenintroduceallthedataintoExcel.
ThesourcecodeusedforwhathasbeenexplainedsofarisshowninFigure40:
Figure40:Hdf5ReaderandWriter.[Output.py]
NowthatallthedataregardingthesubstationlevelhasbeenintroducedintotheExcel
file,thedataregardingtheLinesisintroducedaswell.AverysimilarfunctiontoFigure
40 isdeveloped;withthedifferencethat theonebeforeonlyconsideredthe instant
powermeasuredeveryminuteandthisonegivesonlytheenergythathasgonethrough
thelineduringtheday.Thesourcecodetointroducetheinformationfromthepower
linesispresentedinFigure41:
Figure41:LinesReaderandWriter.[Output.py]
MasterThesisEzequielFredesSáez
55
ThelastpieceofinformationthathastobesavedintoanExcelfileistheonerelatedto
thenodes.Itisneededforeachoftheloadstodefine,whereitislocated,whichhouse
profilewastheoneusedtocreatetheloadandtheenergyconsumptionfromtheday.
ThesourcecodeofthefunctionisinFigure42:
Figure42:NodesReaderandWriter.[Output.py]
AverysimilarfunctionfromFigure42isdevelopedfortheexperimentationpurpose.
ThisfunctionshowsthelocationofeachofthetheftsandstoresthenodeintoanExcel
file.
ThisfunctionisseeninFigure43:
Figure43:TheftsReaderandWriter.[Output.py]
Theoutputofthesimulationissavedinto2differentExcelfiles;onecalledTheory.xlsx
andonecalledLVTest.xlsx.
MasterThesisEzequielFredesSáez
56
6.1.5.-NN.py
TocreatetheNeuralNetworkanewscriptcalledNN.pyhascreatedandanewlibrary
wasimportedcalledneurolab.
Itwas decided to create aNeural Networkwith the information from awhole year
(2013),validateitwiththeinformationofanothercompleteyear(2014)andthenapply
ittoadayofanewyear(1stofJanuary2015).
Alltheloaddataistakenfrompecanstreet;however,theformatisdifferentfromthe
expectedbytheNeuralNetworktrainingfunction.Alltheinformationhastoberead
andstoredcorrectlyintoamatrixsothatitisreadyforprocessing.Thisisdonewiththe
sourcecodefromFigure44,togetall the informationfromtheweather.Exactlythe
sameisdonetogetalltheinformationfromtheenergyconsumption.
Figure44:DataReadingFunction.[NN.py]
Nowthatallthedataisstoredintodifferentvariables,inordertoeasethework,ashort
function is written to save this information correctly formatted into csv files. This
functionismeanttoberunonlyonce,firstbecausethetimeneededtoexecuteover18
housesisquitelongandsecondlybecauseitisnotnecessarytoreadthefilesanymore
iftheyarestoredseparately.
MasterThesisEzequielFredesSáez
57
The source code that goes through each of the houses and store the information
correctlyformattedintocsvfilesisseeninFigure45:
Figure45:NeuralNetworkDataPreparation.[NN.py]
Once all the information is correctly formatted and can be easily read, the Neural
Networkcanbeconstructed.TheNeuralNetworkinputsare:themonth,thedayand
theoutsidetemperature.Andtheoutputistheenergyconsumptionoftheday.
Thefact isthattheenergy isahugevaluecomparedwiththeday,themonthorthe
temperature.Therefore,inordertomaketheNeuralNetworktraining,itisnecessary
tonormalizethedatabeforeitisusedtotraintheneuralnetwork.
Thenormalization isperformedwiththesourcecodefromFigure46;this function is
abletobothnormalizeandunnormalize.
Figure46:NormalizeFunction.[NN.py]
MasterThesisEzequielFredesSáez
58
The normalized data from 2013 is used to train the Neural Network. Validation is
performed with another set of data from 2014. The Neural Network has been
introduced ina for loop,so that itcreatesaNeuralNetwork foreachkindofhouse,
calculatestheerrorforitandtheexpectedenergyconsumptiononthedaytostudy.A
separateNeuralNetwork is trainedforeachhouseprofile.Theerror foreachNeural
Networkandthepredictionvaluesforeachhouseprofilehasbeenstoredforfurther
analysis.AlloftheaboveisexecutedwiththelistingfromFigure47:
Figure47:NeuralNetworkPrototype.[NN.py]
It isused20neuronsper layerand3hiddenlayers;thereasonforsuchanumberof
layersisfortheprogramtobeabletoreachthegoalof1%errorinthetrainingofthe
NeuralNetwork;theprogramtakesalmost1hourtorun.Unfortunately,thissetupover
fitstheNeuralNetwork,theerrorcalculatedwiththevalidationsethasabigvalueinall
thehouseprofiles.AnotherNeuralNetworkisdevelopedwithonly2hiddenlayersand
with3and6neuronsrespectively.ThisNeuralNetworkdoesnotreachthegoalinthe
trainingphaseoftheNeuralNetwork;however,theerrorsmadewiththevalidationset
arevisiblylower.ThisnewNeuralnetworkisshowninFigure48:
Figure48:NeuralNetworkDeveloped.[NN.py]
MasterThesisEzequielFredesSáez
59
Forthepurposeofthisinvestigation,theNeuralNetworkfromFigure48istheonethat
isused.
All the information from the Neural Network, which are house id, percentage of
validation in the Neural Network and expected energy consumption for the day is
storageintoanexcelfilecalledNN.xlsx.
6.1.6.-Comparator.xlsx
AnewExcelfile,namedComparator.xlsxiswherethedatafromalltheExcelfilescreated
throughthesimulationareloaded(LVTheory.xlsx,LVTest.xlsxandNN.xlsx).
InthisnewExcelfile,theusermaychoosefreelyatwhichpercentagedifferenceit is
consideredthatthereissomeonestealingenergybothinthepowerbalanceandinthe
NeuralNetwork;theseareδ1andδ2respectively.
TheoverviewisshowninFigure49:
Figure49:SubstationComparator.[Comparator.xlsx]
Ifenergytheftisdetected,thefollowingsubmethodwilloperateandlookforatheftin
thedifferentpowersystemssubsystemswiththesameδ1asbefore.Theoverviewis
seeninFigure50:
Figure50:SubsystemsComparator.[Comparator.xlsx]
MasterThesisEzequielFredesSáez
60
Finally, it is needed to check whether there has been a change in the energy
consumption of the houses compared to previous years; this is done with the
informationfromtheNeuralNetwork,thischeckisonlydoneifthehouseislocatedin
anenergytheftsubsystem.
TheresultcanbeseeninFigure51:
Figure51:NNComparator.[Comparator.xlsx]
For the purposeof the investigation, also the reality ofwhere the thefts havebeen
locatedisseenalsoinFigure51.Thiswaythecalculationoftheprecision,accuracyand
recallofthetoolcanbedonefaster.
6.2.-ExperimentsScenarios
Nowaseriesofexperimentsareexecutedtotesttheproposedenergytheftmethod.In
thismasterthesis4mainscenariosaredescribed;whichare0thefts,5thefts,10thefts
and15theftsinatotalof75houses.
6.2.1.-Experiment0Thefts
Inthisscenario,noadditionaltheftsareintroducedinthenetwork.Thisscenarioisa
basecase;itisusedasacomparatortootherscenarios.
6.2.2.-Experiment5Thefts
Inthisexperiment,5theftsareintroducedevenlyinthegrid.Thetotalamountofenergy
thatisstoleninthescenarioisaround40kWh.
MasterThesisEzequielFredesSáez
61
TheoverviewoftheresultsintheexperimentcanbeseeninFigure52:
Figure52:Experiment5Thefts.[Comparator.xlsx]
6.2.3.-Experiment10Thefts
In this experiment, 10 thefts are introduced evenly in the grid. The total amount of
energythatisstoleninthescenarioisaround120kWh.
TheoverviewoftheresultsintheexperimentcanbeseeninFigure53:
Figure53:Experiment10Thefts.[Comparator.xlsx]
MasterThesisEzequielFredesSáez
62
6.2.4.-Experiment15Thefts
In this experiment, 15 thefts are introduced evenly in the grid. The total amount of
energythatisstoleninthescenarioisaround170kWh.
TheoverviewoftheresultsintheexperimentcanbeseeninFigure54:
Figure54:Experiment15Thefts.[Comparator.xlsx]
6.3.-ExperimentsResults
Theresultsfromtheexperimentsareexpressedintheoverviewjustbelow.Thereare3
differentlevelsofidentification.Oneintheentiregrid,oneinthesubsystemandthen
thelastoneatthenodelevel.Thefirsttwolevelsaredonebasedonpowerbalance.The
lastlevelisdonebasedontheNeuralNetwork.Basingthetruthofthefindingsonlyin
thelastlevelisnotaccuratetodeterminethestatisticalresultsofthetool.Inorderto
present the accuracy, the precision and the recall of the tool a confusionmatrix is
developed for each of the levels. Precision defines the variability of the tool, while
accuracydescribestheerrorobserved.Recallisavariablerelatedwiththesensitivityof
thetool.
MasterThesisEzequielFredesSáez
63
6.3.1.-NetworkLevel
TheconfusionmatrixisshowninTable1:
Experiment TruePositive FalsePositive TrueNegative FalseNegative
0Thefts 0 0 1 0
5Thefts 1 0 0 0
10Thefts 1 0 0 0
15Thefts 1 0 0 0
Total 3 0 1 0
Table1:ConfusionMatrixGridLevel.
Withthefiguresfromthetable,itispossibletocalculatetheprecisionandtheaccuracy
ofthetool.
Theaccuracyiscalculatedwiththefollowingformula:
!""#$%"& = ($#)*+,-.-/) + ($#)1)2%.-/)*+,-.-/) + 1)2%.-/) = 4
4 = 100%
Theprecisioniscalculatedwiththefollowingformula:
*$)"-,-+7 = ($#)*+,-.-/)($#)*+,-.-/) + 8%9,)*+,-.-/) =
33 = 100%
Therecalliscalculatedwiththefollowingformula:
;)"%99 = ($#)*+,-.-/)($#)*+,-.-/) + 8%9,)1)2%.-/) =
33 = 100%
6.3.2.-SubsystemLevel
TheconfusionmatrixisshowninTable2:
Experiment TruePositive FalsePositive TrueNegative FalseNegative
0Thefts 0 0 7 0
5Thefts 4 0 3 0
10Thefts 5 0 2 0
15Thefts 6 0 1 0
Total 15 0 13 0
Table2:ConfusionMatrixSubsystemsLevel
MasterThesisEzequielFredesSáez
64
Withthefiguresfromthetable,itispossibletocalculatetheprecisionandtheaccuracy
ofthetool.
Theaccuracyiscalculatedwiththefollowingformula:
!""#$%"& = ($#)*+,-.-/) + ($#)1)2%.-/)*+,-.-/) + 1)2%.-/) = 28
28 = 100%
Theprecisioniscalculatedwiththefollowingformula:
*$)"-,-+7 = ($#)*+,-.-/)($#)*+,-.-/) + 8%9,)*+,-.-/) =
1515 = 100%
Therecalliscalculatedwiththefollowingformula:
;)"%99 = ($#)*+,-.-/)($#)*+,-.-/) + 8%9,)1)2%.-/) =
1515 = 100%
6.3.3.-NeuralNetworkLevel
TheconfusionmatrixisshowninTable3:
Experiment TruePositive FalsePositive TrueNegative FalseNegative
0Thefts 0 0 75 0
5Thefts 3 35 35 2
10Thefts 7 34 31 3
15Thefts 9 32 28 6
Total 19 101 169 11
Table3:ConfusionMatrixNeuralNetworkLevel.
Withthefiguresfromthetable,itispossibletocalculatetheprecisionandtheaccuracy
ofthetool.
Theaccuracyiscalculatedwiththefollowingformula:
!""#$%"& = ($#)*+,-.-/) + ($#)1)2%.-/)*+,-.-/) + 1)2%.-/) = 19 + 169
19 + 101 + 169 + 11 = 62,7%
Theprecisioniscalculatedwiththefollowingformula:
*$)"-,-+7 = ($#)*+,-.-/)($#)*+,-.-/) + 8%9,)*+,-.-/) =
1919 + 101 = 15,8%
Therecalliscalculatedwiththefollowingformula:
;)"%99 = ($#)*+,-.-/)($#)*+,-.-/) + 8%9,)1)2%.-/) =
1919 + 11 = 63,3%
MasterThesisEzequielFredesSáez
65
6.4.-ExperimentConclusions
Fromtheresultsoftheexperiments,itcanbedeterminedthatthepowerbalancestep
inthetoolisworkingwithanexcellentprecision,accuracyandrecall.Thispartofthe
toolisabletoidentifycorrectlythesituationswhereenergytheftoccurs
RegardingtheNeuralNetwork level, itcanbeseenthatthetool isnotquiteprecise,
whileitdoesbetterwiththeaccuracyandrecall.Themainreasonforthisbadprecision
isthattheNeuralNetworkprovidesalargenumberoffalsepositives.Thisisbecause
theNeuralNetworkhasnotbeencompletelyoptimized.Thedatathathasbeenselected
for it to be trained and afterwards tested has not been appropriate enough.More
knowledge about the dataset, aswell asmore inputs that are related to the power
consumptionofthehouseswouldhaveleadintoamoreprecisetool.
MasterThesisEzequielFredesSáez
66
7.-ConclusionIn this last chapter of the master thesis; first an overview of the project is done.
Afterwards,somerecommendationsonhowtousethetoolandfurtherimprovement
thatcouldbemade.Finally,thedifferentgoalsthatweredescribedatthebeginningof
theprojectareenumeratedanddescribedwhomuchtheyhavebeenfulfilled.
Atthebeginningofthismasterthesis, itwasdecidedtodevelopaco-simulationtool
thatcouldbeabletodealwithlargesmartgridsscenarios.Theco-simulationmanager
chosenfortheapplicationwasMosaik.Mosaikcamewithademoofhowitworks,and
usingitasthestartingscenario,ithasbeenimprovedtoachievetherequirementsof
theproject.Once theco-simulation toolwasoptimized.Thechoiceofan interesting
application was another step of this investigation. We have considered different
applications, and in the end energy theft was the most relevant, intriguing and
challenging.Severaldetectionmethodsforenergythefthavebeenanalyzed.Thechosen
methodconsistedintwosteps,thefirstusingenergybalancetodiscoverenergytheft
regionsandthesecondstepusingartificialneuralnetworktopredictthedailyenergy
consumption of individual houses and compare it to the actual consumption. The
designedtoolwaseasily reconfigured toperformaseriesofexperiments to test the
proposedenergytheftdetectionmethod.Theresultsoftheexperimentshowthatthe
first stepof the toolworkedverygood,while the secondstepneededsome further
improvements.
Regardingtherecommendationsandimprovementsthatcouldbedevelopedinthetool.
ItseemsobviousthattheNeuralNetworkdidnotworkasexpected.Itcouldbethatthe
modeloftheNeuralNetworkwaswrong,thatthedatasetsselectedasinputswerenot
corrector itcouldsimplybethattheNeuralNetworkisnotthebestwaytoforecast
energy consumption. It is recommended to further study the datasets that are
introducedintheNeuralNetworkmodelandmaybetrywithotherkindofmodelsof
simulators to forecast energy consumption. However, the co-simulation setup
developedinthisprojectusingMosaikwiththeEuropeanLowVoltageTestFeeder(in
PyPower)wasveryeasytouseandreconfigure.
MasterThesisEzequielFredesSáez
67
Theexperimentresultswereeasilyaccessibleandtheseriesofexperimentswereeasy
to set up and realistic. The tool aided the development of a smart gridmethod for
detectingenergy theft indistributiongrid,butcouldbeeasilyusedwithmanyother
applications.
Thegoalsdescribedintheintroductionare:
§ G1:ModifyMosaikexampledemocodeinordertofulfillrequirements(R1,R2
andR3).[Achievedinchapter3]
o This represents one of the biggest part of the project. It was first
necessarytolearntoprogramwithpythonlanguageandthentodesign
andimplementthesoftwaretofulfilltherequirements.
§ G2: Validate the obtained results against the results provided in the
documentation.[Achievedinchapter4]
o Thishasbeenagreatsuccess.Itcanbeseenthatthedifferencebetween
the given results and the results from the simulation is almost
nonexistent.Thismeansthatthetoolwascorrectlyconfigured.
§ G3: Select a smart grid scenario to be implemented in the tool. [Achieved in
chapter2]
o It was selected the scenario given by the European Low Voltage Test
Feeder.Butasthetoolwasdesigntoautomaticallycreateascenariowith
thefilesinthesameformatasinthefeeder,anyotherscenariocanbe
easilytested.
§ G4:Developanenergytheftdiscoverymethod.[Achievedinchapter5]
o The proposed method to discover energy theft is not complete and
sophisticated,butproventobeusefulfordisseminatingtheoperationof
thedevelopedco-simulationsetup.Theuncommonsmartgridscenario
wasachallengingandinterestingapplication.
§ G5:Preparethetooltoexperimentwithit.[Achievedinchapter6.1]
o Anewscripthadtobecreated.Anexcelfileisusedasthewaytoseeand
interpret the results. Most of the existing scripts had some
reconfiguration.
MasterThesisEzequielFredesSáez
68
§ G6:Performseveraltestcasesontheselectedscenarioandgathertestresults.
[Achievedinchapter6.2]
o Once the tool to experiment has been correctly configured; testing
differentscenariosisjustamatteroftime.
§ G7:Interpretthetestresultsanddrawconclusions.[Achievedinchapter6.3]
o The power balance of the tool worked perfectly, while the Neural
Networkstephadsomedifficultieswiththefalsepositives.
§ G8: Provide recommendations for usewith the developed tool. [Achieved in
chapter7]
o The developed tool is really powerful; however, the Neural Network
modeldidnotworkasgoodasexpected.WithamoreaccurateNeural
Networkthistoolwouldhaveworkedmuchbetter.
§ G9:Publishthedevelopedsoftwareinasharedrepository.[Achievedinchapter
7]
o A Git repository was created to store the software developed in this
project.Git isaversioncontrolsystemusedforsoftwaredevelopment
processinorderto:trackprogress,saveversionsofthesoftware,allow
softwaredevelopedcollaboration,annotateauthorsandchangesmade
inthesoftware.ThefullsoftwarecodecanbedownloadedformaDTU
Elektro sharedgit repositoryhttps://git.elektro.dtu.dk/salvage/mosaik-
ELVTF.gitbyauthorizedusers.
Inconclusion,allthegoalsthatweredefinedintheintroductionoftheprojecthave
beenfulfilled.
MasterThesisEzequielFredesSáez
69
8.-References[1]Horizon2020,Call:H2020-INFRAIA-2014-2015Topic:INFRAIA-1-2014-2015Typeof
action: RIA Proposal number: 654113, Proposal acronym: ERIGrid; Proposal-ERIGrid-
JRA2.pdf
[2] European Low Voltage Test Feeder. Retrieved from :
http://ewh.ieee.org/soc/pes/dsacom/testfeeders/index.html
[3]J.Fitzgeraldetal.(eds.),CollaborativeDesignforEmbeddedSystemsChapter2,Co-
modelling and Co-simulation in Embedded Systems Design,Springer-Verlag Berlin
Heidelberg2014,DOI10.1007/978-3-642-54118-6__2,2014
[4] Basic Concepts on SymPy; Retrieved from:
https://simpy.readthedocs.io/en/latest/simpy_intro/basic_concepts.html
[5] Onisick, Joe, Private Cloud Automation, Orchestration, And Measured Service,
NetworkComputing,June23,2011.
[6]Whatispython?;Retrievedfrom:https://www.python.org/doc/essays/blurb/
[7]Czechowski,R.andKosek,A.M.TheMostFrequentEnergyTheftTechniquesand
HazardsinPresentPowerEnergyConsumption.2016JointWorkshoponCyber-Physical
SecurityandResilienceinSmartGrids(CPSR-SG2016),2016
[8]Salinas,S.,Li,M.,andLi,P..Privacy-PreservingEnergyTheftDetectioninSmartGrids.
20129thAnnualIEEECommunicationsSocietyConferenceonSensor,MeshandAdHoc
Communications and Networks (SECON),DOI 978-1-4673-1905-8/12/$31.00 ©2012
IEEE,2012
MasterThesisEzequielFredesSáez
70
[9]Jiang,R.,Lu,R.,Wang,Y.,Luo,J.,Shen,C.,andShen,X..Energy-TheftDetectionIssues
for Advanced Metering Infrastructure in Smart Grid. TSINGHUA SCIENCE AND
TECHNOLOGYISSNll1007-0214ll01/12llpp105-120Volume19,Number2,April2014
[10]McLaughlin,S.,Holbert,B.,Fawaz,A.,Berthier,R.,andZonouz,S..AMulti-Sensor
Energy Theft Detection Framework for Advanced Metering Infrastructures. IEEE
JOURNALONSELECTEDAREASINCOMMUNICATIONS,VOL.31,NO.7,JULY2013,2013
[11]Kadurek,P.,Studentmember,IEEE,J.Blom,J.F.G.Cobben,W.L.Kling,Member,
IEEE1. Theft detection and smart metering practices and expectations in the
Netherlands
[12] Gurney, K. (1997) An Introduction to Neural Networks London: Routledge. UCL
PressLimited,London.ISBN0-203-45151-1.
[13] Pecanstreet Dataport. Retrieved from:
https://dataport.pecanstreet.org/data/interactive
[14] Neural Network Image. Retrieved from: http://cse22-
iiith.vlabs.ac.in/exp4/images/structure.png
MasterThesisEzequielFredesSáez
71
9.-AppendixIn the appendixes it is shown all the code that has been used for the different
simulations.
9.1.-Var.py# Assuming the information of the feeder csv are always in the same format # Variables to change # Values print('Reading Values') path = 'LVData' nodo = ['Busname'] linea = ['Name', 'Bus1', 'Bus2', 'LineCode', 'Length'] impedancias = ['Name', 'R1', 'X1', 'C1', 'R0', 'X0', 'C0'] transformador = ['Name', 'bus1', 'bus2', 'kV_pri', 'kV_sec', 'MVA', '%XHL', '%resistance'] Tap = 0 # CSV carga = ['Name', 'Bus', 'Yearly'] forma = ['Name', 'File'] # Constants START = '2015-01-01 00:00:00' END = 1 * 24 * 3600 # 1 Day GRID_FILE = 'Grid.xlsx' # Layout houses = 75 thefts = 5 print('Values Read')
9.2.-ExcelValidation.py# Assuming the information of the feeder csv are always in the same format # Initialize import csv import numpy import xlsxwriter from Var import nodo, linea, impedancias, transformador, carga, forma, START, END, GRID_FILE, Tap, path # Constants START = START END = END REF = 'REF' PQ = 'PQ' Online = 1 Line0 = 'Slack Line' taps = '{-1: 0.9, 0: 1.0, 1: 1.1}' #################################################################################################### # Pre-treatment of the data # Assuming the information of the feeder csv are always in the same format # Functions # Reads csv and storage values as asked # csv_name the name of the csv file with path # columns name of the variable that has the name of the headers in the correct order # rep number of rows that are not useful in the csv file def column_extraction(csv_name, columns, rep): a = csv.reader(open(csv_name, "r"), delimiter=',') b = numpy.array(list(a))
MasterThesisEzequielFredesSáez
72
output = numpy.array(b) for r in range(0, rep): b = numpy.delete(b, 0, 0) for j in range(0, len(columns)): for i in range(0, len(b[0])): if b[0, i] == columns[j]: if j == 0: output = numpy.array(b) for x in range(0, i): output = numpy.delete(output, 0, 1) for x in range(i, len(b[0])-1): output = numpy.delete(output, 1, 1) else: temp = numpy.array(b) for x in range(0, i): temp = numpy.delete(temp, 0, 1) for x in range(i, len(b[0])-1): temp = numpy.delete(temp, 1, 1) output = numpy.hstack((output, temp)) output = numpy.delete(output, 0, 0) return output # Inserts values in Excel # var variable to insert column per column # col column to begin inserting # row row to begin inserting # kind matrix or header # numero value to insert many times # flt first column in the excel file that will try to change to float def exc_insert(var, col, row, kind, numero, ftl): if kind == 'One': if col >= ftl: try: worksheet.write(row, col, float(var)) except ValueError: worksheet.write(row, col, var) else: worksheet.write(row, col, var) if kind == 'Matrix': for i in range(0, len(var[0])): for j in range(0, len(var)): if i >= ftl: try: worksheet.write(row+j, col+i, float(var[j, i])) except ValueError: worksheet.write(row+j, col+i, var[j, i]) else: worksheet.write(row+j, col+i, var[j, i]) if kind == 'Row': for i in range(0, len(var)): if i >= ftl: try: worksheet.write(row, i, float(var[i])) except ValueError: worksheet.write(row, i, var[i]) else: worksheet.write(row, i, var[i]) if kind == 'Header': bold = workbook.add_format({'bold': True}) for i in range(0, len(var)): worksheet.write(row, i, var[i], bold) if kind == 'Multiple': for i in range(0, len(var)): if col >= ftl: try: worksheet.write(row+i, col, float(numero)) except ValueError: worksheet.write(row+i, col, numero) else: worksheet.write(row+i, col, numero) return # Substitute if in a column there is the same value as in the same column in the other variable (only2x2) # control Variable that will be controlled # check Variable where it will be checked if control exists
MasterThesisEzequielFredesSáez
73
def checkandcopy(control, check): one = control for i in range(0, len(control)): for j in range(0, len(check)): if control[i, 2] == check[j, 0]: one[i, 2] = check[j, 1] return one # Rearange CSV to the way CSV simulator needs them # name Name of the csv file with path # arch Name of the csv file created with path def rearange_csv(name, arch, kind, start): a = csv.reader(open(name, "r"), delimiter=',') b = numpy.array(list(a)) init = numpy.array([['00000000000000000000000000000000000000000'] * len(b[0]) for _ in range(len(b))]) c = numpy.delete(b, 0, 0) for j in range(0, len(c)): init[j, 0] = start + ' ' + c[j, 0] init[j, 1] = float(c[j, 1])*1000 end = init file = open(arch, "w") w = csv.writer(file, delimiter=',') w.writerow([kind]) w.writerow(["Date", "P # [W]"]) w.writerow(["2016-01-01 00:00:00", "0"]) for i in range(0, len(end)): if i < len(c)-1: w.writerow(end[i]) w.writerow(["2016-01-02 00:00:00", "0"]) file.close() return # CSV print('Starting to Treat the Data') bus = column_extraction("%s/Buscoords.csv" % path, nodo, 1) branch = column_extraction("%s/Lines.csv" % path, linea, 1) linetypes = column_extraction("%s/LineCodes.csv" % path, impedancias, 1) trafotypes = column_extraction("%s/Transformer.csv" % path, transformador, 1) TYPE = trafotypes[0, 0] PRI = trafotypes[0, 1] SEC = trafotypes[0, 2] KV_Prim = trafotypes[0, 3] KV_Sec = trafotypes[0, 4] MVA = trafotypes[0, 5] Imaxp = float(MVA)*1000/1.73205080757/float(KV_Prim) Imaxs = float(MVA)*1000/1.73205080757/float(KV_Sec) IMAX = Imaxs X = float(KV_Sec)*float(KV_Sec)/float(MVA)*float(trafotypes[0, 6]) R = float(KV_Sec)*float(KV_Sec)/float(MVA)*float(trafotypes[0, 7]) Ploss = float(trafotypes[0, 7])*float(KV_Sec)*1000*float(trafotypes[0, 7])*float(KV_Sec)/float(R) slack = [PRI, REF, KV_Prim] slackline = [Line0, PRI, SEC, TYPE, 1, Online, Tap] trafo = [TYPE, MVA, Imaxp, Imaxs, Ploss, R, X, taps] for o in range(0, len(branch)): branch[o, 4] = float(branch[o, 4])/1000 for t in range(0, len(linetypes)): linetypes[t, 1] = (float(linetypes[t, 1])+float(linetypes[t, 4]))/3 linetypes[t, 2] = (float(linetypes[t, 2])+float(linetypes[t, 5]))/3 linetypes[t, 3] = (float(linetypes[t, 3])+float(linetypes[t, 6]))/3 linetypes = numpy.delete(linetypes, 4, 1) linetypes = numpy.delete(linetypes, 4, 1) linetypes = numpy.delete(linetypes, 4, 1) motor = column_extraction("%s/Loads.csv" % path, carga, 2) shapes = column_extraction("%s/LoadShapes.csv" % path, forma, 1) Loads = checkandcopy(motor, shapes) for y in range(1, 101): year = '2016-01-01' category = "Load" original = "%s/Load Profiles/Load_profile_%s.csv" % (path, y) upgraded = "%s/Load Profiles V2/Load_profile_%s.csv" % (path, y) rearange_csv(original, upgraded, category, year) # Excel header_Buses = ['Node Name', 'Node Type', 'Base Voltage [kV]', 'Notes']
MasterThesisEzequielFredesSáez
74
header_Branches = ['Name', 'From', 'To', 'Type', 'Lenght [km]', 'Online', 'Tap'] header_LineTypes = ['Type Name', 'R[Ohm/km]', 'X[Ohm/km]', 'C[nF/km]', 'I_max[A]'] header_TrafoTypes = ['Trafo Type', 'S [MVA]', 'I_max_pri [A]', 'I_max_sec [A]', 'P_loss[kW]', 'R[Ohm]', 'X[Ohm]', 'taps'] workbook = xlsxwriter.Workbook(GRID_FILE) formato = workbook.add_format() formato.set_num_format('0.000') worksheet = workbook.add_worksheet('Nodes') exc_insert(bus, 0, 2, 'Matrix', 0, 2) exc_insert(header_Buses, 0, 0, 'Header', 0, 2) exc_insert(slack, 0, 1, 'Row', 0, 2) exc_insert(bus, 1, 2, 'Multiple', PQ, 2) exc_insert(bus, 2, 2, 'Multiple', KV_Sec, 2) worksheet.set_column('A:Z', 15, formato) worksheet = workbook.add_worksheet('Lines') exc_insert(branch, 0, 2, 'Matrix', 0, 4) exc_insert(header_Branches, 0, 0, 'Header', 0, 4) exc_insert(branch, 5, 2, 'Multiple', Online, 4) exc_insert(slackline, 0, 1, 'Row', 0, 4) worksheet.set_column('A:Z', 15, formato) worksheet = workbook.add_worksheet('Line types') exc_insert(linetypes, 0, 1, 'Matrix', 0, 1) exc_insert(header_LineTypes, 0, 0, 'Header', 0, 1) exc_insert(linetypes, 4, 1, 'Multiple', IMAX, 1) worksheet.set_column('A:Z', 15, formato) worksheet = workbook.add_worksheet('Transformer types') exc_insert(trafo, 0, 1, 'Row', 0, 1) exc_insert(header_TrafoTypes, 0, 0, 'Header', 0, 1) worksheet.set_column('A:Z', 15, formato) workbook.close() print('Data Treated')
9.3.-Excel.py# Assuming the information of the feeder csv are always in the same format # Initialize import csv import numpy import xlsxwriter from Var import nodo, linea, impedancias, transformador, START, END, GRID_FILE, Tap, path, thefts, houses # Constants START = START END = END REF = 'REF' PQ = 'PQ' Online = 1 Line0 = 'Slack Line' taps = '{-1: 0.9, 0: 1.0, 1: 1.1}' thefts = thefts houses = houses #################################################################################################### # Pre-treatment of the data # Assuming the information of the feeder csv are always in the same format # Functions # Reads csv and storage values as asked # csv_name the name of the csv file with path # columns name of the variable that has the name of the headers in the correct order # rep number of rows that are not useful in the csv file def column_extraction(csv_name, columns, rep): a = csv.reader(open(csv_name, "r"), delimiter=',') b = numpy.array(list(a)) output = numpy.array(b) for r in range(0, rep): b = numpy.delete(b, 0, 0) for j in range(0, len(columns)): for i in range(0, len(b[0])): if b[0, i] == columns[j]: if j == 0:
MasterThesisEzequielFredesSáez
75
output = numpy.array(b) for x in range(0, i): output = numpy.delete(output, 0, 1) for x in range(i, len(b[0])-1): output = numpy.delete(output, 1, 1) else: temp = numpy.array(b) for x in range(0, i): temp = numpy.delete(temp, 0, 1) for x in range(i, len(b[0])-1): temp = numpy.delete(temp, 1, 1) output = numpy.hstack((output, temp)) output = numpy.delete(output, 0, 0) return output # Inserts values in Excel # var variable to insert column per column # col column to begin inserting # row row to begin inserting # kind matrix or header # numero value to insert many times # flt first column in the excel file that will try to change to float def exc_insert(var, col, row, kind, numero, ftl): if kind == 'One': if col >= ftl: try: worksheet.write(row, col, float(var)) except ValueError: worksheet.write(row, col, var) else: worksheet.write(row, col, var) if kind == 'Matrix': for i in range(0, len(var[0])): for j in range(0, len(var)): if i >= ftl: try: worksheet.write(row+j, col+i, float(var[j, i])) except ValueError: worksheet.write(row+j, col+i, var[j, i]) else: worksheet.write(row+j, col+i, var[j, i]) if kind == 'Row': for i in range(0, len(var)): if i >= ftl: try: worksheet.write(row, i, float(var[i])) except ValueError: worksheet.write(row, i, var[i]) else: worksheet.write(row, i, var[i]) if kind == 'Header': bold = workbook.add_format({'bold': True}) for i in range(0, len(var)): worksheet.write(row, i, var[i], bold) if kind == 'Multiple': for i in range(0, len(var)): if col >= ftl: try: worksheet.write(row+i, col, float(numero)) except ValueError: worksheet.write(row+i, col, numero) else: worksheet.write(row+i, col, numero) return # Rearange CSV to the way CSV simulator needs them # name Name of the csv file with path # arch Name of the csv file created with path def rearange_csv(tipo, name, arch, casa): a = csv.reader(open(name, "r"), delimiter=',') b = numpy.array(list(a)) init = [0, 0] end = [0, 0] if tipo == "Load": for j in range(0, len(b)): if float(b[j, 1]) == float(casa):
MasterThesisEzequielFredesSáez
76
init[0] = b[j, 0][0:19] init[1] = float(b[j, 2])*1000 end = numpy.vstack((end, init)) elif tipo == "Theft": for j in range(0, len(b)): if float(b[j, 1]) == float(casa): init[0] = b[j, 0][0:19] init[1] = float(b[j, 2])*500*(-1) end = numpy.vstack((end, init)) else: print('Type not defined') end = numpy.delete(end, 0, 0) file = open(arch, "w") w = csv.writer(file, delimiter=',') w.writerow([tipo]) w.writerow(["Date", "P # [W]"]) for i in range(0, len(end)): if i < len(end): w.writerow(end[i]) file.close() return # CSV print('Starting to Treat the Data') bus = column_extraction("%s/Buscoords.csv" % path, nodo, 1) branch = column_extraction("%s/Lines.csv" % path, linea, 1) linetypes = column_extraction("%s/LineCodes.csv" % path, impedancias, 1) trafotypes = column_extraction("%s/Transformer.csv" % path, transformador, 1) TYPE = trafotypes[0, 0] PRI = trafotypes[0, 1] SEC = trafotypes[0, 2] KV_Prim = trafotypes[0, 3] KV_Sec = trafotypes[0, 4] MVA = trafotypes[0, 5] Imaxp = float(MVA)*1000/1.73205080757/float(KV_Prim) Imaxs = float(MVA)*1000/1.73205080757/float(KV_Sec) IMAX = Imaxs X = float(KV_Sec)*float(KV_Sec)/float(MVA)*float(trafotypes[0, 6]) R = float(KV_Sec)*float(KV_Sec)/float(MVA)*float(trafotypes[0, 7]) Ploss = float(trafotypes[0, 7])*float(KV_Sec)*1000*float(trafotypes[0, 7])*float(KV_Sec)/float(R) slack = [PRI, REF, KV_Prim] slackline = [Line0, PRI, SEC, TYPE, 1, Online, Tap] trafo = [TYPE, MVA, Imaxp, Imaxs, Ploss, R, X, taps] for o in range(0, len(branch)): branch[o, 4] = float(branch[o, 4])/1000 for t in range(0, len(linetypes)): linetypes[t, 1] = (float(linetypes[t, 1])+float(linetypes[t, 4]))/3 linetypes[t, 2] = (float(linetypes[t, 2])+float(linetypes[t, 5]))/3 linetypes[t, 3] = (float(linetypes[t, 3])+float(linetypes[t, 6]))/3 linetypes = numpy.delete(linetypes, 4, 1) linetypes = numpy.delete(linetypes, 4, 1) linetypes = numpy.delete(linetypes, 4, 1) names = ['26', '93', '94', '171', '187', '252', '370', '410', '545', '585', '624', '739', '744', '861', '871', '890', '898', '1103'] for y in range(len(names)): original = "%s/Profiles.csv" % path upgraded = "%s/Profiles/Load_profile_%s.csv" % (path, names[y]) theft = "%s/Profiles/Theft_profile_%s.csv" % (path, names[y]) rearange_csv("Load", original, upgraded, names[y]) rearange_csv("Theft", original, theft, names[y]) # Excel header_Buses = ['Node Name', 'Node Type', 'Base Voltage [kV]', 'Notes'] header_Branches = ['Name', 'From', 'To', 'Type', 'Lenght [km]', 'Online', 'Tap'] header_LineTypes = ['Type Name', 'R[Ohm/km]', 'X[Ohm/km]', 'C[nF/km]', 'I_max[A]'] header_TrafoTypes = ['Trafo Type', 'S [MVA]', 'I_max_pri [A]', 'I_max_sec [A]', 'P_loss[kW]', 'R[Ohm]', 'X[Ohm]', 'taps'] workbook = xlsxwriter.Workbook(GRID_FILE) formato = workbook.add_format() formato.set_num_format('0.000') worksheet = workbook.add_worksheet('Nodes') exc_insert(bus, 0, 2, 'Matrix', 0, 2) exc_insert(header_Buses, 0, 0, 'Header', 0, 2) exc_insert(slack, 0, 1, 'Row', 0, 2) exc_insert(bus, 1, 2, 'Multiple', PQ, 2) exc_insert(bus, 2, 2, 'Multiple', KV_Sec, 2) worksheet.set_column('A:Z', 15, formato)
MasterThesisEzequielFredesSáez
77
worksheet = workbook.add_worksheet('Lines') exc_insert(branch, 0, 2, 'Matrix', 0, 4) exc_insert(header_Branches, 0, 0, 'Header', 0, 4) exc_insert(branch, 5, 2, 'Multiple', Online, 4) exc_insert(slackline, 0, 1, 'Row', 0, 4) worksheet.set_column('A:Z', 15, formato) worksheet = workbook.add_worksheet('Line types') exc_insert(linetypes, 0, 1, 'Matrix', 0, 1) exc_insert(header_LineTypes, 0, 0, 'Header', 0, 1) exc_insert(linetypes, 4, 1, 'Multiple', IMAX, 1) worksheet.set_column('A:Z', 15, formato) worksheet = workbook.add_worksheet('Transformer types') exc_insert(trafo, 0, 1, 'Row', 0, 1) exc_insert(header_TrafoTypes, 0, 0, 'Header', 0, 1) worksheet.set_column('A:Z', 15, formato) workbook.close() print('Data Treated')
9.4.-LVTest.py# Initialize import random from mosaik.util import connect_many_to_one import mosaik from Excel import names, thefts, houses, START, END, GRID_FILE import warnings warnings.filterwarnings("ignore") # Simulators sim_config = { 'CSV': { 'python': 'mosaik_csv:CSV', }, 'DB': { 'cmd': 'mosaik-hdf5 %(addr)s', }, 'PyPower': { 'python': 'mosaik_pypower.mosaik:PyPower', # 'cmd': 'mosaik-pypower %(addr)s', }, 'WebVis': { 'cmd': 'mosaik-web -s 0.0.0.0:8000 %(addr)s', }, } # Main def main(): random.seed() world = mosaik.World(sim_config) create_scenario(world) world.run(until=END) # Scenario def create_scenario(world): # Start simulators pypower = world.start('PyPower', 'PyP', step_size=60) # Instantiate models grid = pypower.Grid(gridfile=GRID_FILE).children # Create nodes load = [] theft = [] ladron = 0 casa = 0 for q in range(len(names)): prof = names[q] if casa < houses: hfile = 'LVdata/Profiles/Load_profile_%s.csv' % prof loadssim = world.start('CSV', '%s' % prof, sim_start=START, datafile=hfile) if houses > len(names) and casa+(houses//len(names)+1) < houses: hsimulator = loadssim.Load.create((houses//len(names))+1)
MasterThesisEzequielFredesSáez
78
casa += (houses//len(names)+1) elif houses > len(names) and casa + len(names) - q - 1 < houses-1: hsimulator = loadssim.Load.create((houses-casa)//(len(names)-q)) casa += ((houses-casa)//(len(names)-q)) else: hsimulator = loadssim.Load.create(1) casa += 1 try: load = load + hsimulator except ValueError: load = hsimulator except UnboundLocalError: pass else: pass if ladron < thefts: tfile = 'LVdata/Profiles/Theft_profile_%s.csv' % prof theftsim = world.start('CSV', '%s' % prof, sim_start=START, datafile=tfile) if thefts > len(names) and ladron+(thefts//len(names)+1) < thefts: tsimulator = theftsim.Theft.create((thefts//len(names))+1) ladron += (thefts//len(names)+1) elif houses > len(names) and ladron + len(names) - q - 1 < thefts-1: tsimulator = theftsim.Theft.create((houses-ladron)//(len(names)-q)) ladron += ((houses-ladron)//(len(names)-q)) else: tsimulator = theftsim.Theft.create(1) ladron += 1 try: theft = theft + tsimulator except ValueError: theft = tsimulator except UnboundLocalError: pass else: pass # Connect entities buses = filter(lambda e: e.type == 'PQBus', grid) buses = {b.eid.split('-')[1]: b for b in buses} t = 0 for w in range(len(load)): number = int(len(buses)*w/len(load))+1 text = '%s' % number world.connect(load[w], buses[text], 'P') if thefts != 0: if w % (houses//thefts) == 0: try: world.connect(theft[t], buses[text], 'P') t += 1 except IndexError: pass else: pass else: pass # Database db = world.start('DB', 'DB', step_size=60, duration=END) hdf5 = db.Database(filename='LVTest.hdf5') connect_many_to_one(world, load, hdf5, 'P') nodes = [e for e in grid if e.type in 'RefBus, PQBus'] connect_many_to_one(world, nodes, hdf5, 'P', 'Q', 'Vl', 'Vm', 'Va') branches = [e for e in grid if e.type in ('Transformer', 'Branch')] connect_many_to_one(world, branches, hdf5, 'P_from', 'Q_from', 'P_to', 'P_from') # Web visualization webvis = world.start('WebVis', 'WebVis', start_date=START, step_size=60) webvis.set_config(ignore_types=['Topology', 'ResidentialLoads', 'Grid', 'Database']) vis_topo = webvis.Topology() connect_many_to_one(world, nodes, vis_topo, 'P', 'Vm') webvis.set_etypes({ 'RefBus': { 'cls': 'refbus', 'attr': 'P', 'unit': 'P [W]', 'default': 0, 'min': 0, 'max': 100000, }, 'PQBus': {
MasterThesisEzequielFredesSáez
79
'cls': 'pqbus', 'attr': 'Vm', 'unit': 'U [V]', 'default': 400, 'min': 0.8 * 400, 'max': 1.2 * 400, }, }) connect_many_to_one(world, load, vis_topo, 'P') webvis.set_etypes({ 'Load': { 'cls': 'load', 'attr': 'P', 'unit': 'P [W]', 'default': 0, 'min': 0, 'max': 10000, }, }) connect_many_to_one(world, theft, vis_topo, 'P') webvis.set_etypes({ 'Theft': { 'cls': 'theft', 'attr': 'P', 'unit': 'P [W]', 'default': 0, 'min': 0, 'max': 10000, }, }) # Running if __name__ == '__main__': main()
9.5.-Output.pyfrom __future__ import print_function import numpy as np import h5py import matplotlib.pyplot as plt import datetime import os import xlsxwriter # Inserts values in Excel # var variable to insert column per column # col column to begin inserting # row row to begin inserting # formato format for the number introduced def exc_insert(var, col, row, formato): for i in range(0, len(var[0])): for j in range(0, len(var)): if i == 1: try: worksheet.write(row+j, col+i, float(var[j, i])) except ValueError: worksheet.write(row+j, col+i, var[j, i]) else: worksheet.write(row+j, col+i, var[j, i], formato) return # Open hdf5 and print graph # hdf5 name of database # path location database # title figure # fig_name name of the figure # variable kind of variable def print_output(hdf5, path, title, fig_name, variable): hora = workbook.add_format({'num_format': 'dd/mm/yyyy hh:mm'}) with h5py.File(hdf5, 'r') as hf:
MasterThesisEzequielFredesSáez
80
datap = hf.get(path) power = np.array(datap) plt.plot(tiempo, power) plt.autoscale() plt.gcf().autofmt_xdate() plt.ylabel(variable) plt.xlabel('Time') plt.title(title) plt.savefig(fig_name) plt.close() tpower = tiempo.reshape(len(tiempo), 1) tpower = np.hstack((tpower, power.reshape(len(power), 1))) exc_insert(tpower, 0, 1, hora) return # Export energy through lines # sheet name # path location def lineas(sheet, path): work = workbook.add_worksheet(sheet) work.write(0, 0, 'Line') work.write(0, 1, 'Energy') with h5py.File(path, 'r') as hf: for n in range(1, 906): name = 'LINE%s' % n datap = hf.get('/Series/PyP-0.0-%s/P_from' % name) power = np.array(datap) energy = 0 for i in range(len(power)): energy += float(power[i])/1000/60 work.write(n, 0, name) work.write(n, 1, energy) return # Export loads location # sheet name # path location def loads(sheet, path): work = workbook.add_worksheet(sheet) work.write(0, 0, 'Node') work.write(0, 1, 'Profile') work.write(0, 2, 'Energy') with h5py.File(path, 'r') as hf: l = 1 for n in range(1, 906): node = '%s' % n relation = hf.get('/Relations/PyP-0.0-%s' % node) for i in range(len(relation)): profile = (np.array(relation)[i])[0].decode("utf-8") if 'Load' in profile: datap = hf.get('/Series/PyP-0.0-%s/P' % node) power = np.array(datap) energy = 0 for z in range(len(power)): energy += float(power[z])/1000/60 work.write(l, 0, node) work.write(l, 1, profile[11:50]) work.write(l, 2, energy) l += 1 return # Export thefts location # sheet name # path location def thefts(sheet, path): work = workbook.add_worksheet(sheet) work.write(0, 0, 'Node') with h5py.File(path, 'r') as hf: l = 1 for n in range(1, 906): node = '%s' % n relation = hf.get('/Relations/PyP-0.0-%s' % node) for i in range(len(relation)):
MasterThesisEzequielFredesSáez
81
profile = (np.array(relation)[i])[0].decode("utf-8") if 'Theft' in profile: work.write(l, 0, node) l += 1 return run = os.system('source /Users/EzequielFredes/.virtualenvs/mosaik/bin/activate\n python LVTest.py\n') # run = 0 end = 0 tiempo = np.array([datetime.datetime(2016, 1, 1, i, j) for i in range(24) for j in range(60)]) print("Begin plotting output") # Theory workbook = xlsxwriter.Workbook('Comparator/Theory.xlsx') worksheet = workbook.add_worksheet('Theory') worksheet.write(0, 0, 'TimeStamp') worksheet.write(0, 1, 'Power') print_output('LVTheory.hdf5', '/Series/PyP-0.0-LINE1/P_from', 'Power Transformer Secondary', 'Print/PowerTransformerSecondaryTheory.png', 'Power [W]') lineas('Lines Theory', 'LVTheory.hdf5') workbook.close() # Reality workbook = xlsxwriter.Workbook('Comparator/Test.xlsx') worksheet = workbook.add_worksheet('Test') worksheet.write(0, 0, 'TimeStamp') worksheet.write(0, 1, 'Power') while end == 0: if run == 0: print_output('LVTest.hdf5', '/Series/PyP-0.0-LINE1/P_from', 'Power Transformer Secondary', 'Print/PowerTransformerSecondary.png', 'Power [W]') lineas('Lines Test', 'LVTest.hdf5') loads('Nodes Test', 'LVTest.hdf5') thefts('Thefts Test', 'LVTest.hdf5') end = 1 workbook.close() print("Output plotted")
9.6.-NN.py# Assuming the information of the feeder csv are always in the same format # Initialize import csv import numpy import copy import neurolab import xlsxwriter #################################################################################################### # Pre-treatment of the data # Assuming the information of the feeder csv are always in the same format # Functions # Reads csv and storage values # csv_name the name of the csv file with path def weather_reader(csv_name): a = csv.reader(open(csv_name, "r"), delimiter=',') b = numpy.array(list(a)) c = numpy.delete(b, 0, 0) d = numpy.delete(b, 0, 0) e = numpy.delete(b, 0, 0) f = numpy.delete(b, 0, 0) g = numpy.delete(b, 0, 0) h = numpy.delete(b, 0, 0) wyear = e[:, 0] wmonth = f[:, 0] wday = g[:, 0] whour = h[:, 0]
MasterThesisEzequielFredesSáez
82
for i in range(len(d)): d[i, 1] = ((float(c[i, 1])-32)/1.8) for i in range(len(d)): wyear[i] = float(d[i, 0][0:4]) wmonth[i] = float(d[i, 0][5:7]) wday[i] = float(d[i, 0][8:10]) whour[i] = float(d[i, 0][11:13]) output = numpy.column_stack((wyear, wmonth, wday, whour, d[:, 1])) return output # Reads csv and storage values # csv_name the name of the csv file with path def house_reader(csv_name): a = csv.reader(open(csv_name, "r"), delimiter=',') b = numpy.array(list(a)) c = numpy.delete(b, 0, 0) d = numpy.delete(b, 0, 0) e = numpy.delete(b, 0, 0) f = numpy.delete(b, 0, 0) g = numpy.delete(b, 0, 0) h = numpy.delete(b, 0, 0) hyear = e[:, 0] hmonth = f[:, 0] hday = g[:, 0] hhour = h[:, 0] for i in range(len(d)): d[i, 2] = float(c[i, 2])*1000 for i in range(len(d)): hyear[i] = float(d[i, 0][0:4]) hmonth[i] = float(d[i, 0][5:7]) hday[i] = float(d[i, 0][8:10]) hhour[i] = float(d[i, 0][11:13]) output = numpy.column_stack((hyear, hmonth, hday, hhour, d[:, 1], d[:, 2])) return output # Neural data preparation def neuraldata(casa, clima, urba, dest): file = open(dest, "w") a = csv.writer(file, delimiter=',') a.writerow([casa]) a.writerow(["Year", "Month", "Day", "Temperature", "Energy"]) for i in range(2013, 2016): for j in range(1, 13): for k in range(1, 32): average = 0 energy = 0 for x in range(len(clima)): if float(clima[x, 0]) == i and float(clima[x, 1]) == j and float(clima[x, 2]) == k: average += float(clima[x, 4]) average /= 24 for z in range(len(urba)): if float(urba[z, 0]) == i and float(urba[z, 1]) == j and float(urba[z, 2]) == k and\ urba[z, 4] == casa: energy += float(urba[z, 5])/1000 vector = [i, j, k, average, energy] if energy == 0: pass else: a.writerow(vector) file.close() return # Creation Datasets def dataset(name, year): place = "LVData/NN/House%s.csv" % name a = csv.reader(open(place, "r"), delimiter=',') b = numpy.array(list(a)) c = numpy.delete(b, 0, 0) d = numpy.delete(c, 0, 0) e = numpy.array(list(d)) data = [0, 0, 0, 0, 0] for i in range(len(e)):
MasterThesisEzequielFredesSáez
83
if float(e[i, 0]) == year: data = numpy.row_stack((data, [float(e[i, 0]), float(e[i, 1]), float(e[i, 2]), float(e[i, 3]), float(e[i, 4])])) data = numpy.delete(data, 0, 0) return data # Normalize data 1 dim vector # var Variable # do to normalize (0) or unnormalize (1) def normalize(var, do, come): normal = copy.deepcopy(var) original = copy.deepcopy(come) if do == 0: for i in range(len(var)): normal[i] = (var[i]-min(original))/(max(original)-min(original)) elif do == 1: for i in range(len(var)): normal[i] = var[i]*(max(original)-min(original))+min(original) else: print("Wrong Normalization") output = normal return output # CSV print('Neural Network Start') names = ['26', '93', '94', '171', '187', '252', '370', '410', '545', '585', '624', '739', '744', '861', '871', '890', '898', '1103'] # To be run only once estimated time for running is 4 hours weather = weather_reader("LVData/NNWeather.csv")[0::3] houses = house_reader("LVData/NNHouse.csv") for n in range(len(names)): path = "LVData/NN/House%s.csv" % names[n] neuraldata(names[n], weather, houses, path) # NN workbook = xlsxwriter.Workbook('Comparator/NN.xlsx') worksheet = workbook.add_worksheet('NN') worksheet.write(0, 0, 'House') worksheet.write(0, 1, 'Error in NN') worksheet.write(0, 2, 'Expected Consumption in day') for n in range(len(names)): # NN # Create train samples training = dataset(names[n], 2013) t_month = normalize(training[:, 1], 0, training[:, 1]).reshape(len(training), 1) t_day = normalize(training[:, 2], 0, training[:, 2]).reshape(len(training), 1) t_temperature = normalize(training[:, 3], 0, training[:, 3]).reshape(len(training), 1) t_inp = numpy.column_stack((t_month, t_day, t_temperature)) t_energy = normalize(training[:, 4], 0, training[:, 4]).reshape(len(training), 1) # Create validation samples validation = dataset(names[n], 2014) v_month = normalize(validation[:, 1], 0, training[:, 1]).reshape(len(validation), 1) v_day = normalize(validation[:, 2], 0, training[:, 2]).reshape(len(validation), 1) v_temperature = normalize(validation[:, 3], 0, training[:, 3]).reshape(len(validation), 1) v_inp = numpy.column_stack((v_month, v_day, v_temperature)) v_energy = validation[:, 4].reshape(len(validation), 1) # Create output samples ahora = dataset(names[n], 2015) a_month = normalize(ahora[:, 1], 0, training[:, 1]).reshape(len(ahora), 1) a_day = normalize(ahora[:, 2], 0, training[:, 2]).reshape(len(ahora), 1) a_temperature = normalize(ahora[:, 3], 0, training[:, 3]).reshape(len(ahora), 1) a_inp = numpy.column_stack((a_month, a_day, a_temperature)) # Create network with 3 inputs # 3 nurons for input layer, 6 neurons for hidden layer, 1 neuron for output # 3 layers including hidden layer and output layer
MasterThesisEzequielFredesSáez
84
net = neurolab.net.newff([[0, 1], [0, 1], [0, 1]], [3, 6, 1]) # Train network error = net.train(t_inp, t_energy, epochs=2000, show=200, goal=0.01) # Simulate network out = normalize(net.sim(v_inp), 1, training[:, 4]) # Calculate error mistake = 0 percentage = 0 for p in range(len(out)): mistake += abs(out[p]-v_energy[p])/v_energy[p] # Output 01-01-2015 nn = normalize(net.sim(a_inp), 1, training[:, 4]) worksheet.write(n+1, 0, names[n]) worksheet.write(n+1, 1, mistake/len(v_energy)*100) worksheet.write(n+1, 2, nn[0]) workbook.close() print('Neural Network Done')