Investigation of Smart Grids scenarios by the use of co ... · The objective of the project is to understand the European low voltage test feeder in order to implement it into the

ESCUELA TÉCNICA SUPERIOR DE INGENIERÍA (ICAI)

Investigation of Smart Grids scenarios by the use of co-simulation tools with a

European low voltage test feeder and an optimal power flow solver

Autor: Ezequiel Fredes Sáez Director: Anna Magdalena Kosek

Madrid Octubre 2016

AUTORIZACIÓN PARA LA DIGITALIZACIÓN, DEPÓSITO Y DIVULGACIÓN EN RED DE PROYECTOS FIN DE GRADO, FIN DE MÁSTER, TESINAS O MEMORIAS DE BACHILLERATO

1º. Declaración de la autoría y acreditación de la misma. El autor D. Ezequiel Fredes Sáez DECLARA ser el titular de los derechos de propiedad intelectual de la obra: Investigation of Smart Grids scenarios by the use of co-simulation tools with a European low voltage test feeder and an optimal power flow solver, que ésta es una obra original, y que ostenta la condición de autor en el sentido que otorga la Ley de Propiedad Intelectual. 2º. Objeto y fines de la cesión. Con el fin de dar la máxima difusión a la obra citada a través del Repositorio institucional de la Universidad, el autor CEDE a la Universidad Pontificia Comillas, de forma gratuita y no exclusiva, por el máximo plazo legal y con ámbito universal, los derechos de digitalización, de archivo, de reproducción, de distribución y de comunicación pública, incluido el derecho de puesta a disposición electrónica, tal y como se describen en la Ley de Propiedad Intelectual. El derecho de transformación se cede a los únicos efectos de lo dispuesto en la letra a) del apartado siguiente. 3º. Condiciones de la cesión y acceso Sin perjuicio de la titularidad de la obra, que sigue correspondiendo a su autor, la cesión de derechos contemplada en esta licencia habilita para:

a) Transformarla con el fin de adaptarla a cualquier tecnología que permita incorporarla a internet y hacerla accesible; incorporar metadatos para realizar el registro de la obra e incorporar “marcas de agua” o cualquier otro sistema de seguridad o de protección.

b) Reproducirla en un soporte digital para su incorporación a una base de datos electrónica, incluyendo el derecho de reproducir y almacenar la obra en servidores, a los efectos de garantizar su seguridad, conservación y preservar el formato.

c) Comunicarla, por defecto, a través de un archivo institucional abierto, accesible de modo libre y gratuito a través de internet.

d) Cualquier otra forma de acceso (restringido, embargado, cerrado) deberá solicitarse expresamente y obedecer a causas justificadas.

e) Asignar por defecto a estos trabajos una licencia Creative Commons. f) Asignar por defecto a estos trabajos un HANDLE (URL persistente).

4º. Derechos del autor. El autor, en tanto que titular de una obra tiene derecho a:

a) Que la Universidad identifique claramente su nombre como autor de la misma b) Comunicar y dar publicidad a la obra en la versión que ceda y en otras posteriores a través

de cualquier medio. c) Solicitar la retirada de la obra del repositorio por causa justificada. d) Recibir notificación fehaciente de cualquier reclamación que puedan formular terceras

personas en relación con la obra y, en particular, de reclamaciones relativas a los derechos de propiedad intelectual sobre ella.

5º. Deberes del autor. El autor se compromete a:

a) Garantizar que el compromiso que adquiere mediante el presente escrito no infringe ningún derecho de terceros, ya sean de propiedad industrial, intelectual o cualquier otro.

b) Garantizar que el contenido de las obras no atenta contra los derechos al honor, a la intimidad y a la imagen de terceros.

c) Asumir toda reclamación o responsabilidad, incluyendo las indemnizaciones por daños, que pudieran ejercitarse contra la Universidad por terceros que vieran infringidos sus derechos e

MasterThesisEzequielFredesSáez

2

TableofContentsAbstract....................................................................................................................6

English..............................................................................................................................6

Español.............................................................................................................................7

1.-Introduction..........................................................................................................8

2.-LVNetworkDescription.......................................................................................11

2.1.-Nodes.......................................................................................................................12

2.2.-Lines.........................................................................................................................12

2.3.-Linetypes.................................................................................................................13

2.4.-Transformertypes....................................................................................................14

3.-SystemDescription..............................................................................................15

3.1.-IntroductiontoCo-simulationwithMosaik..............................................................15

3.2.-ConfigurationofMosaikandPyPower......................................................................19

3.2.1.-Co-SimulationSetupImplementedinMosaikOverview.........................................19

3.2.2.-Var.py......................................................................................................................21

3.2.3.-Excel.py....................................................................................................................21

3.2.4.-LVTest.py.................................................................................................................27

3.2.5.-Output.py................................................................................................................31

4.-EuropeanLowVoltageTestFeederValidation.....................................................33

5.-Application..........................................................................................................36

5.1.-EnergyTheft.............................................................................................................37

5.2.-EnergyTheftTool.....................................................................................................38

5.2.1.-Powerbalance.........................................................................................................41

5.2.2.-NeuralNetwork.......................................................................................................44

6.-Experiments........................................................................................................46

6.1.-ChangesintheTool..................................................................................................46

6.1.1.-Var.py......................................................................................................................49

6.1.2.-Excel.py....................................................................................................................49

6.1.3.-LVTest.py.................................................................................................................50

6.1.4.-Output.py................................................................................................................54

6.1.5.-NN.py.......................................................................................................................56


3

6.1.6.-Comparator.xlsx......................................................................................................59

6.2.-ExperimentsScenarios.............................................................................................60

6.2.1.-Experiment0Thefts................................................................................................60

6.2.2.-Experiment5Thefts................................................................................................60

6.2.3.-Experiment10Thefts..............................................................................................61

6.2.4.-Experiment15Thefts..............................................................................................62

6.3.-ExperimentsResults.................................................................................................62

6.3.1.-NetworkLevel.........................................................................................................63

6.3.2.-SubsystemLevel......................................................................................................63

6.3.3.-NeuralNetworkLevel..............................................................................................64

6.4.-ExperimentConclusions...........................................................................................65

7.-Conclusion...........................................................................................................66

8.-References..........................................................................................................69

9.-Appendix.............................................................................................................71

9.1.-Var.py......................................................................................................................71

9.2.-ExcelValidation.py...................................................................................................71

9.3.-Excel.py....................................................................................................................74

9.4.-LVTest.py.................................................................................................................77

9.5.-Output.py.................................................................................................................79

9.6.-NN.py.......................................................................................................................81


4

TableofFiguresFigure1:NetworkDescription.[2].................................................................................11

Figure2:ScreenshotofNodes.[Grid.xlsx].....................................................................12

Figure3:ScreenshotofLines.[Grid.xlsx].......................................................................13

Figure4:ScreenshotofLineTypes.[Grid.xlsx]...............................................................13

Figure5:ScreenshotofTransformerTypes.[Grid.xlsx].................................................14

Figure6:Co-simulationStructure..................................................................................15

Figure7:DemoOutputofMosaik..................................................................................18

Figure8:OverviewoftheProgram................................................................................20

Figure9:Variables.[Var.py]...........................................................................................21

Figure10:ColumnExtractionFunction.[Excel.py].........................................................22

Figure11:ExcelInsertFunction.[Excel.py]....................................................................24

Figure12:CheckandCopyFunction.[Excel.py].............................................................26

Figure13:RearrangeCSVFunction.[Excel.py]...............................................................26

Figure14:LoadCreationFunction.[LVTest.py].............................................................27

Figure15:ConnectEntitiesFunction.[LVTest.py]..........................................................27

Figure16:MosaikVisualization......................................................................................28

Figure17:WebVisualizationJavaScriptChanges.[mosaik.js].......................................29

Figure18:WebVisualizationCSSChanges.[main.css]...................................................29

Figure19:SettingVisualizationConfiguration.[LVTest.py]...........................................30

Figure20:ImprovedWebVisualization.........................................................................30

Figure21:OutputPrintingFunction.[Output.py]..........................................................31

Figure22:OutputPlotExample.....................................................................................32

Figure23:RealOutputFeeder.......................................................................................33

Figure24:PercentageDifferencebetweenFeederandCo-simulation.........................34

Figure25:AccumulatedErrorinPercentage.................................................................34

Figure26:BoxChart.......................................................................................................35

Figure27:PaperMethodSummary.[11].......................................................................39

Figure28:FlowDiagramtoevaluateEnergyTheft........................................................40

Figure29:Q-GIS.............................................................................................................43

Figure30:SystemsDesign..............................................................................................43


5

Figure31:ArchitectureofNeuralNetwork.[14]...........................................................44

Figure32:OverviewoftheProgramwithChanges........................................................47

Figure33:LayoutDefinition.[Var.py]............................................................................49

Figure34:ChangesinRearange_csvFunction.[Excel.py]..............................................50

Figure35:HousesandTheftsLoadProfile.[Excel.py]....................................................50

Figure36:CreatingVariableNumberofLoadsandThefts.[LVTest.py].........................51

Figure37:ConnectingLoadsandThefts.[LVTest.py]....................................................52

Figure38:WebVisualizationChanges...........................................................................53

Figure39:ZoomedWebVisualization............................................................................53

Figure40:Hdf5ReaderandWriter.[Output.py]...........................................................54

Figure41:LinesReaderandWriter.[Output.py]...........................................................54

Figure42:NodesReaderandWriter.[Output.py].........................................................55

Figure43:TheftsReaderandWriter.[Output.py].........................................................55

Figure44:DataReadingFunction.[NN.py]....................................................................56

Figure45:NeuralNetworkDataPreparation.[NN.py]..................................................57

Figure46:NormalizeFunction.[NN.py].........................................................................57

Figure47:NeuralNetworkPrototype.[NN.py]..............................................................58

Figure48:NeuralNetworkDeveloped.[NN.py]............................................................58

Figure49:SubstationComparator.[Comparator.xlsx]..................................................59

Figure50:SubsystemsComparator.[Comparator.xlsx].................................................59

Figure51:NNComparator.[Comparator.xlsx]...............................................................60

Figure52:Experiment5Thefts.[Comparator.xlsx]........................................................61

Figure53:Experiment10Thefts.[Comparator.xlsx]......................................................61

Figure54:Experiment15Thefts.[Comparator.xlsx]......................................................62


6

Abstract

English

According to the Danish energy strategy, Denmark's power consumption has to be

independentofsourcesofenergyreliantonfossilfuelsby2050.Inordertoachievethis

goal, renewable energy is the key component to provide energy and reduce the

greenhouseemissions.Forthesakeofmaintainingthequalityandreliabilitythatthe

powersystemhastoday,newwaysofoperatingpowersystemswillbenecessary.Future

SmartGridsscenariosincludingnewcomponentsandcommunicationtechnologycannot

be investigated in the current power system. New tools need to be developed to

representthecomplexityofSmartGridandallowenergysystemsstakeholderstodefine

andexecuterealisticscenarios.

Mosaik allows the co-simulation of different simulation models and simulators and

opensthepossibilitytohavelargescaleSmartGridscenarios.

Theobjectiveof theproject is tounderstand theEuropean lowvoltage test feeder in

order to implement it into the program PYPOWER, that is the optimal power flow

simulator.ThisimplementationwillbeusedtogetherwithMosaiktoperformsimulations

oflargeSmartGridscenarios.

ThefeederimplementationwillbetestedinseveralSmartGridsscenariostocheckthe

robustness of the tool. Finally, the prepared co-simulation set-up will be used to

investigatedifferentSmartGridsapplication,suchasevaluationoftheenergytheft.

Thepossibilitytohaveadvancedandpreciseco-simulationtoolswillhelpenergysystem

stakeholderstodevelopsolutionsintheSmartGridsmuchfasterandaccurate


7

Español

Deacuerdoa laestrategiaenergéticadanesa,elconsumodepotenciaenDinamarca

tienequeserindependientedefuentesdeenergíaprovenientesdecombustiblesfósiles

paraelaño2050.Conobjetodeconseguirlo,lasfuentesdeenergíarenovablessonel

componenteclaveparaproporcionarenergíayreducirlosnivelesdeemisióndegases

deefectoinvernadero.Conlaideademantenerlacalidadyfiabilidadquelossistemas

depotenciahoyendía,nuevosmétodosdeoperacióndelossistemasdepotenciason

necesarios. Escenarios de Smart Grids en el futuro, incluyendo nuevos componentes

tecnológicosysistemasdecomunicación,nopuedenserinvestigadosconlossistemas

depotenciaactuales.Nuevasherramientasnecesitanserinvestigadaspararepresentar

lacomplexidaddelasSmartGridsypermitiralosinteresadosensistemasdeenergía

definiryejecutarescenariosrealistas.

Mosaik es una herramienta que permite la co-simulación de diferentes modelos y

simuladoresya suvezabre laposibilidadde tenerescenariosdeSmartGridsagran

escala.ElobjetivodelproyectoesentenderelEuropeanLowVoltageTestFeederpara

implementarloenelprogramaPyPower;queesunsimuladordeflujosdecarga.Esta

implementaciónserealizarájuntoconMosaikpararealizarsimulacionesenescenarios

deSmartGridsagranescala.

La implementacióndelFeeder será testadaenvariosescenariosdeSmartGridspara

comprobarlafiabilidaddelaherramienta.Finalmente,laherramientadeco-simulación

desarrolladaseemplearáendistintasaplicacionesdeSmartGrids,comoporejemplola

evaluacióndelrobodeenergía.Laposibilidaddetenerherramientasdeco-simulación

avanzadasyprecisasconseguiráque los interesadosensistemasdeenergíaeléctrica

puedandesarrollarsolucionesenlasSmartGridsdemaneramásrápidayprecisa.


8

1.-IntroductionThetermSmartGridsisdaybydaybecomingmoreandmorepopular.Theinstallation

ofmoresophisticatedmeasurementdevicesinpowersystem,bringsmoreinformation,

especiallyinrealtime,fromthegrid;informationthatwasnotquicklyavailableinthe

past.Asnewdataandresourcesareavailable,newmethodsforevaluatingthegridcan

beusedtoanswerdemandinggridchallengesmuchfasterandmoreefficient.

This project is developed as part of the European project called European Research

InfrastructuresupportingSmartGridSystemsTechnologyDevelopment,Validationand

RollOut,orERIGridproject.ThetargetoftheERIGridproject,asstatedintheproposal

oftheprojectistosupportthetechnologydevelopmentaswellastherolloutofSmart

Grid approaches, solutions and concepts in Europe taking a holistic, cyber-physical

systemsapproachintoaccount.ItintegratesthemajorEuropeanresearchcenterswith

a considerable, outstanding Smart Grid research infrastructure to jointly develop

commonmethods,concepts,andprocedures.[1]

Theobjectiveofthismasterthesisistoprovideatoolthatisabletoautomaticallyload

andsimulatelargesmartgridsscenarios,withtheuseofco-simulationandapplythe

results of the simulation to an energy theft scenario. The reason for choosing co-

simulation tool is the ability to join together different simulators, that have been

developed and tested in their domains, and very easily join them together without

needingtocreateasinglesimulatorthatrepresentsmanydomains.

Thisprojecthasseveralgoalsthatareneededtobedonetofulfilltheobjective.Butin

order to achieve the goals, some requirements have to be meet first. These

requirementsare:

• R1:LoadtheEuropeanLowVoltageTestFeederfromtheoriginalfiles.

• R2: Exchange original loads and/or production units from the European Low

VoltageTestFeederwithotherloadsand/orproductionunitsatanybusinthe

EuropeanLowVoltageTestFeeder.

• R3:ReconfigureVisualizationtodisplaylargedistributionnetworks.


9

• R4:Randomlydistribute loadsand/orproduction intobusesatEuropeanLow

VoltageTestFeeder.

These requirements will be developed within the goals. The goals consist on the

milestonesrequiredforthefulfillmentoftheobjectiveoftheproject.Thegoalsmayas

wellbedividedintosubgoals.Foreachofthegoals,itwillbestatedalsowhereinthe

projectitisachieved.

§ G1:ModifyMosaikexampledemocodeinordertofulfillrequirements(R1,R2

andR3).[Achievedinchapter3]

o G1.1: Modify PyPower configuration file and prepare helper scripts.

[Achievedinchapter3.2]

o G1.2: Obtain load and/or production profiles from an open database.

[Achievedinchapter3.2.3]

o G1.3:Prepareconfigurationscriptstodistributeloadand/orproduction

profiles on thebuses of the implemented European LowVoltage Test

Feedernetwork.[Achievedinchapter3.2.3and3.2.4]

o G1.4:ModifyvisualizationfilestodisplayallEuropeanLowVoltageTest

Feederinthevisualizationbrowser.[Achievedinchapter3.2.5]

§ G2: Validate the obtained results against the results provided in the

documentation.[Achievedinchapter4]

§ G3: Select a smart grid scenario to be implemented in the tool. [Achieved in

chapter2]

§ G4:Developanenergytheftdiscoverymethod.[Achievedinchapter5]

o G4.1:Performliteraturereviewonenergytheftdiscoverymethodsand

selectamethodtobeimplemented.[Achievedinchapter5.1]

o G4.2:Modifytheselectedenergytheftdiscoverymethod.[Achievedin

chapter5.2]

§ G5:Preparethetooltoexperimentwith.[Achievedinchapter6.1]

o G5.1: Prepare models and datasets for the investigated method.


§ G6:Performseveraltestcasesontheselectedscenarioandgathertestresults.



10

§ G7:Interpretthetestresultsanddrawconclusions.[Achievedinchapter6.3]

§ G8: Provide recommendations for usewith the developed tool. [Achieved in

chapter7]

§ G9:Publishthedevelopedsoftwareinasharedrepository.[Achievedinchapter

7]


11

2.-LVNetworkDescriptionInthischapteritisdescribedthewholelowvoltagegridthatisusedduringthewhole

project.Inordertobetterexplainthedifferentpartsofthegrid,itisshowntheoutput

ofthetooltosimulatethegrid;thisisanExcelfilecalledGrid.xlsx;thecreationofthis

Excelfileisexplainedinchapter3.Thebestwaytodescribethenetworkisgoingthrough

eachofthesheetsintheGrid.xlsxExcelfilecreatedforthePyPowersimulator.

Inordertohaveatoolthatisabletocreatescenariosandmodelsasstandardaspossible

ithasbeenconsideredallover theproject that the informationof thegrid isalways

giveninthesameformatastheonefromtheEuropeanTestFeeder;therefore,thetool

configuredismorethanenoughtodevelopthesimulationsneeded.Asaresult,during

this project, and in the different experiments performed, the characteristics of the

physical network are to be left unchanged. The European Low Voltage Test Feeder

consistsofaradialnetworkasseeninFigure1withanominalfrequencyof50Hz.Inthe

northwesterspointofthenetwork,thereisasubstationwithatransformerthatlinks

themediumvoltagetothelowvoltage.Thetransformeris11/0.416KVtransformation

rate.

Figure1:NetworkDescription.[2]


12

2.1.-Nodes

Thefirstofthesheetsisthenodessheet.Inthissheet4differentcolumnsmaybefound.

Thefirstcolumnisthenameofeachnode.Forsimplicitypurposes,PyPowerconsiders

thatthereferencenodeisthehighsideofthetransformerandthesecondarysideis

consideredasthefirstnode.Asinthegridisnotmeanttohavegeneratorsin,allthe

nodes are considered to be PQbuseswith the exceptionof the primary side of the

transformerthatistheReferencenode.Themeaningofthereferencenodeisthatitis

thenodewiththeangleof0º,sotherestofthenodeshavedifferentanglesbutwith

comparisonto thereferenceone.The fact that thenodesarePQmeans thatall the

nodes are considered as consumers, none of it has network stability or generation

purposes.Finally,thethirdcolumnisusedforhavingthebasevoltageofeachofthe

nodes.Asitisobvious,thereferencenodehasthebasevoltageofthehighsideofthe

primarysideofthetransformerandtherestofnodeshavethevoltageofthesecondary

sideofthetransformer.Inthegridthatisusedduringtheproject,thereareatotalof

906nodesplustheprimarysideofthetransformerasareference.Thebasevoltages

thatareusedare11kVand416Vrespectively.

AscreenshotofthenodesistheexcelfileisseeninFigure2:

Figure2:ScreenshotofNodes.[Grid.xlsx]

2.2.-Lines

Thenextworksheetisdedicatedtothelinesinthegrid.Thefirstcolumnisusedforthe

nameofthelineonly;thenthesecondandthethirdcolumnareusedtodefinewhichis

thenodeoforiginandwhichoneisthedestinynode.Aseachofthebranchesmayhave

adifferentkindofwireconnectingthenodes,inthefourthcolumnitcanbeseenwhich

kindofcableisconnectingthenodes.


13

Thenextcolumnisusedforhavingthelengthoftheline,expressedinkm.Thecolumn

namedOnlineisusedtodefinewhetherthelineisconnectedornot.Inthisscenarioall

thelinesarealwaysconnected.Thereisonelastcolumnthatonlyappliesforthefirst

row that is called Taps; this value is used to define which transforming rate the

transformer is using. There is a total of 905 lines plus the transformer, which for

simplicitypurposesisconsideredasaline.

AscreenshotofthelinesistheexcelfileseeninFigure3:

Figure3:ScreenshotofLines.[Grid.xlsx]

2.3.-Linetypes

Linetypesworksheetisusedtodefineallthedifferentkindsofwiresthatcouldbeused

inthenetwork.Inthisproject10differentkindofwiresareconsidered.Foreachofthe

wires there are four columns to describe the characteristics of the wires, this is,

resistance,inductanceandcapacitance.Plus,anextracolumnforthemaximumcurrent

thatcangothroughthewire.Thefirstofthecolumnsisusedforthenamingofthekind

ofwirethatisusedbeforeintheLinesworksheet.

AscreenshotofthelinetypesistheexcelfileisseeninFigure4:

Figure4:ScreenshotofLineTypes.[Grid.xlsx]


14

2.4.-Transformertypes

This lastworksheet isusedtodescribethedifferenttransformersthatmaybe inthe

network.Asinthisprojectonlyonetransformerisconsidered,onlyonetransformeris

described. There is a column for the name of the transformer, another one for the

nominalpower,themaximumcurrentsontheprimaryandsecondaryside,thepower

loss and the impedances of the transformer and one last column to define which

transformationsratethetransformercoulduse.

AscreenshotofthetransformertypesintheexcelfileisseeninFigure5:

Figure5:ScreenshotofTransformerTypes.[Grid.xlsx]


15

3.-SystemDescription

3.1.-IntroductiontoCo-simulationwithMosaik

Co-simulation is the termusedwhendifferent subsystems (so called simulators) are

modeled, and simulated, in a distributed way. This means that, in co-simulation

software, each of the programs are independent from the others. Having different

simulatorsworkingatthesame,insteadofjustonlyone,timehasmainly2advantages.

Ononehand, it represents lesscomputationalefforthavingtheprocessesseparated

andthenexchangingthenecessaryinformationbetweenthem.Ontheotherhand,and

mostimportant,itallowstoreuseexistingmodelsandsimulatorsofcomplexscenarios

that have been developed, tested and verified; instead of having to develop these

simulatorsinsideasinglemultidomainsimulationenvironment.Forexample,anuclear

plantmayhaveaverycomplexsimulatorthatworksproperly;withco-simulationone

wouldonlyneedtodefinewhichdatahastobeexchangedbetweenanuclearplantand

therestoftheinvestigatedpowersystem.

Thisindependencyavoidshavingcouplingproblemswhilestillthedifferentsubsystems

areabletoexchangedatabetweeneachother.[3]

Asimplewayofdepictingtheideaofco-simulationmaybefoundFigure6:

Figure6:Co-simulationStructure.

Synchronizer

Model1 Compiler1

Model2 Compiler2

Model3 Compiler3


16

Inthisproject,thesynchronizerthatisusedisMosaik.Theroleofthesynchronizeristo

exchange the data and configure and execute simulators. It is very important to

understandthefundamentbehindtheprogram.Mosaikgrantsthepossibilitytohave

severalsimulatorsandsimulationmodelsworkingtogether,andasaresult,havingthe

possibilitytoanalyzelargeSmartGridscenarios.Thisprogramallowstheusertocreate

andrunco-simulationsetupsworkingatthesametime.

The whole co-simulation system is made of different models that can even have

different compilers and they communicate between each other through the

synchronizer,thatinMosaikitiscalledOrchestrator.ThewayMosaikco-simulatesand

exchangedatabetweenthedifferentsimulatorsisthankstoadiscreteeventsimulator

library.Eachofthedifferentsubmodelsisrunningwiththeirownsimulatorinstanceor

processandcommunicatewiththeothermodelswhenacertaineventoccurs.Thisway

ofinteractingallowsaswelltohaveeventsthatareyielded,orinotherwordsstopped,

until a certain event happens. There is also a possibility of having asynchronous

simulations. [4]ThewayMosaikcoordinatestheexecutionofthesimulatorsandthe

data exchanged between them is thanks to anOrchestrator. TheOrchestrator is an

automatedcoordinatorandmanagerofcomputersystemsandservices.Itcanbeseen

asanadvancedautomationsystemandcontrolmethod.[5]

MosaikisanApplicationProgrammingInterface(API)whichallowstosettheframework

for Smart Grid co-simulation. Mosaik is written in Python language. Python is an

interpreter (like C++ or Java) object oriented with a high-level and dynamic

programming language [6]. Mosaik is an open source software that provides a

demonstrationscenariofortheuserstobeginworkingwithandunderstandingthetool.

In this thesis, thedemo,provided inMosaikwebpage,hasbeenusedas thestarting

point for the implementationof the chosenpower system scenario.As a result, the

demo provided with slight modifications is used in this chapter to explain the tool

capabilities. Inside the demo, there are four simulators that are used with some

modificationsthroughoutthewholeproject.Inthischapter,itisexplainedthewayeach

simulatorworks and howMosaik joins them together. Before describing them, it is

necessarytomakeadifferencebetweensimulatorsandMosaikadditionaltools.


17

Simulatorsareusedtosimulateoremulatearealsystemwheretheadditionaltoolsonly

aidtheco-simulationsetup,withforexamplevisualization,datalogging.Thesimulators

are in a domain that is interesting in the co-simulation system (power, ICT, energy,

mechanical,social,etc.)thatcontainsobjectsarepartofthemulti-domainscenario.

Thenameofeachsimulatorandashortdescriptionofeachofthemislistedbelow:

• Mosaik_csv:Thisisasimulatorthatisusedtosimulatecommaseparatedvalues

(CSV) files. This simulator is able to open, a csv file, read it and store the

informationindifferentvariables,basedoncolumnnames.Thissimulatorcan

beusedtosimulateanyrecordedsystembehavior;forexample,timeseriesofa

homepowerconsumption.

• Mosaik_PyPower: This simulator is an adaptation of the optimal power flow

solvernamedPyPower.Thissimulatorisinchargeofcalculatingthepowerflow

andthevoltagedropinthenodesofthesystem.

Thenameofeachadditionaltoolandashortdescriptionofeachofthemislistedbelow:

• Mosaik-hdf5:Itisnecessarytostoretheinformationproducedinthesimulation.

Thistoolgathersallthedatafromthesystemandsaveitwithaspecificstructure

intoahdf5file.Thetoolcanbeconfiguredtostoreonlyselectedtimeseries

• Mosaik-web: This tool provides the web visualization of the scenario. It is

possibletoselectthewaythecomponentsintheset-uparedisplayedintheweb

browser.Itispossibletochangethelocationinthescreen,thecolor,theshape

andmanymoreattributes.

Both simulators and tools are connected via Mosaik simulator API. In the Mosaik

ecosystem, tools to recordordisplaydataare treatedas simulators.Thesoftware is

usedinthefollowingwaythroughoutthedemo.Firstthesimulatorsareinitializedin

Mosaik.ThentheCSVsimulatorisusedinordertostoretheloadprofilesintovariables

thatareusedinthesystem.Atthesametime,thePyPowersimulatorisruntocalculate

thepowerflowbetweenthebranches.


18

Theinformationofthelocationofthebranches,theirlongitudeandtheirimpedancesis

loadedfromaJSONfilethathastobegiventothePyPowerprogramasaconfiguration

file. Manipulating this file provides the user with the possibility to change the grid

configuration.AlltheinformationloadedfromtheCSVandJSONfilesandtheoutputof

thePyPowersimulationaresavedwith theuseof thehdf5simulator.This simulator

takesallthedataandsaveitstructurallyintoahdf5file.Finally,thewebsimulatorallows

theuser to see the topology that is beingused and also real time values calculated

throughthepowerflowsimulator.

Figure7showstheoutputoftheco-simulationviathewebvisualization.

Figure7:DemoOutputofMosaik.

Asitisseeninthefigure;thereisanumberofnodesthatcanbevisualized.Oneofthe

nodesisdeterminedasthereferencebus(RefBus)andtherestaredefinedasPQBus.

TheconnectionsbetweenthedifferentPQBusesandwiththeRefBusarethephysical

lines inthenetwork.Allthisnodesandconnectionsareintroducedandsimulatedby

PyPower. Component nodes, like for example Batt and PV, represent the power

consumptionandproductionthatisconnectedtoaPQBus.TheBattandPVnodesare

connectedtothePQBusbyICTconnections;onlydataisintroducedinthePQBus.This

BattandPVconsumptionarebeingsimulatedwiththeCSVsimulator.Theconnection

betweenthetwosimulatorsisdonethankstoMosaik.Inordertosimulateanimpactof

aPVproductionorthepowerflowofthePQBus,thedatabetweenPVsimulatorand

loadflowcalculatorneedtobeexchanged.Mosaikfacilitatesthisexchangeandre-runs

calculationsforeverysimulationstep.


19

3.2.-ConfigurationofMosaikandPyPower

Inordertosimulateapowersystemnetwork,PyPowerneedstheinformationofthe

characteristicsofthegridandtheloadsinaveryspecificformatandstructure.However,

theinformationandcharacteristicsofthesystemarepresentedintheEuropeanTest

Feederinanothercompletelydifferentformatandstructure.Asaresult,itisnecessary

toreadalltheinformationthatiscontainedinthesource,modifyitandafterwardssave

itinthecorrespondingformatforMosaiktoconfigurePyPowercorrectly.Inordertobe

abletocreateandsimulateascenariosomestepshavetobefollowed.First,ithastobe

developedafilewithalltheinformationfromthegrid.Secondonevariablewithallthe

informationoftheloadshastobedefined;andofcourse,oneloadprofilefileforeach

oftheloads.Afterwards,theco-simulationsetupisreadyforexecution.

3.2.1.-Co-SimulationSetupImplementedinMosaikOverview

Beforeexplaining indetailhoweachof thedifferentpartsof theprogramworkand

communicatewiththeotherparts,itisinterestingtographicallyexplaintheoverview

oftheco-simulationimplementationsetup.Asitisexplainedthroughtheprojectthere

aremainly4pythonscriptsthatweredevelopedforthepurposeofthisthesis.Theyare

Var.py,Excel.py,LVTest.pyandOutput.py.Thesescriptsarepresentedinchapter3.2.2

to3.2.5.

Thesefilesaredividedinto2differentcategories;configurationfilesandhelperfiles.

Theconfiguration filesaremeant tobeaccessibleby theuser.Theusermaychange

themtosimulatedifferentscenariosorobtaindifferentoutputs.Var.pyandOutput.py

are theconfiguration files.Thehelper filesare filesmade toease theuseof theco-

simulationsetupaswellasfacilitatingfuturemodifications.Theyarenotmeanttobe

changedduringthescenarioreconfigurationoncecreated.Excel.pyandLVTest.pyare

thehelperfiles.

Theideaoftheconfigurationfilesisthatalltheinformationthathastobegiventothe

program is provided in Var.py script; and the information of the simulation that is

neededoraskedbytheuserasdescribedinthescenariocanbeselectedonOutput.py.

Therestoffilesarenotmeanttobetouchedorused,sothatthesystemcanusethem

internallytoperformthesimulation.


20

This way the risk of changing anything that is necessary is minimized and also the

applicationwould turn to bemoreuser friendly, avoiding large source code that an

unexperiencedusermaynotunderstand.

InFigure8 there isanoverviewofhowtheco-simulationsetup implementationhas

beendeveloped:

Figure8:OverviewoftheProgram.

Inthefigure,therectanglesarethepythonscripts(ingreentheconfigurationscripts

andinredthehelperscripts);theellipsesaretheinputtothescriptsandthetriangles

aretheoutputsofit.Thearrowsmeanflowofcommunicationbetweenthefilesand

theyhavealsodrawnthedirectionofthedataflow.

Theprocessisdescribedasfollows;theuseronlyneedstorunthescriptinOutput.py;

thisscriptautomaticallycallsLVTest.py.InLVTest.pyfirstofall,Excel.pyandVar.pyare

run.Excel.pyreadsallthevariablesfromVar.pyandtheinformationfromtheEuropean

Low Voltage Test Feeder (ELVTF) and formats it to the proper structure that the

simulator needs. The grid information is saved inGrid.xlsx. Afterwards, in LVTest.py

Mosaik iscalledandtheco-simulation issettledup.After thesimulation isdonethe

resultsofthesimulationarestoredintoseveraldifferentdocumentsbyOutput.pyfor

furtheranalysis.


21

3.2.2.-Var.py

Thisscriptisusedtodefinewhichcolumnshavetobetakenoutofthecsvfilesgivenby

theEuropeanLowVoltageTestFeeder.Itisimportanttosituatethesevariableshere

becauseitmightbepossiblethatwithfutureupgradesofthetool,theusermightbe

willingtoaddnewcomponentsorcharacteristicstothetool.Forexample,itcouldbe

possible to add the geographical coordinates of the nodes; that can be passed to

visualizationoranyothersimulationtool.Thisinformationisgiveninthecsvfilesbutat

themoment is not used. It also has three very important variables in the proposed

scenario;thestartdate,theenddateandthenamethatisusedfortheExcelfilewith

allthegridinformation.

ThewaythevariablesaredefinedinthescriptcanbeseeninFigure9:

Figure9:Variables.[Var.py]

3.2.3.-Excel.py

Withthepurposeofavoidinglargescriptsandanykindofcouplingissues,anewscript

calledExcel.pywascreated.Thisscripthastheobjectiveofcreatingandformattingall

thenecessarydatatobeprocessedbythePyPowertool.Tobeginwith,thedifferentcsv

fileswiththeinformationofthegridandloadshavetobereadandstoredintovariables.

Duringthisproject, it isassumedthatthistool isusedalways. It isassumedthatthe

structureoftheEuropeanLowVoltageTestFeederfileisnotmodified.

AsthefileformatthefeederisCommaSeparatedValues(csv)files,anewlibraryforthe

scripthastobeimported.Thislibraryisthecsvlibrary.


22

Toautomatizetheprocessofextractingtheinformationfromthecsvfiles,afunction

hasbeencreatedtoallowtheusertoextracttheinformationneededpercolumnsfrom

the csv file and storage the data into a variable. To storage the information into

variables,aspecialstructureofvariablehastobeused,theyarearrays.Tobeableto

createarraysandworkwith themafterwards,another libraryhas tobe importedas

well.Thisisthenumpylibrary.

Thesourcecodementionedbeforetoextracttheinformationfromthecsvandstorage

themintoavariableisshowninFigure10:

Figure10:ColumnExtractionFunction.[Excel.py]

Inthisfunctionthereare3inputs,csv_namethatisthenamewiththecorresponding

paththecsvinquestionhas;columnsthatisavariablethathasthenameoftheheaders

ofthecolumnsthattheuserwantstoextract;andfinallyrepthatisthenumberofrows

thathavetobedeletedfromthecsvbeforetreatingthedata.

Thenameandpathofthecsvfileisnotgoingtobeaccessiblefortheuser.Itisnecessary

tohavethecsvfileswiththeoriginalnamefromthefeederwebpageandinsideafolder

intheprojectcalledLVData.Thereasonfordoingthis,isthatthiswayitisavoidhaving

missingdata,ordatachangedbytheuser.


23

Thevariablecolumns,withthenameofthedifferentcolumnstheuserwantstoextract

fromthecsvandstorageintovariables,isavariablethatisdefinedbytheuserinanother

script,Var.py.Itisclearthatitwouldbenecessaryfortheusertoatleastopenthecsv

filesandunderstandthemalittlebitiftheuseriswillingtoextractanothercolumnthat

theonesdefinedbydefault.

Oncethisfunctionhasbeencreated,itisexecutedinallthesevencsvfilesprovidedin

the European LowVoltage Test Feeder to extract the information. This csv files are

Buscoords.csv, LineCodes.csv, Lines.csv, Loads.csv, LoadShapes.csv, Source.csv,

Transformer.csv.Itisnotworthytogoovereachofthefilesprovidedinthefeederand

explainingtheirinformation.However,itisnecessarytounderstandwhichinformation

isprovided to the simulator, thishasbeenalreadydescribed in chapter2, asall the

informationisstorageintheExcelfile.

Nowthatalltheinformationhasbeenreadfromthecsvfiles,ithastobesavedtoafile

forthesimulatortocreatethegrid.Thesimulatorthatreadandrunthegridshapeis

PyPower;thissimulatorneedstohaveintroducedtheinformationfromthegrid.The

wayofinputtingthegridinformationintothesimulationcanbedoneusingonlytwo

differentkindoffiles,JSONandExcel,bothformattedinthecorrectway.

Originally,inthedemoversionofMosaik,thisfilewasaJSONfile;butitwasdecidedto

savetheinformationintoExcelformat.

Thereasonbehindthisdecisionweremainlytwo.First,havinganExcelfileallowsthe

usertoautomaticallyunderstandthestructureoftheinformationthatisrepresented.

Second,ifsmallchangesaretobemade,theExcelfileismoreaccessibletotheuser.

ThenamethisExcel filehas isGrid.xlsx;andthestructurefromthefile isalwaysthe

same,forthePyPowersimulatortoreadthedata.


24

Functionexc_insert,aspresentedinFigure11createstheGrid.xlsxfile.

Figure11:ExcelInsertFunction.[Excel.py]

Basic Python packages are not able to create and write Excel files, but there is an

externallibrarythatcanbeusedtodoso.Thislibraryiscalledxlsxwriter.Thevaluesare

introduced in Excel one by one, but itmakes theworkmuch easier if a function is

created,thatintroducesthedatajustcellpercell,rowperrow,columnpercolumnor

asamatrixforlargeamountofdata.Itisalsousefultoinserttheheadersofthedifferent

columnsinExcel;butthisisjustfortheuserinformationasthefirstrowoftheexcelfile

isnotreadbythePyPowersimulator.

Beforegettingintoexplainingthedifferentwaystheinformationmaybeintroducedin

theExcelfile,itisexplainedwhatisthepurposeofthedifferentvariablesthatenterin

thefunction.Varisthevariableusedastheinputdatathatneedstobeintroducedin

theExcelfile.


25

Itcanbefromanysizeorformat,butdependingonthewayitwantstobeintroduceda

distinct“if”caseisused.Variablesrowandcolarethestartingrowandcolumnwhere

itisnecessarytointroducethedata.

The ending row or column is not necessary to be given as an input because it is

automaticallycalculatedwiththelengthofthevariable.Then,variablekindisusedto

specifythewaydatahastobeintroducedintheExcelfile,orinotherwords,whichcase

hasbeenselected.Thereissomeinformationthatisnotpresentinthecsvfilesfromthe

feeder;whenthissituationhappens, the informationhastobe introducedmanually,

withthatpurposeonmindthevariablenumeroisused.

Finally,thedatareadfromthecsvfilesisinaspecificformat,thatisastringformat;for

thesystemtoworkcorrectly;thecellsthatrepresentvalues(thisisforexamplelengths,

voltagesorcurrents)havetobeintroducedwithanumberformat,thatiscalledfloat.

Thelastvariablecalledftlisusedtodeterminewhichcolumnisthefirstcolumnfrom

withtheExcelwritertriestoconvertthedatafromstringelementstofloat.

TherearefivedifferentwaysofintroducingtheinformationintotheExcelfile.Itcould

beintroducedjustonedatainonecell;thisiscontemplatedinthecaseOne.

Also,itcouldbeintroducedasamatrixofelements,thisistheeasiestwaytodoit,but

theinformationhastobeall introducedinthesamevariable;thisiscontemplatedin

caseMatrix.RowisusedtointroducethedatarowperrowandHeaderworksexactly

thesamebutitisusedonlyfortheheadersofthedifferentcolumns.Atlast,Multipleis

usedtointroducedthevaluespecifiedtheamountoftimesofthevariableintroduced.

Duetothewaytheinformationisgiveninthecsvfilesfromthefeeder,onevariable

containsthenames,locationsandshapesoftheloadsandanothervariablecontainsfor

eachshape,the loadprofilethathastobeused.Anotherfunctionwasdevelopedto

matchthistwovariableandendwiththevariableLoads,thathasforeachshape


26

This function isnotuniqueforthispurposeandcanbeusedforanymatchbetween

variablesthatisnecessary;itisrepresentedinFigure12:

Figure12:CheckandCopyFunction.[Excel.py]

Finally,the loadshapeshavetobereformattedtothestructurethattheMosaik_csv

simulatorusestoreadcsvfilesandtranslatethemintoloads.

This function is created, having inmind the format the European Low Voltage Test

Feedercsvfilesare,butwhenadifferentscenarioisrunonthelowvoltagegrid,other

loadprofilesareusedandveryprobablywithdifferentformat;therefore,thefunction

wouldneedtobereadjustedinthefuture.

ThemostimportantinformationisthattheMosaik_csvsimulatorneedsacsvfilewith

thefirstrowasthenameofthevariable;thesecondrowwiththenameoftheattribute

thevariablehas,inthiscase,itispowerinW.

Fromthethirdrowon,dataisstructuredintwocolumns,thefirstcolumnisthedate

withthefollowingformat:YYYY-MM-DDHH:mm:ss;andthesecondcolumnisthevalue

oftheattribute.

ThesourcecodeisshownintheFigure13:

Figure13:RearrangeCSVFunction.[Excel.py]


27

Thewaythisfunctionworksisthatfirstitopens,readsandstoresthedatainavariable

fromeachcsvshapefile; thenthe informationfromthevariable is introduced inthe

properformatintoanothercsvfilewithexactlythesamenameastheoriginal,butina

differentfolder.

3.2.4.-LVTest.py

Allthefilesandvariablesarereadytobeusedbythedifferentsimulatorsprovidedby

theMosaik demo, only some little readjustments have to be done. As a result, it is

explainedindetailhowMosaikworksandthedifferentinformationthathastobegiven

tothesimulators.WhenusingMosaik;firstallthedifferentsimulatorsmustbedefined

andinitialized.ThePyPowersimulatorhastoreadtheExcelfilewiththeinformation

fromthegridtobeabletocreateit.Oncethisisdone,theprogramisreadytocreate

the different loads that are connected in the grid. As eachof the loadsmayhave a

differentloadshapeorprofile,theMosaik_csvsimulatorhastobeinitializedforeachof

theloads.Thisisnotsystematicandtherehastobecreatedasmallfunctiontocreate

theloadswiththeirownprofileandstorageallthatinformationinsideavariable.

ThewayithasbeendoneisshowninFigure14:

Figure14:LoadCreationFunction.[LVTest.py]

Thevariableloadresultsinavariablewiththeinformationofthelocationandtheshape

ofeachoftheloadsinthenetwork.Aseachoftheloadsisconnectedinaspecificnode,

acertainfunctiontoconnectproperlytheloadshastobedeveloped.

Thecodecreatedtofirstidentifythenodesandthenconnecteachloadtotheproper

PQBusnodeisinFigure15:

Figure15:ConnectEntitiesFunction.[LVTest.py]


28

Therestofthecodeisverysimilartotheoneprovidedinthedemo.Theonlythingthat

maybenecessarytomodifyistherangewithinthenodesareconsideredtobeinlimits.

TheoutputofthesimulationcanbeseeninFigure16:

Figure16:MosaikVisualization.

Asitcanbeseen,duetothehugenumberofnodesinthesimulation,itisdifficultto

visualizethelocationofthenodes.Tryingtoimprovethissituation,thecharacteristics

of the visualization have been changed in the Web Visualization Simulator. The

advanced configurations have been changed in the JavaScript that contained all the

functionsthatintroducedtheinformationintothebrowser.

Thesizeofthevisualizationwindowwaschangedtoautomaticallyfitinthewindowsize.

Alsothesizeofthenodesandthelengthbetweenlinkswasreducedtotrytofitthem

asmuchaspossibleinthewindow.

Thelocationofthesitepackagethatismodifiedforimprovingthevisualizationislocated

in the following path: /Users/EzequielFredes/.virtualenvs/mosaik/lib/python3.5/site-

packages/mosaik_web/html/media


29

TheparametersthathasbeenchangedareseeninFigure17:

Figure17:WebVisualizationJavaScriptChanges.[mosaik.js]

Thecolorofthedifferentnodesinthesystem,this isReferenceNode,PQBusesand

Loadswaschangedtohaveamorevisualdifferenceandthenamesinthelegendwas

setcorrespondently.Allofthishadtobedonefromanotherfilecalledmain.css.Thisfile

containstheformatinwhichthevisualizationisshown.Wheneveranewkindofnode

wantstobevisualized,itisnecessarytocreatethenodeinthecssfileandatthesame

timedeclareitintheLVTest.py.Thenodesthathasbeennecessaryfortheprojectand

thecolorsselectedareshowninFigure18

Figure18:WebVisualizationCSSChanges.[main.css]


30

Finally,itcanbeseenthatthenodesarefilledinwithsomecolors;thesecolorscanbe

green, yellowor red.According to the instantvalueofanattribute, thecolorof the

insidepartofthenodechangesfromgreen(iftheattributeiswithinrange)toored(if

theattributeisoutofrange).

Originally, the values and attributes of interest are decided in the LVTest.py. So the

intention is not being able to change it at themoment. They are referenced to the

maximumvaluesofpowerallowedinthegrid.

In the future, the script may be modified to be able to change the values in the

configurationfiles.Rightnow,thewaythattheyaresetiswiththecodefromFigure19

inthescript:

Figure19:SettingVisualizationConfiguration.[LVTest.py]

Withallthemodificationsthathavebeenmade,thewebvisualizationhasbeenslightly

improved;inFigure20thenewwebvisualizationcanbeseen:

Figure20:ImprovedWebVisualization.


31

3.2.5.-Output.py

Output.pyisascriptcreatedwiththeonlypurposeofautomatizingtherunningofthe

co-simulationandatthesametimeperformvisualanalysisoftheoutputofthetool.

Theobjectiveofthisfileistorepresenttheoutputoftheprogram,thisisthecurvesof

voltageandpowerinthedifferentnodesofinterestsandafterwardsapplythetoolused.

First,ashortcommandiswrittenwiththeideathattheapplicationis launchedfrom

here; so from now on, it is only necessary to run this script to run the whole co-

simulation.

Then, introducingawhile function, the systemwaitsuntil the simulation is done, to

accessthedatabaseandprintthenecessaryinformation.

Inordertoaccessthedatabase,duetothefactthatthefileformatishdf5anewlibrary

hastobeimportedtobeabletogetthedata.Thislibraryiscalledh5py.Aswell,inorder

toplotthedifferentcurves,anotherlibraryhastobeimported.Thenameofthisnew

libraryismatplotlib.

Theoutputof theprogramisperformed intwoways.First,aplot isdonetovisually

watchtheoutputofthedesiredvariable.Thisfigureisautomaticallysavedinthedesired

folder.Afterwards,thenumericalvaluesaresavedinanewexcelfilewiththepurpose

offuturetreatmentandevaluation.

Thecodeusedineachofthevariablestoprinttheplotandsavetheinformationinan

excelfileisinFigure21:

Figure21:OutputPrintingFunction.[Output.py]


32

Thiscodejustshowediscopiedforeverysinglevariablethattheuseriswillingtoprint.

TheoutputplotfromthecodecanbeseeninFigure22:

Figure22:OutputPlotExample.

Inchapter4,thetooloutputisvalidatedagainstthevaluesprovidedintheEuropean

LowVoltageTestFeeder.


33

4.-EuropeanLowVoltageTestFeederValidationIn this chapter it is verified whether the created co-simulation setup reflect the

EuropeanLowVoltageTestFeedergrid.Thiskindofanalysishastobemadepriorany

kindofexperimentation,toassuretheprecisionofthetool. Inordertoevaluatethe

validityofthetool,thefirstthingthatneedstobedoneiscomparetheresultsobtained

inthetestsimulationwiththedataprovidedinthedocumentationoftheEuropeanTest

Feeder;thiswayitispossibletoevaluatetheaccuracyofthesolution.Beforeanykind

ofanalysisismade,ithastobesaidthatintheEuropeanLowVoltageTestFeeder,the

informationanddataisprovidedphaseperphase,howeverinthetool,onlythethree

phaseelectricpowergridisbeingconsidered.Allresultsfromthesimulationarebeing

storedintoanExcelfile.ResultsofthevalidationarestoredinValidation.xlsxfile.This

excelfilecontainstheinstantpowercalculationofthe2simulatorsusedintheEuropean

LowVoltageTestFeederandalsotheoutputoftheMosaikco-simulationsetupcreated

inthisproject.Thevariablethatiscomparedisthepoweratthesecondarysideofthe

transformerinthesubstation.

First,therealoutputgivenintheEuropeanLowVoltageTestFeederisshowedinFigure

23.ItcanbeeasilyobservedthesimilaritywithFigure22fromchapter3.2.5.

Figure23:RealOutputFeeder.


34

Withthisinformation,thedifferencebetweentheFeederpowerandthepowergiven

bytheMosaikco-simulationiscompared.Thisdifferencerepresentedinpercentageis

showedinFigure24:

Figure24:PercentageDifferencebetweenFeederandCo-simulation.

TheaccumulatederrorwascalculatedandcanbeseeninFigure25:

Figure25:AccumulatedErrorinPercentage.


35

Theoverallerroroftheenergywasof0.014%.Forthepurposeofthisinvestigation,the

errorisacceptable

OtherimportantcalculationsaretheRootMeanSquareError,whichisof117W.Alow

valuecomparedtothemeanvalueoftheinstantpowerisabove20kW.Thereisalsoa

boxchartoftheerror,inFigure26,thatshowsthatmostofthevaluesareclosetothe

medianwithsomeoutliers.

Figure26:BoxChart.

-200 0 200 400 600 800 1000 1200 1400

BoxChart


36

5.-ApplicationThereisabroadrangeofpossibilitiesthatco-simulationwithMosaikbringsinregards

toSmartGridsscenarios.Usersareabletoloadandsimulatehugescenariosandthanks

totheco-simulationpossibility,itisabletoconnectdifferentsimulationinstancesinto

thenetwork;thiscouldbeasimulationofapowerplan,orasimulatorofaveryspecial

consumer.Themost straightforwardapplication that theprogramoffers is real time

monitoringofthegrid,wheretheusercancheckthestatusofanynodeandthehistory

ofit.Thetoolcouldbeusedforadifferentkindofapplications;forexample,tofindand

estimate weak points in the grid where some sort of compensators should be

introduced.Thetoolcouldalsobeusedtoforecastthefutureenergyneedsandenergy

balance.

Inthisproject,variousoptionswerestudied,suchasnetworkrelianceorchallengesof

networkhacking.Ithasbeenselectedthestudyofenergytheftastheapplication.At

themoment,thereislittleadvancedmeteringpracticestoevaluateandstudyenergy

theft.Withtheongoinginstallationofsmartmetersandadditionalmeasurementsinthe

distributiongrid,advancedmethodscanbeusedtoidentifyenergytheft.Thepossibility

ofremotelyidentifyingenergytheftsisaninterestingtopictobestudiedintheproject.

Someauthors,suchasRongJiangetal[9],StephenMcLaughlingetal[10]andSergio

Salinasetal [8]; focus the studyof theenergy theft innetworkattacks in the smart

meters.Thetechniqueusedbythemismorerelatedtocomputersciencesprocedures.

Thetechniquesconsistofclassificationproceduresand/oramathematicalmodelbased

on“honestycoefficients”.Thetoolthathasbeendevelopedinthismasterthesisismore

orientedtopowersystems,soothertechniquesarestudied.

With use of prepared co-simulation setup, it is possible to automatically generate a

networklayoutandvisualizeit,andatthesametime,introducenewpowernodesinto

the system. During the first phase of this project, only loads or consumers where

introduced,but it ispossibletointroduceotherkindsofnodessuchasgeneratorsor

compensators.Afterallthisdataandinformationareautomaticallyintroducedinthe

simulator,itispossibletoobservetheoperationofthepowersystem.


37

Alltheoutputfromthesimulatorissavedintoadatabasethatafterwardsissavedand

couldbeused for future studies. Finally, studying theenergy theftspossibilitieswas

chosenasthemostappropriateapplicationwiththegivingbackground.Theapproach

choseninthisworkistousehistoricalmeasurementstoidentifyenergytheft.

Energytheftrepresentapercentageofthelosesthatarepresentedintoday’spower

systems.Asthisactivityatthemomentdoesnotrepresentarisktothestabilityofthe

grid, thepurposeof the tool is tohave real timemonitoringof thegridbutwithout

evaluating any kind of energy theft. Afterwards, with the information from the day

before,thetoolisruntoevaluatewhetheralltheconsumersinthenetworkwerehonest

orifthereisanykindofenergytheftintheareaandsubsequentlyidentifythetheft.

Thenextsectionintroducesenergytheft,explainshowitoccursandfinallydescribes

differentprocedurestodetectit.

5.1.-EnergyTheft

Energy theft consists of the illegal, unsupervised and unauthorized use of electricity

fromthegrid.Themainreasonswhyenergytheftisanissueare:

• Safety:Unsupervisedusecancauseelectrocutionsorfires:

• Economical:Thisenergystolenismoneynotpaidtothegridadministrator.

Themostcommontechniquesforenergytheftare[7]:

1. Hiddenconnectioninfrontofthemeter.

2. Physicaltamperingwiththeterminals.

3. Physicaltamperingwithananalogmeter’smechanism,suchasamagneticfield.

4. Tamperingwithadigitalmetersoftware.

There are different ways and techniques the energy companies have nowadays to

evaluateenergy theft. In the future,with thepossibilityofhaving smartmeters, the

electricalcompanieswouldbeabletodomoreadvancedevaluationoftheenergytheft.


38

However,thecompaniespossessingandusingtheinformationoftheloadprofileofthe

smartmeters,posesaconcernregardingprivacyofthecitizens.

Thecompanycouldmakeuseofthisprofilesandforwardittothirdpartiesthatcould

haveeconomicinterestsonit[8].

One common technique used to find energy thefts is using the so called honesty

coefficients[9].Withthesecoefficientsitcouldbeevaluatedovertimeifaconsumeris

moreapproachedtobeanenergytheft.Theadvantageofthistechniqueisthatisnot

necessary tohave themeasurementsof thesmartmeter inorder to identifyenergy

theft;thisway,privacyoftheconsumersisassured.

Another way is observing all the different variables at the same time and all the

possibilitiesofenergytheftatonce[10].Thissystemavoidshavingahighnumberof

falsepositivesbuttheamountofinformationnecessarytocalculateitmightbedifficult

tohandleandanalyze.

InthisworkwewillinvestigateandmodifyamethodproposedbyP.Kadureketal[11]

andevaluateitwithuseofdevelopedco-simulationandrealresidentialconsumption

date.

5.2.-EnergyTheftTool

TheprocedureproposedbyP.Kadureketal[11]consistsoftwodifferentsteps:

1. Detection of illegal use of electricity. In order to determinewhether there is

some energy theft in the system, an evaluation procedure is done at the

substation level. If thereexistsanenergyunbalanceat thesubstation level,a

sequencetoidentifythelocationofthetheftisinitialized.

2. Theftlocalizationprocedure.Inthisstep,theauthorassumesthatthevaluesof

thecurrentmeasuredbythesmartmetersiscorrectduetothetheftprocedure

that is evaluated. However, the values of the voltages could not match the

measuredcurrent.Therefore,withthemeasuredcurrentfromthesmartmeters,

theestimatedvoltagesatthelocationarecalculated.Ifthereexistadifference

betweenthesetwovalues,itisidentifiedtheenergytheft(s).


39

ThismethodissummarizedinFigure27:

Figure27:PaperMethodSummary.[11]


40

Themethodthat isusedinthismasterthesis ishavinganenergybalancestepanda

secondarystepofhistoricaldataofthedifferentloads.Thismethodissupposedtobe

runonceadaywiththedatafromthedaybeforeandthehistoricaldataoftwoyears.

Itisnotaverycompletetechniquebutitcannotbestudiedforwardinthisprojectdue

tothelackofinformationfromothervariablesthatcanbestudiedinenergytheft.The

tool is divided in two steps. First it is trained to see if it is able to detect whether

somebody is stealingenergy.Afterwards, it isprogrammedtosee if it isable to find

wheretheenergythefttakesplace.Thesetwostepstogetherconsistofsevendifferent

phasesthataredescribedbelow.

Aflowdiagramcanbedrawntoexplaintheproposedmethod,showninFigure28:

Figure28:FlowDiagramtoevaluateEnergyTheft.

Asaresult,theonlywayofenergytheftthatisevaluatedisphysicaltamperingofthe

metersattheconsumers’dwelling.


41

5.2.1.-Powerbalance

Themethodproposedinthisprojectistohavetheloadsmeasuredineachofthesmart

metersandloadflowiscalculatedfromthedata.Theresultfromtheco-simulationis

comparedtotheactualpowerthatismeasuredinthesubstation.

Phase1ofthetoolconsistonreceivingallthemeasurementsfromthesmartmeters.

Theco-simulationisperformedinPhase2andisstructuredthefollowingway;firstthe

toolisgivenallthemeasurementsfromthesmartmetersofalltheconsumersinthe

networkplusthemeasurementsatthesubstationlevel.Oncethetoolhasbeengiven

alltheinformation,itintroducesallthemeasurementsintothesimulationasconsumers

intheexactplacetheyarelocated,andthenasimulationiscarriedout.

Oncethesimulationisrun,theprogramhasallthepowerflowthroughthedifferent

linesinthesystemandasaresult,thepowerflowingfromthesubstationintothegrid.

Thispowerflowisthemostinterestingfortheapplicationintheveryfirststep.Thetool

hastocomparetheenergyconsumedduringthedaythathasbeencalculatedthrough

thetool,withtheonethathasbeengivenasdataasitwasmeasuredinthesmartmeter

connectedthere.Ifthesetwomeasurementsaresimilarwithinathresholdvalue,itis

determinedthatthereisnoenergytheftinthenetwork.However,ifthedifferencein

energies is higher than the threshold value, it is further studied if there is someone

stealingenergyfromthegrid.ThiscomparisonisdoneduringPhase3andthethreshold

valuehasthenameofδ1.

Thevalueofthethresholdneedstobeselectedcarefully,asithasaninfluenceonthe

accuracyofthedevelopedalgorithm.Butthequestionremainthere,howcanaproper

thresholdvaluebeselected?Theapproachfollowedtoselectthebestthresholdvalue

isexplainednow.First,a“standard”gridhad tobeselected; itwasdecided thanan

amountof75consumers isanaveragenumberof consumers inanetwork this size.

Then, a theftwas introduced in the simulationatdifferent locations in thegrid; the

networkconsistsof906nodes,so itwasconnectedatnodenumber1,300,600and

900. For each of the co-simulations, the percentage difference between the energy

consumedandthetheoreticalenergythathadtobeconsumedismade.


42

From the results in all the simulations, only the one with the worst percentage

differenceisbechosenasareferenceforstudyingenergytheft.Theresultthathasbeen

seen,isthatapercentagedifferenceof1%istheonethatbestsuitstheapplication.If

thepowerbalancethresholdisreached,sometechnicalornon-technicallosesoccurred.

Technical losescan includefaults,changes inpowersystemtopology,measurements

error. Non-technical loses include connection or disconnection of loads and/or

productionthatarenotreflectedinthemodel;oneexampleofnon-technicallosesis

energytheft.Thedistinctionbetweenthesetwodifferentkindoflosesisperformedin

Phase6and7

InsidePhase3thereisaniterativemethod.Thepowergridisdividedintosubgridswith

measurementsonthepointofthecommoncoupling.

In the iterative process the same energy balance is performed as before, but on

subsystems.Thesubsystemisdividedintwolevelsof2subsystems,soitcomparesthe

energybalance in the2 subsystemsandevaluatewhether there is imbalance in the

subsystem.Inthecasethatinthesubsystemthereisanimbalanceaswell,2moresub

subsystemsarestudied.Thiswaytheareawheretheenergytheftistakingplacecanbe

evaluated more precisely. In order to perform this kind of monitoring on real life

equipment,metermeasurementsareneeded,notonlyintheconsumernodes(smart

meters),butalsoonthedifferentlinesthatareevaluatedascriticalorimportant.The

reasonwhythesystemsaredividedintwobytwoisthatthepowerlinescanonlyhave

a divisionof twoat a time.Adivision in three is not allowed. The additional design

decisionisachoiceoflinesthatareconsideredcritical.Ithasbeendecidedthatthebest

wayofdefiningthesubsystemsandsubsubsystemsisfirstbytheline’sgeographical

locationandnextdistributingthenodesasevenlyaspossibleintosubsubsystems.

In the Mosaik visualization, geographical location is not depicted; however, in the

EuropeanLowVoltageTestFeedercontainsthelocation.

Dueto timeconstraintsandthe fact thatunderstandingandmanipulating JavaScript

filesisnotthepurposeofthisproject,anotherprogramwasusedtovisualizethenodes

andthepositionofthelines.ThisprogramiscalledQ-GIS.Thewaythisprogramworks

isthat,twolayershavetobeinsertedintheprogram.


43

Oneforthepositionofthenodesandoneforthestartingnodeandtheendingnodeof

eachofthelines.Thiswaynowitispossibletoseewhichofthelinesiseachone.

TheendingresultthatcanbeseenintheprogramisinFigure29:

Figure29:Q-GIS.

ThefinalshapeofthenetworkconsideringthesubsystemsisshowninFigure30:

Figure30:SystemsDesign.

Therearetwomainsubsystemsselected.Subsystem1beginsinLINE141.Thesecond

subsystemisbiggerandbeginsinLINE110.


44

Both subsystems are divided into two more sub subsystems; however, the first

subsystem is smaller than the second subsystem, so in order to graphically see the

difference,inFigure30onlythesubsubsystemsfromthesecondsubsystemareshown.

Insubsystem2therearetwosubsubsystems.Subsubsystem1beginsinLINE302;sub

subsystem2beginsinLINE296.

5.2.2.-NeuralNetwork

Thesecondpartofthetool,findinphase4and5.consistsofaNeuralNetworkthat

evaluates whether there has been a significant change in the consumption of the

houses.InordertotraintheNeuralNetworkatrainingdataandvalidationsetofdata

havetobeavailable.BeforeexplaininghowtheNeuralNetworkhasbeendesigned,itis

interestingtoexplainwhataNeuralNetworkisandhowitworks.AnartificialNeural

Networkisasystemthat is inspiredbythebiologicalneuralnetworksthatareinthe

animals’brains[12].Theyareusedtoprojectfunctionswhicharedependentofacertain

numberofinputs.Thewaythemodelprojectsorapproximatesthedifferentoutputsis

donewithdifferentkindsoflearningalgorithms.Thelearningalgorithmusedinthistool

isthefeedforward.Thisisthesimplestofallthelearningalgorithmsasthedataonly

movesforwardanddoesnotmakeanykindofloops.Intheneuralnetwork,itispossible

toselectthenumberofneuronsthatarenecessaryandthenumberofhiddenlayers

thattheuserwishestohave.Figure31isanexampleofthearchitectureofaNeural

Network.

Figure31:ArchitectureofNeuralNetwork.[14]


45

Inthisproject,anartificialNeuralNetworkisusedtomodeltheindividualhomeenergy

consumption.TheinputsthataregiventotheNeuralNetworkarethemonth,theday

inthemonthandthetemperature.Theoutputthatisexpectedfromthesystemisthe

energythatshallbeconsumedintheday.

During the project it has been investigated different artificial Neural Network

architectures;inordertoobtainthemostaccurateresultsaspossibleandareasonable

computationaltime.

Alltheinputdataisintroducedinayearlybasis;therefore,theideaistohaveonewhole

yearofdatainordertotraintheNeuralNetwork,anotheryearinordertovalidateit,

andfinallyuseitinthedayofthecomingyearwheretheenergycalculationisneeded.

TheNeuralNetworkcheckisdonethesamewaythepowerbalancecheckisdone,by

selectingthethresholdvalueδ2.Thisisappliedinthephase5.Themethodchecksthe

percentage difference between the given value in the smart meter and the value

estimatedbytheNeuralNetwork.DuetothefactthattheNeuralNetwork is justan

approximationof the reality and that thedifference that is introducedwhena theft

comes intoplay is50%oftheoriginalprofile.Thethresholdtodetermine ifa load is

stealingenergyhasbeensettoaδ2of50%.Alargethresholdvaluehasbeenchosen,

butwithmoreaccuratemodelsofthebehavioroftheloadsitcouldbereduced.


46

6.-ExperimentsDuring theprojecta seriesofexperimentshavebeenperformedwith theuseofco-

simulationtoolandproposedenergytheftmethod.

Prioranyexperimentation,areconfigurationofthetoolisneededinordertoperform

the needed experiments. The scenario that is implemented in the experimentation

consistof75housesconnectedinthenetworkobtainedintheEuropeanLowVoltage

TestFeeder.Thesehousesareconnectedevenlyinthesystem.

The75housesarecreatedwith18differentloadprofiles(obtainedfrompecanstreet)

withanaverageenergyconsumptionof23kWheach.Theoverallenergyconsumedin

thesystemisaround1750kWh.

Inthisnetworkconfigurationthetheftsaregoingtobeplaced.Theftsarecreatedwith

the same 18 load profiles as the houses; with the difference that the energy

consumptionisonlyhalfoftheoriginalloadprofile.Furthermore,whenatheftappears

in the configuration, what is expected to be seen is that the energy consumption

measured is less than the one calculated. Therefore, the thefts have to reduce the

energyconsumptionmeasuredinthesmartmeters,inresult,theyareintroducedinthe

systemasnegativevalueoftheloadprofile.Thismeansthatthetheftsactintheco-

simulationasifaproducerisconnectedtothenode.

Inthissectionthesubchaptersaredividedthefollowingway:

• Chapter6.1:Overviewofthereconfigurationinthetool.

• Chapters6.1.1to6.1.6:Detailedexplanationofthemodificationsinthetool.

• Chapter 6.2: Presentation of the results in the experiments and further

calculations.

6.1.-ChangesintheTool

Thesourceofdatathatwaschosenforthisprojectwaspecanstreet[13];therefore,the

tool had to be reconfigured to fit the new data format. All the code that has been

explainedandshownbeforeisthecodethatwouldbeusedinordertoworkwith,as

theideaisthatalltheinformationcomesfromtheEuropeanLowVoltageTestFeeder

orequivalent format.All thedataregardingthepowerconsumptionandtheoutside

temperatureisprovidedbypecanstreet[13]maintainingthesamegridlayout.


47

Firstithastobeexplainedhowthedifferentscriptshavebeenrewrittentoperformthe

proposedscenario.

Theoverviewofthedifferentchangesthatareperformed,willbeexplainedwiththe

sameFigure8fromchapter3.2.1thatwasusedtoexplainthedevelopmentofthetool

highlightingthechanges.

ThenewoverviewisshowninFigure32:

Figure32:OverviewoftheProgramwithChanges.

Thesimulationoverviewisverysimilarastheonepresentedinchapter3.Insideofthe

scriptssomechangeshavebeenmade(thatareexplained in thischapter).Themain

differenceisthatnow2newfileshavebeencreated.NN.pyisapythonscriptthathas

to be run separately from the Mosaik tool and calculates the expected energy

consumption of the nodes in the desired day. Comparator.xlsx is an Excel file that

gathersalltheresultsfromthedifferentsimulatorsandvisuallydefineanddiscoverthe

theftsintheregion.Intheoverviewtherunprocessisdefinedfor2differentscripts,

Output.pyandNN.py.Bothscriptsarenotconnectedsotheyhavetoberunbeforeany

resultscanbeseenbuttheorderoftheprocessdoesnotaffecttheresult.


48

Inordertoperformtheexperiment,scenariodesignermustbeabletointroducethefts

inthenetwork.Inchapter5itwasalreadydefinedthatthetheftdiscoverywasgoingto

be done comparing the theoretical energy consumption with the real energy

consumption.Thetheoreticalenergyconsumptioniscalculatedatthesubstationlevel

withthesimulationofthemeasurementsinthesmartmetersandinthehouseswith

theNeuralNetwork.Therealenergyconsumptioniswhatactuallyhasbeenreadbythe

smartmeters.

Itwasalsodiscussedinchapter5thattheenergytheftmethodthatisbeingstudiedis

physicaltamperingofthemeters.

In conclusion, the difference that should be expected in the measurements and

calculations is that therealenergyconsumption is lowerthanthetheoreticalenergy

consumption.Therefore,inordertosimulateatheft,theloadprofileusedisthesame

asintheloadsbutdecreasedby50%.Thistheftprofileisacting,notasaloadbutasa

producer;sotheeffect isthesameoneas ifaphotovoltaicpanelorabatterywould

havebeenconnectedinthenode.

Inordertoruntheproposedscenario,itisneededtoloadtheprofilesofthehousesin

MosaikandPyPower.Todoso,thefirstExcel.pythatwasusedissavedasacopyand

renamedafterExcel_Validation.py;thereasonforthischangeisthattheoriginalscript

wasusedtovalidatetheresultsfromthefeeder.Now,smallchangesaremadeinthe

Excel.pyfiletointroducethenewloadprofilesintothesystem.

The new configuration consists of 18 different houses profiles, that can generate

whatevernumberofhousesasneeded,connectedevenlytothegrid.Theideaforthe

powerbalancecheck is that firstasimulationwiththe loadprofiles fromthehouses

(theoreticallymeasured in the smartmeters) is performed. Then, the total power is

comparedwiththeactualpowermeasuredinthesmartmeter.However,therealpower

flowing from the substation is not a variable that is available. Therefore, a new

simulationisperformedwithavariablenumberofenergytheftstoobtainthepower

flowfromthesubstation.


49

Inordertohavethedatafromthetheoreticalpowerflowandbeingabletocompareit

withtherealpowerflow,thedatabase(hdf5file)fromthefirstsimulationiscopiedand

savedwithanothername,LVTheory.hdf5,andthevaluesfromthisfilearecompared

withtheonesobtainedintheLVTest.hdf5.

6.1.1.-Var.py

Thisscriptisoneoftheleastaffected,theonlythingthatisneededtoaddisthenumber

ofloadsandtheamountoftheftsthatareinthescenario.

Forthepurposeofthisinvestigation,severaltestswithdifferentamountoftheftsneed

tobeperformed

ThewaythattheyaredefinedisshowninFigure33:

Figure33:LayoutDefinition.[Var.py]

6.1.2.-Excel.py

Duetothefactthatthesamegriddistributionisused,thereisnoneedtochangeallthe

coderegardinginthisfile.TheloadswerecreatedfromtheEuropeanLowVoltageTest

Feeder,however,intheexperimentationpart,itwasdecidedtouseanyresidentialload.

Therefore,newloadsimulatorshavetobedeveloped,andthis isdonewiththedata

downloadedfrompecanstreet.

Inordertoincludenewloadsintheco-simulation,csvsimulatorisinitializedwithdata

downloadedfrompecanstreetdatabase.

However,aslightmodificationhastobemade.Aswitchcasehasbeenintroducedinthe

functiontoallowittodifferentiatebetweenloadsandthefts.Whentheprofileistobe

savedasaload,itisjustsavedthewayitis.Ontheotherhand,whentheprofileisa

theftprofile, the instantpowerthat issaved intheprofile isonlyhalfof theoriginal

power,plusitischangedtoanegativevalue.Whichmeansthattheenergyconsumption

atthenodeswheretheyareconnectedisgoingtobelower.


50

ThesourcecodeusedisshowninFigure34:

Figure34:ChangesinRearange_csvFunction.[Excel.py]

Thereare18differentprofilesareavailableandthetheftshavethesameloadprofiles

asthehouses.Thefunctionabovehastobecalledforeachhouseandforeachtheft,as

showninFigure35:

Figure35:HousesandTheftsLoadProfile.[Excel.py]

TherestoftheExcel.pyfileremainsunchanged.

6.1.3.-LVTest.py

Thisscriptisthemostimportantforthereconfiguration,becauseitiswhereMosaikand

simulatorsareinitialized.Themaindifficultyduringtheexperimentsistosetcorrectly

theexactnumberofhousesandtheftsthatmustbestudied.Inordertodoso,different

situationshavetobeanalyzed.

Forexample:

• Howitshouldworkwhentheusersetsnoloads?


51

• How should itworkwhen it sets less loads than the amount of profiles that

exists?

• Howtosettheloadswhenhavingabiggernumberthantheamountofprofiles

thatexists?

Thisfunctiondevelopedtosolvethesequestionsisexecutedwiththesourcecodefrom

Figure36:

Figure36:CreatingVariableNumberofLoadsandThefts.[LVTest.py]

Thesolutiontocreatemorehousesthanprofilesistocreateahighernumberofloads

withthesameprofileandthenconnectthemindifferentplaces.Thissolutionapplies

exactlythesamewaytothethefts.However,insomesituationsitisdifficulttocreate

exactly the number of loads the user defines. So two “if” statements have been

developedwhencreatinganumberofloadswiththesameprofile.


52

Inthefirststatement,thenumberofloadsissimplytheentirepartofthedivisionofthe

numberofhousesdividedbythenumberofprofilesplusone.Ifatsomestagecreating

thisnumberofloadswouldoverpassthenumberofdesiredloads,lessloadsshouldbe

created.Originallytheideawastocreatethemonebyone.Butforbiggerscenarios,it

isneededtocreatemorethanonebutlessthaninthecalculationbefore.Thenumber

ofloadscreatedisthenumberofremainingloadstobecreateddividedbythenumber

ofprofilesnotusedsofar.

Nowtheonlythingmissingisconnectingthehousesandtheloads.Asintheprojectonly

mechanicaltamperingofthesmartmetersisconsideredasanenergytheftpossibility,

thetheftsareonlyconnectedinthesamenodesastheloads;therefore,itisnotpossible

tocreatemoretheftsthanloads.Inordertohavethetheftsasevenaspossible,they

areconnectedatevendistancesbetweeneachother.

ThisisdonewiththesourcecodefromFigure37:

Figure37:ConnectingLoadsandThefts.[LVTest.py]

Inthisscript,oneofthesimulatorsthatisinitializedisthewebvisualizationsimulator.

Forexperimentationpurposes,theftshavebeenintroducedinthevisualizationasanew

kind of node. This way it is possible to graphically check that the thefts have been

created.


53

Therefore, this nodewas introduced in theweb visualization andnow thewebpage

lookslikeinFigure38:

Figure38:WebVisualizationChanges.

Asitisdifficulttoseethethefts,azoomfigurehasbeentakenaswellinFigure39:

Figure39:ZoomedWebVisualization.


54

6.1.4.-Output.py

Theoutput.py script isusedtoeasetheanalysisof theexperiments results.Thetest

resultsandthetheoreticalresultshavebeencomparedusingExcel;sotheinformation

fromthetheoreticalpowerflowandtheactualpowerflowcanbefoundintheexcel

file;introducedthroughpythonandafterwardsitautomaticallyupdatesthevaluesand

givestheuserthedifferenceinpowerconsumption.Inordertodoso,thescriptread

thedatabase(hdf5file),andthenintroduceallthedataintoExcel.

ThesourcecodeusedforwhathasbeenexplainedsofarisshowninFigure40:

Figure40:Hdf5ReaderandWriter.[Output.py]

NowthatallthedataregardingthesubstationlevelhasbeenintroducedintotheExcel

file,thedataregardingtheLinesisintroducedaswell.AverysimilarfunctiontoFigure

40 isdeveloped;withthedifferencethat theonebeforeonlyconsideredthe instant

powermeasuredeveryminuteandthisonegivesonlytheenergythathasgonethrough

thelineduringtheday.Thesourcecodetointroducetheinformationfromthepower

linesispresentedinFigure41:

Figure41:LinesReaderandWriter.[Output.py]


55

ThelastpieceofinformationthathastobesavedintoanExcelfileistheonerelatedto

thenodes.Itisneededforeachoftheloadstodefine,whereitislocated,whichhouse

profilewastheoneusedtocreatetheloadandtheenergyconsumptionfromtheday.

ThesourcecodeofthefunctionisinFigure42:

Figure42:NodesReaderandWriter.[Output.py]

AverysimilarfunctionfromFigure42isdevelopedfortheexperimentationpurpose.

ThisfunctionshowsthelocationofeachofthetheftsandstoresthenodeintoanExcel

file.

ThisfunctionisseeninFigure43:

Figure43:TheftsReaderandWriter.[Output.py]

Theoutputofthesimulationissavedinto2differentExcelfiles;onecalledTheory.xlsx

andonecalledLVTest.xlsx.


56

6.1.5.-NN.py

TocreatetheNeuralNetworkanewscriptcalledNN.pyhascreatedandanewlibrary

wasimportedcalledneurolab.

Itwas decided to create aNeural Networkwith the information from awhole year

(2013),validateitwiththeinformationofanothercompleteyear(2014)andthenapply

ittoadayofanewyear(1stofJanuary2015).

Alltheloaddataistakenfrompecanstreet;however,theformatisdifferentfromthe

expectedbytheNeuralNetworktrainingfunction.Alltheinformationhastoberead

andstoredcorrectlyintoamatrixsothatitisreadyforprocessing.Thisisdonewiththe

sourcecodefromFigure44,togetall the informationfromtheweather.Exactlythe

sameisdonetogetalltheinformationfromtheenergyconsumption.

Figure44:DataReadingFunction.[NN.py]

Nowthatallthedataisstoredintodifferentvariables,inordertoeasethework,ashort

function is written to save this information correctly formatted into csv files. This

functionismeanttoberunonlyonce,firstbecausethetimeneededtoexecuteover18

housesisquitelongandsecondlybecauseitisnotnecessarytoreadthefilesanymore

iftheyarestoredseparately.


57

The source code that goes through each of the houses and store the information

correctlyformattedintocsvfilesisseeninFigure45:

Figure45:NeuralNetworkDataPreparation.[NN.py]

Once all the information is correctly formatted and can be easily read, the Neural

Networkcanbeconstructed.TheNeuralNetworkinputsare:themonth,thedayand

theoutsidetemperature.Andtheoutputistheenergyconsumptionoftheday.

Thefact isthattheenergy isahugevaluecomparedwiththeday,themonthorthe

temperature.Therefore,inordertomaketheNeuralNetworktraining,itisnecessary

tonormalizethedatabeforeitisusedtotraintheneuralnetwork.

Thenormalization isperformedwiththesourcecodefromFigure46;this function is

abletobothnormalizeandunnormalize.

Figure46:NormalizeFunction.[NN.py]


58

The normalized data from 2013 is used to train the Neural Network. Validation is

performed with another set of data from 2014. The Neural Network has been

introduced ina for loop,so that itcreatesaNeuralNetwork foreachkindofhouse,

calculatestheerrorforitandtheexpectedenergyconsumptiononthedaytostudy.A

separateNeuralNetwork is trainedforeachhouseprofile.Theerror foreachNeural

Networkandthepredictionvaluesforeachhouseprofilehasbeenstoredforfurther

analysis.AlloftheaboveisexecutedwiththelistingfromFigure47:

Figure47:NeuralNetworkPrototype.[NN.py]

It isused20neuronsper layerand3hiddenlayers;thereasonforsuchanumberof

layersisfortheprogramtobeabletoreachthegoalof1%errorinthetrainingofthe

NeuralNetwork;theprogramtakesalmost1hourtorun.Unfortunately,thissetupover

fitstheNeuralNetwork,theerrorcalculatedwiththevalidationsethasabigvalueinall

thehouseprofiles.AnotherNeuralNetworkisdevelopedwithonly2hiddenlayersand

with3and6neuronsrespectively.ThisNeuralNetworkdoesnotreachthegoalinthe

trainingphaseoftheNeuralNetwork;however,theerrorsmadewiththevalidationset

arevisiblylower.ThisnewNeuralnetworkisshowninFigure48:

Figure48:NeuralNetworkDeveloped.[NN.py]


59

Forthepurposeofthisinvestigation,theNeuralNetworkfromFigure48istheonethat

isused.

All the information from the Neural Network, which are house id, percentage of

validation in the Neural Network and expected energy consumption for the day is

storageintoanexcelfilecalledNN.xlsx.

6.1.6.-Comparator.xlsx

AnewExcelfile,namedComparator.xlsxiswherethedatafromalltheExcelfilescreated

throughthesimulationareloaded(LVTheory.xlsx,LVTest.xlsxandNN.xlsx).

InthisnewExcelfile,theusermaychoosefreelyatwhichpercentagedifferenceit is

consideredthatthereissomeonestealingenergybothinthepowerbalanceandinthe

NeuralNetwork;theseareδ1andδ2respectively.

TheoverviewisshowninFigure49:

Figure49:SubstationComparator.[Comparator.xlsx]

Ifenergytheftisdetected,thefollowingsubmethodwilloperateandlookforatheftin

thedifferentpowersystemssubsystemswiththesameδ1asbefore.Theoverviewis

seeninFigure50:

Figure50:SubsystemsComparator.[Comparator.xlsx]


60

Finally, it is needed to check whether there has been a change in the energy

consumption of the houses compared to previous years; this is done with the

informationfromtheNeuralNetwork,thischeckisonlydoneifthehouseislocatedin

anenergytheftsubsystem.

TheresultcanbeseeninFigure51:

Figure51:NNComparator.[Comparator.xlsx]

For the purposeof the investigation, also the reality ofwhere the thefts havebeen

locatedisseenalsoinFigure51.Thiswaythecalculationoftheprecision,accuracyand

recallofthetoolcanbedonefaster.

6.2.-ExperimentsScenarios

Nowaseriesofexperimentsareexecutedtotesttheproposedenergytheftmethod.In

thismasterthesis4mainscenariosaredescribed;whichare0thefts,5thefts,10thefts

and15theftsinatotalof75houses.

6.2.1.-Experiment0Thefts

Inthisscenario,noadditionaltheftsareintroducedinthenetwork.Thisscenarioisa

basecase;itisusedasacomparatortootherscenarios.


Inthisexperiment,5theftsareintroducedevenlyinthegrid.Thetotalamountofenergy

thatisstoleninthescenarioisaround40kWh.


61

TheoverviewoftheresultsintheexperimentcanbeseeninFigure52:

Figure52:Experiment5Thefts.[Comparator.xlsx]


In this experiment, 10 thefts are introduced evenly in the grid. The total amount of

energythatisstoleninthescenarioisaround120kWh.




62


In this experiment, 15 thefts are introduced evenly in the grid. The total amount of

energythatisstoleninthescenarioisaround170kWh.



6.3.-ExperimentsResults

Theresultsfromtheexperimentsareexpressedintheoverviewjustbelow.Thereare3

differentlevelsofidentification.Oneintheentiregrid,oneinthesubsystemandthen

thelastoneatthenodelevel.Thefirsttwolevelsaredonebasedonpowerbalance.The

lastlevelisdonebasedontheNeuralNetwork.Basingthetruthofthefindingsonlyin

thelastlevelisnotaccuratetodeterminethestatisticalresultsofthetool.Inorderto

present the accuracy, the precision and the recall of the tool a confusionmatrix is

developed for each of the levels. Precision defines the variability of the tool, while

accuracydescribestheerrorobserved.Recallisavariablerelatedwiththesensitivityof

thetool.


63

6.3.1.-NetworkLevel

TheconfusionmatrixisshowninTable1:

Experiment TruePositive FalsePositive TrueNegative FalseNegative

0Thefts 0 0 1 0

5Thefts 1 0 0 0

10Thefts 1 0 0 0

15Thefts 1 0 0 0

Total 3 0 1 0

Table1:ConfusionMatrixGridLevel.

Withthefiguresfromthetable,itispossibletocalculatetheprecisionandtheaccuracy

ofthetool.

Theaccuracyiscalculatedwiththefollowingformula:

!""#$%"& = ($#)*+,-.-/) + ($#)1)2%.-/)*+,-.-/) + 1)2%.-/) = 4

4 = 100%

Theprecisioniscalculatedwiththefollowingformula:

*$)"-,-+7 = ($#)*+,-.-/)($#)*+,-.-/) + 8%9,)*+,-.-/) =

33 = 100%

Therecalliscalculatedwiththefollowingformula:

;)"%99 = ($#)*+,-.-/)($#)*+,-.-/) + 8%9,)1)2%.-/) =

33 = 100%

6.3.2.-SubsystemLevel



0Thefts 0 0 7 0

5Thefts 4 0 3 0

10Thefts 5 0 2 0

15Thefts 6 0 1 0

Total 15 0 13 0

Table2:ConfusionMatrixSubsystemsLevel


64


ofthetool.


!""#$%"& = ($#)*+,-.-/) + ($#)1)2%.-/)*+,-.-/) + 1)2%.-/) = 28

28 = 100%


*$)"-,-+7 = ($#)*+,-.-/)($#)*+,-.-/) + 8%9,)*+,-.-/) =

1515 = 100%


;)"%99 = ($#)*+,-.-/)($#)*+,-.-/) + 8%9,)1)2%.-/) =

1515 = 100%

6.3.3.-NeuralNetworkLevel



0Thefts 0 0 75 0

5Thefts 3 35 35 2

10Thefts 7 34 31 3

15Thefts 9 32 28 6

Total 19 101 169 11

Table3:ConfusionMatrixNeuralNetworkLevel.


ofthetool.


!""#$%"& = ($#)*+,-.-/) + ($#)1)2%.-/)*+,-.-/) + 1)2%.-/) = 19 + 169

19 + 101 + 169 + 11 = 62,7%


*$)"-,-+7 = ($#)*+,-.-/)($#)*+,-.-/) + 8%9,)*+,-.-/) =

1919 + 101 = 15,8%


;)"%99 = ($#)*+,-.-/)($#)*+,-.-/) + 8%9,)1)2%.-/) =

1919 + 11 = 63,3%


65

6.4.-ExperimentConclusions

Fromtheresultsoftheexperiments,itcanbedeterminedthatthepowerbalancestep

inthetoolisworkingwithanexcellentprecision,accuracyandrecall.Thispartofthe

toolisabletoidentifycorrectlythesituationswhereenergytheftoccurs

RegardingtheNeuralNetwork level, itcanbeseenthatthetool isnotquiteprecise,

whileitdoesbetterwiththeaccuracyandrecall.Themainreasonforthisbadprecision

isthattheNeuralNetworkprovidesalargenumberoffalsepositives.Thisisbecause

theNeuralNetworkhasnotbeencompletelyoptimized.Thedatathathasbeenselected

for it to be trained and afterwards tested has not been appropriate enough.More

knowledge about the dataset, aswell asmore inputs that are related to the power

consumptionofthehouseswouldhaveleadintoamoreprecisetool.


66

7.-ConclusionIn this last chapter of the master thesis; first an overview of the project is done.

Afterwards,somerecommendationsonhowtousethetoolandfurtherimprovement

thatcouldbemade.Finally,thedifferentgoalsthatweredescribedatthebeginningof

theprojectareenumeratedanddescribedwhomuchtheyhavebeenfulfilled.

Atthebeginningofthismasterthesis, itwasdecidedtodevelopaco-simulationtool

thatcouldbeabletodealwithlargesmartgridsscenarios.Theco-simulationmanager

chosenfortheapplicationwasMosaik.Mosaikcamewithademoofhowitworks,and

usingitasthestartingscenario,ithasbeenimprovedtoachievetherequirementsof

theproject.Once theco-simulation toolwasoptimized.Thechoiceofan interesting

application was another step of this investigation. We have considered different

applications, and in the end energy theft was the most relevant, intriguing and

challenging.Severaldetectionmethodsforenergythefthavebeenanalyzed.Thechosen

methodconsistedintwosteps,thefirstusingenergybalancetodiscoverenergytheft

regionsandthesecondstepusingartificialneuralnetworktopredictthedailyenergy

consumption of individual houses and compare it to the actual consumption. The

designedtoolwaseasily reconfigured toperformaseriesofexperiments to test the

proposedenergytheftdetectionmethod.Theresultsoftheexperimentshowthatthe

first stepof the toolworkedverygood,while the secondstepneededsome further

improvements.

Regardingtherecommendationsandimprovementsthatcouldbedevelopedinthetool.

ItseemsobviousthattheNeuralNetworkdidnotworkasexpected.Itcouldbethatthe

modeloftheNeuralNetworkwaswrong,thatthedatasetsselectedasinputswerenot

corrector itcouldsimplybethattheNeuralNetworkisnotthebestwaytoforecast

energy consumption. It is recommended to further study the datasets that are

introducedintheNeuralNetworkmodelandmaybetrywithotherkindofmodelsof

simulators to forecast energy consumption. However, the co-simulation setup

developedinthisprojectusingMosaikwiththeEuropeanLowVoltageTestFeeder(in

PyPower)wasveryeasytouseandreconfigure.


67

Theexperimentresultswereeasilyaccessibleandtheseriesofexperimentswereeasy

to set up and realistic. The tool aided the development of a smart gridmethod for

detectingenergy theft indistributiongrid,butcouldbeeasilyusedwithmanyother

applications.

Thegoalsdescribedintheintroductionare:

§ G1:ModifyMosaikexampledemocodeinordertofulfillrequirements(R1,R2

andR3).[Achievedinchapter3]

o This represents one of the biggest part of the project. It was first

necessarytolearntoprogramwithpythonlanguageandthentodesign

andimplementthesoftwaretofulfilltherequirements.

§ G2: Validate the obtained results against the results provided in the

documentation.[Achievedinchapter4]

o Thishasbeenagreatsuccess.Itcanbeseenthatthedifferencebetween

the given results and the results from the simulation is almost

nonexistent.Thismeansthatthetoolwascorrectlyconfigured.

§ G3: Select a smart grid scenario to be implemented in the tool. [Achieved in

chapter2]

o It was selected the scenario given by the European Low Voltage Test

Feeder.Butasthetoolwasdesigntoautomaticallycreateascenariowith

thefilesinthesameformatasinthefeeder,anyotherscenariocanbe

easilytested.

§ G4:Developanenergytheftdiscoverymethod.[Achievedinchapter5]

o The proposed method to discover energy theft is not complete and

sophisticated,butproventobeusefulfordisseminatingtheoperationof

thedevelopedco-simulationsetup.Theuncommonsmartgridscenario

wasachallengingandinterestingapplication.

§ G5:Preparethetooltoexperimentwithit.[Achievedinchapter6.1]

o Anewscripthadtobecreated.Anexcelfileisusedasthewaytoseeand

interpret the results. Most of the existing scripts had some

reconfiguration.


68

§ G6:Performseveraltestcasesontheselectedscenarioandgathertestresults.


o Once the tool to experiment has been correctly configured; testing

differentscenariosisjustamatteroftime.

§ G7:Interpretthetestresultsanddrawconclusions.[Achievedinchapter6.3]

o The power balance of the tool worked perfectly, while the Neural

Networkstephadsomedifficultieswiththefalsepositives.

§ G8: Provide recommendations for usewith the developed tool. [Achieved in

chapter7]

o The developed tool is really powerful; however, the Neural Network

modeldidnotworkasgoodasexpected.WithamoreaccurateNeural

Networkthistoolwouldhaveworkedmuchbetter.

§ G9:Publishthedevelopedsoftwareinasharedrepository.[Achievedinchapter

7]

o A Git repository was created to store the software developed in this

project.Git isaversioncontrolsystemusedforsoftwaredevelopment

processinorderto:trackprogress,saveversionsofthesoftware,allow

softwaredevelopedcollaboration,annotateauthorsandchangesmade

inthesoftware.ThefullsoftwarecodecanbedownloadedformaDTU

Elektro sharedgit repositoryhttps://git.elektro.dtu.dk/salvage/mosaik-

ELVTF.gitbyauthorizedusers.

Inconclusion,allthegoalsthatweredefinedintheintroductionoftheprojecthave

beenfulfilled.


69

8.-References[1]Horizon2020,Call:H2020-INFRAIA-2014-2015Topic:INFRAIA-1-2014-2015Typeof

action: RIA Proposal number: 654113, Proposal acronym: ERIGrid; Proposal-ERIGrid-

JRA2.pdf

[2] European Low Voltage Test Feeder. Retrieved from :

http://ewh.ieee.org/soc/pes/dsacom/testfeeders/index.html

[3]J.Fitzgeraldetal.(eds.),CollaborativeDesignforEmbeddedSystemsChapter2,Co-

modelling and Co-simulation in Embedded Systems Design,Springer-Verlag Berlin

Heidelberg2014,DOI10.1007/978-3-642-54118-6__2,2014

[4] Basic Concepts on SymPy; Retrieved from:

https://simpy.readthedocs.io/en/latest/simpy_intro/basic_concepts.html

[5] Onisick, Joe, Private Cloud Automation, Orchestration, And Measured Service,

NetworkComputing,June23,2011.

[6]Whatispython?;Retrievedfrom:https://www.python.org/doc/essays/blurb/

[7]Czechowski,R.andKosek,A.M.TheMostFrequentEnergyTheftTechniquesand

HazardsinPresentPowerEnergyConsumption.2016JointWorkshoponCyber-Physical

SecurityandResilienceinSmartGrids(CPSR-SG2016),2016

[8]Salinas,S.,Li,M.,andLi,P..Privacy-PreservingEnergyTheftDetectioninSmartGrids.

20129thAnnualIEEECommunicationsSocietyConferenceonSensor,MeshandAdHoc

Communications and Networks (SECON),DOI 978-1-4673-1905-8/12/$31.00 ©2012

IEEE,2012


70

[9]Jiang,R.,Lu,R.,Wang,Y.,Luo,J.,Shen,C.,andShen,X..Energy-TheftDetectionIssues

for Advanced Metering Infrastructure in Smart Grid. TSINGHUA SCIENCE AND

TECHNOLOGYISSNll1007-0214ll01/12llpp105-120Volume19,Number2,April2014

[10]McLaughlin,S.,Holbert,B.,Fawaz,A.,Berthier,R.,andZonouz,S..AMulti-Sensor

Energy Theft Detection Framework for Advanced Metering Infrastructures. IEEE

JOURNALONSELECTEDAREASINCOMMUNICATIONS,VOL.31,NO.7,JULY2013,2013

[11]Kadurek,P.,Studentmember,IEEE,J.Blom,J.F.G.Cobben,W.L.Kling,Member,

IEEE1. Theft detection and smart metering practices and expectations in the

Netherlands

[12] Gurney, K. (1997) An Introduction to Neural Networks London: Routledge. UCL

PressLimited,London.ISBN0-203-45151-1.

[13] Pecanstreet Dataport. Retrieved from:

https://dataport.pecanstreet.org/data/interactive

[14] Neural Network Image. Retrieved from: http://cse22-

iiith.vlabs.ac.in/exp4/images/structure.png


71

9.-AppendixIn the appendixes it is shown all the code that has been used for the different

simulations.

9.1.-Var.py# Assuming the information of the feeder csv are always in the same format # Variables to change # Values print('Reading Values') path = 'LVData' nodo = ['Busname'] linea = ['Name', 'Bus1', 'Bus2', 'LineCode', 'Length'] impedancias = ['Name', 'R1', 'X1', 'C1', 'R0', 'X0', 'C0'] transformador = ['Name', 'bus1', 'bus2', 'kV_pri', 'kV_sec', 'MVA', '%XHL', '%resistance'] Tap = 0 # CSV carga = ['Name', 'Bus', 'Yearly'] forma = ['Name', 'File'] # Constants START = '2015-01-01 00:00:00' END = 1 * 24 * 3600 # 1 Day GRID_FILE = 'Grid.xlsx' # Layout houses = 75 thefts = 5 print('Values Read')

9.2.-ExcelValidation.py# Assuming the information of the feeder csv are always in the same format # Initialize import csv import numpy import xlsxwriter from Var import nodo, linea, impedancias, transformador, carga, forma, START, END, GRID_FILE, Tap, path # Constants START = START END = END REF = 'REF' PQ = 'PQ' Online = 1 Line0 = 'Slack Line' taps = '{-1: 0.9, 0: 1.0, 1: 1.1}' #################################################################################################### # Pre-treatment of the data # Assuming the information of the feeder csv are always in the same format # Functions # Reads csv and storage values as asked # csv_name the name of the csv file with path # columns name of the variable that has the name of the headers in the correct order # rep number of rows that are not useful in the csv file def column_extraction(csv_name, columns, rep): a = csv.reader(open(csv_name, "r"), delimiter=',') b = numpy.array(list(a))


72

output = numpy.array(b) for r in range(0, rep): b = numpy.delete(b, 0, 0) for j in range(0, len(columns)): for i in range(0, len(b[0])): if b[0, i] == columns[j]: if j == 0: output = numpy.array(b) for x in range(0, i): output = numpy.delete(output, 0, 1) for x in range(i, len(b[0])-1): output = numpy.delete(output, 1, 1) else: temp = numpy.array(b) for x in range(0, i): temp = numpy.delete(temp, 0, 1) for x in range(i, len(b[0])-1): temp = numpy.delete(temp, 1, 1) output = numpy.hstack((output, temp)) output = numpy.delete(output, 0, 0) return output # Inserts values in Excel # var variable to insert column per column # col column to begin inserting # row row to begin inserting # kind matrix or header # numero value to insert many times # flt first column in the excel file that will try to change to float def exc_insert(var, col, row, kind, numero, ftl): if kind == 'One': if col >= ftl: try: worksheet.write(row, col, float(var)) except ValueError: worksheet.write(row, col, var) else: worksheet.write(row, col, var) if kind == 'Matrix': for i in range(0, len(var[0])): for j in range(0, len(var)): if i >= ftl: try: worksheet.write(row+j, col+i, float(var[j, i])) except ValueError: worksheet.write(row+j, col+i, var[j, i]) else: worksheet.write(row+j, col+i, var[j, i]) if kind == 'Row': for i in range(0, len(var)): if i >= ftl: try: worksheet.write(row, i, float(var[i])) except ValueError: worksheet.write(row, i, var[i]) else: worksheet.write(row, i, var[i]) if kind == 'Header': bold = workbook.add_format({'bold': True}) for i in range(0, len(var)): worksheet.write(row, i, var[i], bold) if kind == 'Multiple': for i in range(0, len(var)): if col >= ftl: try: worksheet.write(row+i, col, float(numero)) except ValueError: worksheet.write(row+i, col, numero) else: worksheet.write(row+i, col, numero) return # Substitute if in a column there is the same value as in the same column in the other variable (only2x2) # control Variable that will be controlled # check Variable where it will be checked if control exists


73

def checkandcopy(control, check): one = control for i in range(0, len(control)): for j in range(0, len(check)): if control[i, 2] == check[j, 0]: one[i, 2] = check[j, 1] return one # Rearange CSV to the way CSV simulator needs them # name Name of the csv file with path # arch Name of the csv file created with path def rearange_csv(name, arch, kind, start): a = csv.reader(open(name, "r"), delimiter=',') b = numpy.array(list(a)) init = numpy.array([['00000000000000000000000000000000000000000'] * len(b[0]) for _ in range(len(b))]) c = numpy.delete(b, 0, 0) for j in range(0, len(c)): init[j, 0] = start + ' ' + c[j, 0] init[j, 1] = float(c[j, 1])*1000 end = init file = open(arch, "w") w = csv.writer(file, delimiter=',') w.writerow([kind]) w.writerow(["Date", "P # [W]"]) w.writerow(["2016-01-01 00:00:00", "0"]) for i in range(0, len(end)): if i < len(c)-1: w.writerow(end[i]) w.writerow(["2016-01-02 00:00:00", "0"]) file.close() return # CSV print('Starting to Treat the Data') bus = column_extraction("%s/Buscoords.csv" % path, nodo, 1) branch = column_extraction("%s/Lines.csv" % path, linea, 1) linetypes = column_extraction("%s/LineCodes.csv" % path, impedancias, 1) trafotypes = column_extraction("%s/Transformer.csv" % path, transformador, 1) TYPE = trafotypes[0, 0] PRI = trafotypes[0, 1] SEC = trafotypes[0, 2] KV_Prim = trafotypes[0, 3] KV_Sec = trafotypes[0, 4] MVA = trafotypes[0, 5] Imaxp = float(MVA)*1000/1.73205080757/float(KV_Prim) Imaxs = float(MVA)*1000/1.73205080757/float(KV_Sec) IMAX = Imaxs X = float(KV_Sec)*float(KV_Sec)/float(MVA)*float(trafotypes[0, 6]) R = float(KV_Sec)*float(KV_Sec)/float(MVA)*float(trafotypes[0, 7]) Ploss = float(trafotypes[0, 7])*float(KV_Sec)*1000*float(trafotypes[0, 7])*float(KV_Sec)/float(R) slack = [PRI, REF, KV_Prim] slackline = [Line0, PRI, SEC, TYPE, 1, Online, Tap] trafo = [TYPE, MVA, Imaxp, Imaxs, Ploss, R, X, taps] for o in range(0, len(branch)): branch[o, 4] = float(branch[o, 4])/1000 for t in range(0, len(linetypes)): linetypes[t, 1] = (float(linetypes[t, 1])+float(linetypes[t, 4]))/3 linetypes[t, 2] = (float(linetypes[t, 2])+float(linetypes[t, 5]))/3 linetypes[t, 3] = (float(linetypes[t, 3])+float(linetypes[t, 6]))/3 linetypes = numpy.delete(linetypes, 4, 1) linetypes = numpy.delete(linetypes, 4, 1) linetypes = numpy.delete(linetypes, 4, 1) motor = column_extraction("%s/Loads.csv" % path, carga, 2) shapes = column_extraction("%s/LoadShapes.csv" % path, forma, 1) Loads = checkandcopy(motor, shapes) for y in range(1, 101): year = '2016-01-01' category = "Load" original = "%s/Load Profiles/Load_profile_%s.csv" % (path, y) upgraded = "%s/Load Profiles V2/Load_profile_%s.csv" % (path, y) rearange_csv(original, upgraded, category, year) # Excel header_Buses = ['Node Name', 'Node Type', 'Base Voltage [kV]', 'Notes']


74

header_Branches = ['Name', 'From', 'To', 'Type', 'Lenght [km]', 'Online', 'Tap'] header_LineTypes = ['Type Name', 'R[Ohm/km]', 'X[Ohm/km]', 'C[nF/km]', 'I_max[A]'] header_TrafoTypes = ['Trafo Type', 'S [MVA]', 'I_max_pri [A]', 'I_max_sec [A]', 'P_loss[kW]', 'R[Ohm]', 'X[Ohm]', 'taps'] workbook = xlsxwriter.Workbook(GRID_FILE) formato = workbook.add_format() formato.set_num_format('0.000') worksheet = workbook.add_worksheet('Nodes') exc_insert(bus, 0, 2, 'Matrix', 0, 2) exc_insert(header_Buses, 0, 0, 'Header', 0, 2) exc_insert(slack, 0, 1, 'Row', 0, 2) exc_insert(bus, 1, 2, 'Multiple', PQ, 2) exc_insert(bus, 2, 2, 'Multiple', KV_Sec, 2) worksheet.set_column('A:Z', 15, formato) worksheet = workbook.add_worksheet('Lines') exc_insert(branch, 0, 2, 'Matrix', 0, 4) exc_insert(header_Branches, 0, 0, 'Header', 0, 4) exc_insert(branch, 5, 2, 'Multiple', Online, 4) exc_insert(slackline, 0, 1, 'Row', 0, 4) worksheet.set_column('A:Z', 15, formato) worksheet = workbook.add_worksheet('Line types') exc_insert(linetypes, 0, 1, 'Matrix', 0, 1) exc_insert(header_LineTypes, 0, 0, 'Header', 0, 1) exc_insert(linetypes, 4, 1, 'Multiple', IMAX, 1) worksheet.set_column('A:Z', 15, formato) worksheet = workbook.add_worksheet('Transformer types') exc_insert(trafo, 0, 1, 'Row', 0, 1) exc_insert(header_TrafoTypes, 0, 0, 'Header', 0, 1) worksheet.set_column('A:Z', 15, formato) workbook.close() print('Data Treated')

9.3.-Excel.py# Assuming the information of the feeder csv are always in the same format # Initialize import csv import numpy import xlsxwriter from Var import nodo, linea, impedancias, transformador, START, END, GRID_FILE, Tap, path, thefts, houses # Constants START = START END = END REF = 'REF' PQ = 'PQ' Online = 1 Line0 = 'Slack Line' taps = '{-1: 0.9, 0: 1.0, 1: 1.1}' thefts = thefts houses = houses #################################################################################################### # Pre-treatment of the data # Assuming the information of the feeder csv are always in the same format # Functions # Reads csv and storage values as asked # csv_name the name of the csv file with path # columns name of the variable that has the name of the headers in the correct order # rep number of rows that are not useful in the csv file def column_extraction(csv_name, columns, rep): a = csv.reader(open(csv_name, "r"), delimiter=',') b = numpy.array(list(a)) output = numpy.array(b) for r in range(0, rep): b = numpy.delete(b, 0, 0) for j in range(0, len(columns)): for i in range(0, len(b[0])): if b[0, i] == columns[j]: if j == 0:


75

output = numpy.array(b) for x in range(0, i): output = numpy.delete(output, 0, 1) for x in range(i, len(b[0])-1): output = numpy.delete(output, 1, 1) else: temp = numpy.array(b) for x in range(0, i): temp = numpy.delete(temp, 0, 1) for x in range(i, len(b[0])-1): temp = numpy.delete(temp, 1, 1) output = numpy.hstack((output, temp)) output = numpy.delete(output, 0, 0) return output # Inserts values in Excel # var variable to insert column per column # col column to begin inserting # row row to begin inserting # kind matrix or header # numero value to insert many times # flt first column in the excel file that will try to change to float def exc_insert(var, col, row, kind, numero, ftl): if kind == 'One': if col >= ftl: try: worksheet.write(row, col, float(var)) except ValueError: worksheet.write(row, col, var) else: worksheet.write(row, col, var) if kind == 'Matrix': for i in range(0, len(var[0])): for j in range(0, len(var)): if i >= ftl: try: worksheet.write(row+j, col+i, float(var[j, i])) except ValueError: worksheet.write(row+j, col+i, var[j, i]) else: worksheet.write(row+j, col+i, var[j, i]) if kind == 'Row': for i in range(0, len(var)): if i >= ftl: try: worksheet.write(row, i, float(var[i])) except ValueError: worksheet.write(row, i, var[i]) else: worksheet.write(row, i, var[i]) if kind == 'Header': bold = workbook.add_format({'bold': True}) for i in range(0, len(var)): worksheet.write(row, i, var[i], bold) if kind == 'Multiple': for i in range(0, len(var)): if col >= ftl: try: worksheet.write(row+i, col, float(numero)) except ValueError: worksheet.write(row+i, col, numero) else: worksheet.write(row+i, col, numero) return # Rearange CSV to the way CSV simulator needs them # name Name of the csv file with path # arch Name of the csv file created with path def rearange_csv(tipo, name, arch, casa): a = csv.reader(open(name, "r"), delimiter=',') b = numpy.array(list(a)) init = [0, 0] end = [0, 0] if tipo == "Load": for j in range(0, len(b)): if float(b[j, 1]) == float(casa):


76

init[0] = b[j, 0][0:19] init[1] = float(b[j, 2])*1000 end = numpy.vstack((end, init)) elif tipo == "Theft": for j in range(0, len(b)): if float(b[j, 1]) == float(casa): init[0] = b[j, 0][0:19] init[1] = float(b[j, 2])*500*(-1) end = numpy.vstack((end, init)) else: print('Type not defined') end = numpy.delete(end, 0, 0) file = open(arch, "w") w = csv.writer(file, delimiter=',') w.writerow([tipo]) w.writerow(["Date", "P # [W]"]) for i in range(0, len(end)): if i < len(end): w.writerow(end[i]) file.close() return # CSV print('Starting to Treat the Data') bus = column_extraction("%s/Buscoords.csv" % path, nodo, 1) branch = column_extraction("%s/Lines.csv" % path, linea, 1) linetypes = column_extraction("%s/LineCodes.csv" % path, impedancias, 1) trafotypes = column_extraction("%s/Transformer.csv" % path, transformador, 1) TYPE = trafotypes[0, 0] PRI = trafotypes[0, 1] SEC = trafotypes[0, 2] KV_Prim = trafotypes[0, 3] KV_Sec = trafotypes[0, 4] MVA = trafotypes[0, 5] Imaxp = float(MVA)*1000/1.73205080757/float(KV_Prim) Imaxs = float(MVA)*1000/1.73205080757/float(KV_Sec) IMAX = Imaxs X = float(KV_Sec)*float(KV_Sec)/float(MVA)*float(trafotypes[0, 6]) R = float(KV_Sec)*float(KV_Sec)/float(MVA)*float(trafotypes[0, 7]) Ploss = float(trafotypes[0, 7])*float(KV_Sec)*1000*float(trafotypes[0, 7])*float(KV_Sec)/float(R) slack = [PRI, REF, KV_Prim] slackline = [Line0, PRI, SEC, TYPE, 1, Online, Tap] trafo = [TYPE, MVA, Imaxp, Imaxs, Ploss, R, X, taps] for o in range(0, len(branch)): branch[o, 4] = float(branch[o, 4])/1000 for t in range(0, len(linetypes)): linetypes[t, 1] = (float(linetypes[t, 1])+float(linetypes[t, 4]))/3 linetypes[t, 2] = (float(linetypes[t, 2])+float(linetypes[t, 5]))/3 linetypes[t, 3] = (float(linetypes[t, 3])+float(linetypes[t, 6]))/3 linetypes = numpy.delete(linetypes, 4, 1) linetypes = numpy.delete(linetypes, 4, 1) linetypes = numpy.delete(linetypes, 4, 1) names = ['26', '93', '94', '171', '187', '252', '370', '410', '545', '585', '624', '739', '744', '861', '871', '890', '898', '1103'] for y in range(len(names)): original = "%s/Profiles.csv" % path upgraded = "%s/Profiles/Load_profile_%s.csv" % (path, names[y]) theft = "%s/Profiles/Theft_profile_%s.csv" % (path, names[y]) rearange_csv("Load", original, upgraded, names[y]) rearange_csv("Theft", original, theft, names[y]) # Excel header_Buses = ['Node Name', 'Node Type', 'Base Voltage [kV]', 'Notes'] header_Branches = ['Name', 'From', 'To', 'Type', 'Lenght [km]', 'Online', 'Tap'] header_LineTypes = ['Type Name', 'R[Ohm/km]', 'X[Ohm/km]', 'C[nF/km]', 'I_max[A]'] header_TrafoTypes = ['Trafo Type', 'S [MVA]', 'I_max_pri [A]', 'I_max_sec [A]', 'P_loss[kW]', 'R[Ohm]', 'X[Ohm]', 'taps'] workbook = xlsxwriter.Workbook(GRID_FILE) formato = workbook.add_format() formato.set_num_format('0.000') worksheet = workbook.add_worksheet('Nodes') exc_insert(bus, 0, 2, 'Matrix', 0, 2) exc_insert(header_Buses, 0, 0, 'Header', 0, 2) exc_insert(slack, 0, 1, 'Row', 0, 2) exc_insert(bus, 1, 2, 'Multiple', PQ, 2) exc_insert(bus, 2, 2, 'Multiple', KV_Sec, 2) worksheet.set_column('A:Z', 15, formato)


77

worksheet = workbook.add_worksheet('Lines') exc_insert(branch, 0, 2, 'Matrix', 0, 4) exc_insert(header_Branches, 0, 0, 'Header', 0, 4) exc_insert(branch, 5, 2, 'Multiple', Online, 4) exc_insert(slackline, 0, 1, 'Row', 0, 4) worksheet.set_column('A:Z', 15, formato) worksheet = workbook.add_worksheet('Line types') exc_insert(linetypes, 0, 1, 'Matrix', 0, 1) exc_insert(header_LineTypes, 0, 0, 'Header', 0, 1) exc_insert(linetypes, 4, 1, 'Multiple', IMAX, 1) worksheet.set_column('A:Z', 15, formato) worksheet = workbook.add_worksheet('Transformer types') exc_insert(trafo, 0, 1, 'Row', 0, 1) exc_insert(header_TrafoTypes, 0, 0, 'Header', 0, 1) worksheet.set_column('A:Z', 15, formato) workbook.close() print('Data Treated')

9.4.-LVTest.py# Initialize import random from mosaik.util import connect_many_to_one import mosaik from Excel import names, thefts, houses, START, END, GRID_FILE import warnings warnings.filterwarnings("ignore") # Simulators sim_config = { 'CSV': { 'python': 'mosaik_csv:CSV', }, 'DB': { 'cmd': 'mosaik-hdf5 %(addr)s', }, 'PyPower': { 'python': 'mosaik_pypower.mosaik:PyPower', # 'cmd': 'mosaik-pypower %(addr)s', }, 'WebVis': { 'cmd': 'mosaik-web -s 0.0.0.0:8000 %(addr)s', }, } # Main def main(): random.seed() world = mosaik.World(sim_config) create_scenario(world) world.run(until=END) # Scenario def create_scenario(world): # Start simulators pypower = world.start('PyPower', 'PyP', step_size=60) # Instantiate models grid = pypower.Grid(gridfile=GRID_FILE).children # Create nodes load = [] theft = [] ladron = 0 casa = 0 for q in range(len(names)): prof = names[q] if casa < houses: hfile = 'LVdata/Profiles/Load_profile_%s.csv' % prof loadssim = world.start('CSV', '%s' % prof, sim_start=START, datafile=hfile) if houses > len(names) and casa+(houses//len(names)+1) < houses: hsimulator = loadssim.Load.create((houses//len(names))+1)


78

casa += (houses//len(names)+1) elif houses > len(names) and casa + len(names) - q - 1 < houses-1: hsimulator = loadssim.Load.create((houses-casa)//(len(names)-q)) casa += ((houses-casa)//(len(names)-q)) else: hsimulator = loadssim.Load.create(1) casa += 1 try: load = load + hsimulator except ValueError: load = hsimulator except UnboundLocalError: pass else: pass if ladron < thefts: tfile = 'LVdata/Profiles/Theft_profile_%s.csv' % prof theftsim = world.start('CSV', '%s' % prof, sim_start=START, datafile=tfile) if thefts > len(names) and ladron+(thefts//len(names)+1) < thefts: tsimulator = theftsim.Theft.create((thefts//len(names))+1) ladron += (thefts//len(names)+1) elif houses > len(names) and ladron + len(names) - q - 1 < thefts-1: tsimulator = theftsim.Theft.create((houses-ladron)//(len(names)-q)) ladron += ((houses-ladron)//(len(names)-q)) else: tsimulator = theftsim.Theft.create(1) ladron += 1 try: theft = theft + tsimulator except ValueError: theft = tsimulator except UnboundLocalError: pass else: pass # Connect entities buses = filter(lambda e: e.type == 'PQBus', grid) buses = {b.eid.split('-')[1]: b for b in buses} t = 0 for w in range(len(load)): number = int(len(buses)*w/len(load))+1 text = '%s' % number world.connect(load[w], buses[text], 'P') if thefts != 0: if w % (houses//thefts) == 0: try: world.connect(theft[t], buses[text], 'P') t += 1 except IndexError: pass else: pass else: pass # Database db = world.start('DB', 'DB', step_size=60, duration=END) hdf5 = db.Database(filename='LVTest.hdf5') connect_many_to_one(world, load, hdf5, 'P') nodes = [e for e in grid if e.type in 'RefBus, PQBus'] connect_many_to_one(world, nodes, hdf5, 'P', 'Q', 'Vl', 'Vm', 'Va') branches = [e for e in grid if e.type in ('Transformer', 'Branch')] connect_many_to_one(world, branches, hdf5, 'P_from', 'Q_from', 'P_to', 'P_from') # Web visualization webvis = world.start('WebVis', 'WebVis', start_date=START, step_size=60) webvis.set_config(ignore_types=['Topology', 'ResidentialLoads', 'Grid', 'Database']) vis_topo = webvis.Topology() connect_many_to_one(world, nodes, vis_topo, 'P', 'Vm') webvis.set_etypes({ 'RefBus': { 'cls': 'refbus', 'attr': 'P', 'unit': 'P [W]', 'default': 0, 'min': 0, 'max': 100000, }, 'PQBus': {


79

'cls': 'pqbus', 'attr': 'Vm', 'unit': 'U [V]', 'default': 400, 'min': 0.8 * 400, 'max': 1.2 * 400, }, }) connect_many_to_one(world, load, vis_topo, 'P') webvis.set_etypes({ 'Load': { 'cls': 'load', 'attr': 'P', 'unit': 'P [W]', 'default': 0, 'min': 0, 'max': 10000, }, }) connect_many_to_one(world, theft, vis_topo, 'P') webvis.set_etypes({ 'Theft': { 'cls': 'theft', 'attr': 'P', 'unit': 'P [W]', 'default': 0, 'min': 0, 'max': 10000, }, }) # Running if __name__ == '__main__': main()

9.5.-Output.pyfrom __future__ import print_function import numpy as np import h5py import matplotlib.pyplot as plt import datetime import os import xlsxwriter # Inserts values in Excel # var variable to insert column per column # col column to begin inserting # row row to begin inserting # formato format for the number introduced def exc_insert(var, col, row, formato): for i in range(0, len(var[0])): for j in range(0, len(var)): if i == 1: try: worksheet.write(row+j, col+i, float(var[j, i])) except ValueError: worksheet.write(row+j, col+i, var[j, i]) else: worksheet.write(row+j, col+i, var[j, i], formato) return # Open hdf5 and print graph # hdf5 name of database # path location database # title figure # fig_name name of the figure # variable kind of variable def print_output(hdf5, path, title, fig_name, variable): hora = workbook.add_format({'num_format': 'dd/mm/yyyy hh:mm'}) with h5py.File(hdf5, 'r') as hf:


80

datap = hf.get(path) power = np.array(datap) plt.plot(tiempo, power) plt.autoscale() plt.gcf().autofmt_xdate() plt.ylabel(variable) plt.xlabel('Time') plt.title(title) plt.savefig(fig_name) plt.close() tpower = tiempo.reshape(len(tiempo), 1) tpower = np.hstack((tpower, power.reshape(len(power), 1))) exc_insert(tpower, 0, 1, hora) return # Export energy through lines # sheet name # path location def lineas(sheet, path): work = workbook.add_worksheet(sheet) work.write(0, 0, 'Line') work.write(0, 1, 'Energy') with h5py.File(path, 'r') as hf: for n in range(1, 906): name = 'LINE%s' % n datap = hf.get('/Series/PyP-0.0-%s/P_from' % name) power = np.array(datap) energy = 0 for i in range(len(power)): energy += float(power[i])/1000/60 work.write(n, 0, name) work.write(n, 1, energy) return # Export loads location # sheet name # path location def loads(sheet, path): work = workbook.add_worksheet(sheet) work.write(0, 0, 'Node') work.write(0, 1, 'Profile') work.write(0, 2, 'Energy') with h5py.File(path, 'r') as hf: l = 1 for n in range(1, 906): node = '%s' % n relation = hf.get('/Relations/PyP-0.0-%s' % node) for i in range(len(relation)): profile = (np.array(relation)[i])[0].decode("utf-8") if 'Load' in profile: datap = hf.get('/Series/PyP-0.0-%s/P' % node) power = np.array(datap) energy = 0 for z in range(len(power)): energy += float(power[z])/1000/60 work.write(l, 0, node) work.write(l, 1, profile[11:50]) work.write(l, 2, energy) l += 1 return # Export thefts location # sheet name # path location def thefts(sheet, path): work = workbook.add_worksheet(sheet) work.write(0, 0, 'Node') with h5py.File(path, 'r') as hf: l = 1 for n in range(1, 906): node = '%s' % n relation = hf.get('/Relations/PyP-0.0-%s' % node) for i in range(len(relation)):


81

profile = (np.array(relation)[i])[0].decode("utf-8") if 'Theft' in profile: work.write(l, 0, node) l += 1 return run = os.system('source /Users/EzequielFredes/.virtualenvs/mosaik/bin/activate\n python LVTest.py\n') # run = 0 end = 0 tiempo = np.array([datetime.datetime(2016, 1, 1, i, j) for i in range(24) for j in range(60)]) print("Begin plotting output") # Theory workbook = xlsxwriter.Workbook('Comparator/Theory.xlsx') worksheet = workbook.add_worksheet('Theory') worksheet.write(0, 0, 'TimeStamp') worksheet.write(0, 1, 'Power') print_output('LVTheory.hdf5', '/Series/PyP-0.0-LINE1/P_from', 'Power Transformer Secondary', 'Print/PowerTransformerSecondaryTheory.png', 'Power [W]') lineas('Lines Theory', 'LVTheory.hdf5') workbook.close() # Reality workbook = xlsxwriter.Workbook('Comparator/Test.xlsx') worksheet = workbook.add_worksheet('Test') worksheet.write(0, 0, 'TimeStamp') worksheet.write(0, 1, 'Power') while end == 0: if run == 0: print_output('LVTest.hdf5', '/Series/PyP-0.0-LINE1/P_from', 'Power Transformer Secondary', 'Print/PowerTransformerSecondary.png', 'Power [W]') lineas('Lines Test', 'LVTest.hdf5') loads('Nodes Test', 'LVTest.hdf5') thefts('Thefts Test', 'LVTest.hdf5') end = 1 workbook.close() print("Output plotted")

9.6.-NN.py# Assuming the information of the feeder csv are always in the same format # Initialize import csv import numpy import copy import neurolab import xlsxwriter #################################################################################################### # Pre-treatment of the data # Assuming the information of the feeder csv are always in the same format # Functions # Reads csv and storage values # csv_name the name of the csv file with path def weather_reader(csv_name): a = csv.reader(open(csv_name, "r"), delimiter=',') b = numpy.array(list(a)) c = numpy.delete(b, 0, 0) d = numpy.delete(b, 0, 0) e = numpy.delete(b, 0, 0) f = numpy.delete(b, 0, 0) g = numpy.delete(b, 0, 0) h = numpy.delete(b, 0, 0) wyear = e[:, 0] wmonth = f[:, 0] wday = g[:, 0] whour = h[:, 0]


82

for i in range(len(d)): d[i, 1] = ((float(c[i, 1])-32)/1.8) for i in range(len(d)): wyear[i] = float(d[i, 0][0:4]) wmonth[i] = float(d[i, 0][5:7]) wday[i] = float(d[i, 0][8:10]) whour[i] = float(d[i, 0][11:13]) output = numpy.column_stack((wyear, wmonth, wday, whour, d[:, 1])) return output # Reads csv and storage values # csv_name the name of the csv file with path def house_reader(csv_name): a = csv.reader(open(csv_name, "r"), delimiter=',') b = numpy.array(list(a)) c = numpy.delete(b, 0, 0) d = numpy.delete(b, 0, 0) e = numpy.delete(b, 0, 0) f = numpy.delete(b, 0, 0) g = numpy.delete(b, 0, 0) h = numpy.delete(b, 0, 0) hyear = e[:, 0] hmonth = f[:, 0] hday = g[:, 0] hhour = h[:, 0] for i in range(len(d)): d[i, 2] = float(c[i, 2])*1000 for i in range(len(d)): hyear[i] = float(d[i, 0][0:4]) hmonth[i] = float(d[i, 0][5:7]) hday[i] = float(d[i, 0][8:10]) hhour[i] = float(d[i, 0][11:13]) output = numpy.column_stack((hyear, hmonth, hday, hhour, d[:, 1], d[:, 2])) return output # Neural data preparation def neuraldata(casa, clima, urba, dest): file = open(dest, "w") a = csv.writer(file, delimiter=',') a.writerow([casa]) a.writerow(["Year", "Month", "Day", "Temperature", "Energy"]) for i in range(2013, 2016): for j in range(1, 13): for k in range(1, 32): average = 0 energy = 0 for x in range(len(clima)): if float(clima[x, 0]) == i and float(clima[x, 1]) == j and float(clima[x, 2]) == k: average += float(clima[x, 4]) average /= 24 for z in range(len(urba)): if float(urba[z, 0]) == i and float(urba[z, 1]) == j and float(urba[z, 2]) == k and\ urba[z, 4] == casa: energy += float(urba[z, 5])/1000 vector = [i, j, k, average, energy] if energy == 0: pass else: a.writerow(vector) file.close() return # Creation Datasets def dataset(name, year): place = "LVData/NN/House%s.csv" % name a = csv.reader(open(place, "r"), delimiter=',') b = numpy.array(list(a)) c = numpy.delete(b, 0, 0) d = numpy.delete(c, 0, 0) e = numpy.array(list(d)) data = [0, 0, 0, 0, 0] for i in range(len(e)):


83

if float(e[i, 0]) == year: data = numpy.row_stack((data, [float(e[i, 0]), float(e[i, 1]), float(e[i, 2]), float(e[i, 3]), float(e[i, 4])])) data = numpy.delete(data, 0, 0) return data # Normalize data 1 dim vector # var Variable # do to normalize (0) or unnormalize (1) def normalize(var, do, come): normal = copy.deepcopy(var) original = copy.deepcopy(come) if do == 0: for i in range(len(var)): normal[i] = (var[i]-min(original))/(max(original)-min(original)) elif do == 1: for i in range(len(var)): normal[i] = var[i]*(max(original)-min(original))+min(original) else: print("Wrong Normalization") output = normal return output # CSV print('Neural Network Start') names = ['26', '93', '94', '171', '187', '252', '370', '410', '545', '585', '624', '739', '744', '861', '871', '890', '898', '1103'] # To be run only once estimated time for running is 4 hours weather = weather_reader("LVData/NNWeather.csv")[0::3] houses = house_reader("LVData/NNHouse.csv") for n in range(len(names)): path = "LVData/NN/House%s.csv" % names[n] neuraldata(names[n], weather, houses, path) # NN workbook = xlsxwriter.Workbook('Comparator/NN.xlsx') worksheet = workbook.add_worksheet('NN') worksheet.write(0, 0, 'House') worksheet.write(0, 1, 'Error in NN') worksheet.write(0, 2, 'Expected Consumption in day') for n in range(len(names)): # NN # Create train samples training = dataset(names[n], 2013) t_month = normalize(training[:, 1], 0, training[:, 1]).reshape(len(training), 1) t_day = normalize(training[:, 2], 0, training[:, 2]).reshape(len(training), 1) t_temperature = normalize(training[:, 3], 0, training[:, 3]).reshape(len(training), 1) t_inp = numpy.column_stack((t_month, t_day, t_temperature)) t_energy = normalize(training[:, 4], 0, training[:, 4]).reshape(len(training), 1) # Create validation samples validation = dataset(names[n], 2014) v_month = normalize(validation[:, 1], 0, training[:, 1]).reshape(len(validation), 1) v_day = normalize(validation[:, 2], 0, training[:, 2]).reshape(len(validation), 1) v_temperature = normalize(validation[:, 3], 0, training[:, 3]).reshape(len(validation), 1) v_inp = numpy.column_stack((v_month, v_day, v_temperature)) v_energy = validation[:, 4].reshape(len(validation), 1) # Create output samples ahora = dataset(names[n], 2015) a_month = normalize(ahora[:, 1], 0, training[:, 1]).reshape(len(ahora), 1) a_day = normalize(ahora[:, 2], 0, training[:, 2]).reshape(len(ahora), 1) a_temperature = normalize(ahora[:, 3], 0, training[:, 3]).reshape(len(ahora), 1) a_inp = numpy.column_stack((a_month, a_day, a_temperature)) # Create network with 3 inputs # 3 nurons for input layer, 6 neurons for hidden layer, 1 neuron for output # 3 layers including hidden layer and output layer


84

net = neurolab.net.newff([[0, 1], [0, 1], [0, 1]], [3, 6, 1]) # Train network error = net.train(t_inp, t_energy, epochs=2000, show=200, goal=0.01) # Simulate network out = normalize(net.sim(v_inp), 1, training[:, 4]) # Calculate error mistake = 0 percentage = 0 for p in range(len(out)): mistake += abs(out[p]-v_energy[p])/v_energy[p] # Output 01-01-2015 nn = normalize(net.sim(a_inp), 1, training[:, 4]) worksheet.write(n+1, 0, names[n]) worksheet.write(n+1, 1, mistake/len(v_energy)*100) worksheet.write(n+1, 2, nn[0]) workbook.close() print('Neural Network Done')

Documents

Investigation of Smart Grids scenarios by the use of co ... · The objective of the project is to understand the European low voltage test feeder in order to implement it into the