CIRED Workshop - Helsinki 14-15 June 2016

Paper 0120

Paper No 0120 Page 1 / 4

MMIINNIINNGG SSMMAARRTT MMEETTEERR DDAATTAA –– CCAASSEE FFIINNLLAANNDD

Antti MUTANEN Harri NISKA Pertti JÄRVENTAUSTA Tampere University of Technology University of Eastern Finland Tampere University of Technology

Finland Finland Finlandantti.mutanen@tut.fi harri.niska@uef.fi pertti.jarventausta@tut.fi

ABSTRACT

Smart meters collect a lot of data on customer levelelectricity consumption and this, together with other datasources e.g. environmental information and public opendata, provides an excellent basis for data mining. As a partof a recent smart grid project conducted in Finland,several different ways of mining smart meter data werestudied. The project brought advances in customerclassification and clustering, load profiling, spatial loadanalytics, behaviour change detection and loadforecasting.

INTRODUCTIONSmart metering is spreading quickly. Many distributionsystem operators (DSOs) in western countries haveinstalled, are installing, or are planning to install smartmeters. The drivers for smart meter implementation aremultiple. In some countries legal and regulatoryrequirements are driving the change as governments areenforcing the transition to liberated energy markets whereconsumers can freely choose their energy retailer. Smartmetering is also seen as an essential enabler for smart gridsthat can host large amounts of CO2 free distributedgeneration. They also enable new kind of dynamic tariffs thatsupport energy efficiency targets and operation of theelectricity markets. Sometimes, smart meter investments aremade on economic and financial grounds, with theassumption that on the long run smart meters and the newservices they enable, provide more revenue.

The primary role of smart meters is to provide fine-grainedinterval data on end customer energy consumption to theutility, but the cost of installing smart meters may not bejustified if the meters are used only for reading energyconsumption data. In addition to simple meter reading, otherfunctions with added value are needed to make the smartmeter investments profitable. Already today, severalpossibilities to utilize smart meters in electricity distributionand energy retail businesses exist, for example: remotedisconnection and reconnection of electricity supply,interruption and power quality statistics collection, alarmsbased on exceptional events (i.e. network faults and voltageviolations), web services for reporting customers electricityconsumption, and demand side management.

In addition to the above mentioned rather direct ways of

using smart meters, some added value can be extracted fromthe huge amount of historical electricity consumption datastored in meter data management (MDM) systems. Datamining can be used to extract useful information that isnormally hidden within this vast amount of measurements.

This paper reviews the Finnish smart metering backgroundand presents some smart meter data mining applicationsdeveloped in Finland during the recent years.

CASE FINLANDIn 2009, the Finnish Government passed a new act [1],which states that at least 80% of the customers of each DSOmust have a smart meter by 31 December 2013. The lawrequires that the installed smart meters are read at least oncea day, provide hourly energy measurements and that thesemeasurements are used in balance settlement. If requestedby the customer, the smart meter must have a standardisedinterface for real-time consumption monitoring. Otherrequirements include; registrations of long interruptions (˃3min), possibility to execute or relay load control commands,and storing the smart meter measurements at least six years.Also, the metering system must encourage customers to useelectricity in efficient and economical manner.

Another law [2], that has been passed to fulfil the EuropeanUnion energy efficiency directive, requires that all energyretailers must provide their customers, at least once a year,with a report that shows the energy consumption on thereporting period and on three previous years. This reportmust include comparison to other similar energy end usersand information how to improve the energy efficiency.

Although only 80 % smart meter penetration is required bylaw, in practice almost every customer (98 %) in Finland issupplied with a smart meter. This makes Finland anexcellent place for developing smart meter basedfunctionalities and electricity consumption data miningmethods. Between the years 2010 and 2015, several Finnishuniversities, research institutes, DSOs, and industry partnersco-operated in a 52 million euro Smart Grids & EnergyMarkets -program (SGEM). Smart metering was one of thesubtopics in SGEM and the research work done during thisproject resulted in several advances in the field of smartmeter data mining, some of which are presented in thispaper.

CIRED Workshop - Helsinki 14-15 June 2016

Paper 0120

Paper No 0120 Page 2 / 4

SMART METER DATA MININGWhen combined with external information sources,electricity consumption interval data provides useful insightsinto customer behaviour. The external information sourcescan be weather observations and forecasts (outdoortemperature, solar irradiance etc.), socio-economic data,building information, customer location, contractualinformation, or information collected directly from the endusers. With data mining methods the above mentionedinformation sources and smart meter interval data can becombined and useful information can be extracted.

Clustering and load profilingIn Finland, customer class load profiles are widely used indistribution network calculation and analysis. Distributionnetwork operation, operation planning, and networkplanning could all benefit from better customer classificationand load profiles.

During the SGEM project, we developed a load profilingmethod that clusters customers into similarly behavinggroups and calculates statistical load profiles [3]. In additionto cluster profiles, individual load profiles are calculated forthose large customers that exhibit unique consumptioncharacteristics. Seasonal temperature dependencies arecalculated for every cluster and individual profile and whencombined with outdoor temperature forecasts, the new loadprofiles can be used also for load forecasting [4].

The developed clustering procedure starts with customerwise temperature dependency calculation. Next, temperaturenormalization to long term monthly temperatures is done sothat measurements are comparable between different years.Pattern vectors describing the average hourly consumptionevery day of the week in a certain month are calculated.Analysis of variance (ANOVA) is used to determinewhether weekdays should be modelled separately or with acommon model. A two-stage clustering procedure utilizingweighted K-means clustering is performed. After the firststage, outliers (customers with the highest distances fromcluster centers) and candidates for individual load profiling(customers with the highest weighted distances from clustercenters) are separated from others. Weighting is doneaccording to yearly energies. After the second clusteringstage, cluster wise temperature dependencies are calculatedand load profiles are formed using the existing Finnish loadprofile format where each hour of the year has anexpectation value and a standard deviation. Fig. 1 shows thediversity of cluster center shapes typically found amongFinnish electricity consumers.

Customer behaviour change detectionSometimes customers change their electricity consumptionhabits, for example due to a change in the heating solution orpurchase of a micro generator or electric vehicle. Possibilityto detect these kind of abrupt changes makes load profiling

more dynamic and guarantees that the customerclassification and load profiles are always up-to-date. Oncea change is detected, the old pre-change measurementsshould be omitted and only the post-change measurementsshould be used in load profiling.

The newly developed change detection method [5] is basedon studying weekly segments of smart meter data. Changesare detected based on weekly energies and weekly profileshapes. In the shape based detection, all customers are firstclustered and then changes in the individual consumer’sweekly cluster memberships are compared with previousyear memberships. If the sum of differences is larger than apredefined detection threshold, a change is declared. Fig. 2shows an example. The upper subfigures show the hourlyelectricity consumptions during the years 2009 and 2010.The subfigures in the middle show the weekly clustermemberships. The sum of membership differences(subfigure in bottom right) confirms that there has been achange in the intra-week behaviour from week 30 onward,but the weekly energies (subfigure in bottom left) have notchanged.

Comparison curvesIn order to fulfil the requirements given in [2], the mostFinnish electricity retailers have provided their customers

Fig. 2. Change detection example.

Fig. 1. Example of clustering results (type weeks for october).

CIRED Workshop - Helsinki 14-15 June 2016

Paper 0120

Paper No 0120 Page 3 / 4

with access to an on-line portal where they can view theirelectricity consumption history and compare theirconsumption with other similar end users throughcomparison curves. The calculation of comparison curveshas its own challenges and to overcome these challenges anew comparison curve formation methods have beenpresented in [6] and [7].

The method proposed in [6] groups customers based on theavailable customer type information and then conductsinterval data based clustering to reduce the effect ofclassification errors occurred in the first grouping phase.The method proposed in [7] starts with a classification basedon the two most significant attributes; the type of housingand the heating system, and then continues with weighted K-means clustering. Clustering is applied to a set of vectorsthat contained information on daily load profiles and selectedcustomer background variables (in this case gross floor areaand the number of residents in different age groups).

In these studies, the information characterizing customers iscollected directly from customers either through the on-lineportal [6] or through a separate questionnaire [7]. Thisinformation could be extracted also from open databases(e.g. national building information database), but theinformation might not be up-to-date. Fig. 3 presents aprototype program where the customer can select theinformation used in the comparison curve calculation.Finally, the difference with the comparison group is showngraphically and in euro.

Load forecastingShort-term load forecasting is required by dynamicoptimization of control actions, electricity markettransactions, and distribution network operation. However,forecasting errors cause high balancing and network costs,thus cancelling the financial benefits from active demand.Extensive smart metering data can be utilized to improvethe accuracy of the existing models as well as developingnew enhanced modelling schemes for load forecasting. Inthis context, data mining techniques, including machine andstatistical learning, are of interest in enriching data-based

models, as well as hybrid models based on theircombinations with other techniques such as physically basedmodelling.

During the SGEM project, several different models utilizinghourly metered consumption data were evaluated for short-term load forecasting. The studied models were; a clusterprofile based predictor, a Kalman-filter based predictor withinput nonlinearities and physically based main structure, aneural network (NN) model [4], and a support vectormachine (SVM) model [8]. The NN and SVM models werethe most accurate, but also the other methods had theirrelative merits. The work is now continued in the Academyfunded RESPONSE project (2015–2018), where the aim isto develop a hybrid model, which combines the benefits ofphysical and purely data-driven modelling schemes inforecasting of loads and their load control responses.

Spatial data analytics and modellingCombining smart metering data with available geographicinformation will open totally new possibilities for dataanalytics in the planning and operation of future electricitynetworks and energy systems. In Finland, as well as in EU-level, different registries of the societies are opening inparallel with legislative frameworks (INSPIRE, PSI) thatdrive for the release and use of public sector data. Potentialdatasets include e.g. socioeconomic grid data, building andpopulation information, land-use data, meteorologicalobservations, and weather forecasts.

During the SGEM project, several GIS based data analyticsand modelling tool demonstrations were carried out bycombining smart metering data with other availablegeographic information [e.g. 9, 10]. Those experimentswere mainly focused on the use of socioeconomic grid data(Statistics Finland) and building information (PopulationRegister Centre) in spatial load modelling and what-if typeload analytics. Data mining techniques such as unsupervisedand supervised neural networks were used to modelinteractions between smart metering data and complexspatial information. Fig. 4 shows a screenshot from thedeveloped scenario based load prediction tool [10]. This

Fig. 4. Spatial load analytics tool based on smart metering dataand building information.

Fig. 3. Comparison curve with selectable customer typeattributes. Source [5]

CIRED Workshop - Helsinki 14-15 June 2016

Paper 0120

Paper No 0120 Page 4 / 4

tool can calculate, for example, how changes in heatingsolutions affect the regional peak loads and shows on themap where the load increases and where it decreases.

Future development needs and opportunitiesThe SGEM project provided many answers, but also newresearch questions arose. For example, the developedchange detection method is able to detect changes and pointout when the change happened, but it cannot specify whatcaused the change. There is a need to develop more specificmethods for detecting and identifying customers withphotovoltaic panels, heat pumps and electric vehicles.

In future, the utilization of demand response in a larger andmore dynamic scale requires development of new loadmodels that can forecast load responses accurately indifferent situations, can adapt to changes in load behaviourand are easy to maintain and update. In this context, smartmetering data and its enrichment using data miningtechniques play an important role and can enable data-driven model creation and updating.

The Finnish transmission network operator Fingrid has beenassigned a task of implementing a national data hub that willcentralize all information exchange in electricity markets.All smart meter measurements normally exchanged betweencustomers, DNOs and electricity retailers will be stored in asingle database. Expected to finish in 2019, the data hub willoffer unprecedented possibilities for smart meter datamining.

CONCLUSIONSThis paper presented smart meter data mining relatedstudies conducted during the SGEM-project. During thisproject, it was proven that clustering can improve theaccuracy of customer class load profiles and a methodologyfor smart meter data based clustering was developed. Inorder to make the load profiling more dynamic, a method fordetecting changes in customer behaviour was developed. Incomparison curve calculation, smart meter measurementswere combined with information characterizing consumersand consumption sites so that customers can compare theirelectricity consumption habits with other similar customers.Neural network and support vector machine basedforecasters were successfully applied to short term loadforecasting. Geographical grid data available in opendatabases was combined with smart meter data in order tocreate spatial load analyses.

To sum up, smart metering data and its mining together withopening public sector data will enable the development ofnew powerful data analytics and services for enablingreliable, and cost-efficient operation and planning of futureflexible energy systems as well as providing tools for

managing customer side energy efficiency and flexibility ofprosumer demand and generation.

REFERENCES

[1] REG 1.3.2009/66. Valtioneuvoston asetussähköntoimitusten selvityksestä ja mittauksesta(Finnish council of state act on electricity settlementand metering, in Finnish).

[2] REG 1211/2009. Laki energiamarkkinoilla toimivienyritysten energiatehokkuuspalveluista (Law on energyefficiency services for companies operating in energymarkets, in Finnish).

[3] A. Mutanen, P. Järventausta, M. Kärenlampi & P.Juuti, “Improving Distribution Network Analysis withNew AMR-based Load Profiles,” InternationalConference and Exhibition on Electricity Distribution(CIRED), Stockholm, Sweden, June 10–13, 2013.

[4] P. Koponen, A. Mutanen & H. Niska, “Assessment ofSome Methods for Short-Term Load Forecasting,”IEEE PES Innovative Smart Grid Technologies(ISGT) European Conference, Istanbul, Turkey, Oct.12–15, 2014.

[5] T. Chen, A. Mutanen, P. Järventausta & H. Koivisto,“Change detection of electricity Customer behaviorbased on AMR measurements,” IEEE PowerTechconference, Eindhoven, Netherlands, June 29 – July 2,2015.

[6] V. Välipirtti, “Development of comparison feedbackfor private customers in electricity consumptionreporting service,” M.Sc. Thesis, Tampere Universityof Technology, Tampere, Finland, 2014.

[7] M. Mononen, J. Saarenpää, M. Johansson & H. Niska,“Data-driven method for providing feedback tohouseholds on electricity consumption,” InternationalConference on Intelligent Sensors, Sensor Networksand Information Processing (ISSNIP) conference,Singapore, April 21–24, 2014.

[8] H. Niska, P. Koponen & A. Mutanen, ”Evolving smartmeter data driven model for short-term forecasting ofelectric loads,” IEEE Conference on IntelligentSensors, Sensor Networks and InformationProcessing (ISSNIP), Singapore, April 7–9, 2015.

[9] J. Saarenpää, M. Kolehmainen, M. Mononen & H.Niska, “A data mining approach for producing smallarea statistics-based load profiles for distributionnetwork planning,” IEEE ICIT conference, Seville,Spain, March 17–19, 2015.

[10] H. Niska, J. Saarenpää, T. Räsänen, J. Tiirikainen &M. Kolehmainen, “Scenario based load prediction toolfor distribution planning and management,”International Conference on Electricity Distribution(CIRED), Frankfurt, Germany, 6–9 June, 2011.