Smart Grids and Big Data

Integration and Governance Challenges and OpportunitiesFor Public Utilities

Smart Grid and Big Data

There are any number of vendors and publications stating that the IT departments of utilities need to invest in Big Data to manage Smart Grids.

Saying something does not make it true, even if you saying it very loudly and very often. It just makes it noisy.

Let's swap out marketing and hype for logic and math and separate the signal from the noise.

What is a Smart Grid?

First of all, what is a Smart Grid? It can mean different things to different people in a utility depending on their perspective:Customer

Distribution

Transmission

Generation

Regulatory

Smart Grid Customer Perspective

Smart metering means going from one meter reading per month to a reading every fifteen minutes, a 3,000-fold increase. For every one million meters, a utility can except to process 96 million reads per day.

Time-Of-Day and Time-Of-Use billing considerations are now in play

Meters can communicate to customers but that raises the question, who should be in charge of managing load based on this information?

Smart Grid Distribution Perspective

Advanced sensoring and switching devices at the distribution feeder level can make distribution system automation affordable

Selective load control

Managing distribution generation and even islanding

Smart Grid Transmission Perspective

Increase stability and control by combining phase measurement units (PMUs) and GPS with a Supervisory Control and Data Acquisition Unit (SCADA) at a central control facility

Flexible AC Transmission Systems (FACTS) are involved in the derivation of Interconnection Reliability Operating Limits (IROL) and require expensive monitoring devices, although they are still less costly than building new lines

Distributed and autonomous control could be used to eventually create a self-healing grid

Smart Grid Generation Perspective

The Unit Commitment Problem (UCP) refers to scheduling power generators (units) to meet electricity demand (load). Always complex and critical, the variability of wind farms requires different algorithms.

Additionally, the move from regulated to deregulated markets means moving to day-ahead and real-time markets. Day-ahead requires making a commitment based on a prior-day forecast while real-time requres adjusting the output per unit hourly with surplus/deficit units being traded on the Independent System Operator (ISO) market.

Smart Grid Regulatory Perspective

Smart Meters are considered (rightly or wrongly) to be a significant privacy and security risk and there will likely be a patchwork of federal and state regulations of varying technical feasibility.

As emerging sources of energy are developed, there will be additional regulations.

As new technologies are developed, there will be additional regulations.

Basically, there will be additional regulations.

Smart Grid IT Perspective

There are a lot of managers that will drive up the heat and intensity without providing much clarity.

There are a lot of vendors that have big ticket items that they say will fix our problems.

Let's step back and clearly define the problem so we can identify what form a solution would take before we start writing checks for vaporware.

In this next section, we'll come up with a clear problem definition and come up with an algorithmic approach to the problem. We should at the very least have a good idea of the Big-O of our proposed solution space.

Once we have a framework, we can more intelligently choose an implementation.

Smart Grid IT Perspective

Smart Grid IT Perspective

Smart Grid Requirements

Essentially, there has been one fundamental technical change: More devices are reporting more data more frequentlyWhat are these devices again?smart meters

sensors

syncophasors

What do we mean by reporting?There are just devices, so each one individually is really only capable of generating a text based log file. It could be fixed or variable length, xml or json but it will be text. Also, all of these devices now have an IP address so we will receive it on the network somehow. Basically, we will not be getting a hand-drawn picture on microfiche (don't laugh: eGIS does).

What do we mean by more?We know that smart meters generate 3K time more reading intervals than traditional meters. Their payload is a lot bigger, too. We also know that there are more sensors and control devices that are used, but we don't have hard numbers on that. Since the company will likely grow let's call it 10,000 or 10^5.

What do we mean by data?This type of data is called time series data: which Wikipedia tells us is a sequence of data points, measured typically at successive points in time spaced at uniform time intervals.

What do we mean by frequently For smart meters, that's typically (but not exclusively) 15 minute intervals. Sensors and syncophasors may be more or less but we are talking about minutes. So we're not talking about days or hours anymore but at least we aren't talking about or seconds or milliseconds. We'll call it near-real time or a 10^3 increase in speed (60 * 24 = 1440) .

Smart Grid Requirements

Essentially, there has been one fundamental requirement change: provide more frequent and robust analytics What do we mean by provideThere will need to be both ad-hoc and structured analytics and reporting. It is worth noting that data at scale is often not amenable to the same types of reports that are used for more modest, enterprise-size data.

What do we mean by more frequent?For most use cases, the difference between advanced and standard is speed, not detail. A generation system is advanced if it can resolve unit commitment problems for the real-time, rather than daily, market. A transmission system is considered advanced if it can resolve phase issues before they cause a problem. The engineering problems are well defined by the laws of physics; we just need to be faster in order to be more reliable, effective and affordable.

What do we mean by robust?If we give a generation analyst access to such a deep, broad and fast pool of data, there are multiple algorithms that can be run against that data to possibly develop new strategies for managing unit commitment in a deregulated market with a wind farm based that could never be tried without that data.

What do we mean by analytics?Analytics, or analysis of data, is a process of inspecting, cleaning, transforming, and modeling data with the goal of discovering useful information, suggesting conclusions, and supporting decision making. The major catagories or analytics that are typically performed on time series data are on the following slide.

Time Series Data Requirements

IndexingGiven a query time series Q, and some similarity/dissimilarity measure D(Q, C), find the most similar time series in database DB

ClusteringFind natural groupings of the time series in database DB under some similarity/dissimilarity measure D(Q, C)

ClassificationGiven an unlabeled time series Q, assign it to one of two or more predefined classes

PredictionGiven a time series Q containing n data points, predict the value at time n + 1.

SummarizationGiven a time series Q containing n data points where n is an extremely large number, create a (possibly graphic) approximation of Q which retains its essential features but fits on a single page, screen, etc.

Anomoly DetectionGiven a time series Q, assumed to be normal, and an unannotated time series R, find all sections of R which contain anomalies or surprising/interesting/unexpected occurrences

Segmentation Given a time series Q containing n data points, construct a model Q1 from K piecewise segments (K