Efficient PMU Data Analysis through High …...2015/10/14 · Efficient PMU Data Analysis through High Performance Data Management Platform 10/14/2015 Bo Lucy Yang, Jun Yamazaki,

© Hitachi, Ltd. 2015. All rights reserved.

NASPI WG meeting

Data & Network Management Task Team

Efficient PMU Data Analysis through

High Performance Data Management Platform

Hitachi America Ltd.

Big Data Research Laboratory

10/14/2015

Bo Lucy Yang, Jun Yamazaki, Norifumi Nishikawa,

Hsiu-Khuern Tang, Alex Wang, Anshuman Sahu


1. Platform Architecture

2. High Performance Data Management

3. Integrated PMU applications

Contents

1

4. Visualization


Background

• PMUs have become increasingly popular in North America

• Many PMU data analysis tools have been developed

• Grid dynamics monitoring

• Oscillation detection

• Model validation

Challenges

Increasing PMU data size requires;

• Fast loading of historical data (Hundreds of TB/year)

• Efficient data cleansing against missing and bad data

• Effective data analysis

Motivation

2

To accelerate integrated online/offline analysis

fully utilizing every PMU data into grid operation

PMU installation in North America

0

500

1000

1500

2000

2500

2010 2015

Num

be

r

166

～2,000*1

*1: Silverstein, A. May 2015. NERC Board of Trustees

Meeting, May 5, 2015, "An Update on the North

American SynchroPhasor Initiative"


High performance data management platform for PMU data apps

• High speed database engine

• Integrated PMU applications

• Visualizations

1-1. Platform Architecture

3

Hitachi Visualization

Oscillation

Analysis

Hitachi Data

Analysis

Package

Hitachi Machine

Learning

Hitachi Database

Common Framework

For Third Party Tools

Contingency

Analysis

Other

Tools


1-2. Platform Architecture

4

• Total integration from fast data acquisition to various applications

and effective visualization

Online

PMU

ED

OD

PAA

Historical

PMU

VSA

ED

OD

PAA

VSA

ML

DSA

Case.1

Case.2

…

Hitachi Visualization

Oscillation

Analysis

Hitachi Data

Analysis

Package

Hitachi

Machine

Learning

Hitachi Database

Common Framework For

Third Party Tools

Contingency

Analysis Dynamic Model

Online

PMU SCADA Topology

SE

DSA

MV

Historical

PMU

CA


5×108

4×108

3×108

2×108

1×108A

ddre

ss (

sect

or)

0 1000 2000

Processing time (sec)

2-1. Hitachi database technology

5

Application of the outcome of “Development of the fastest database engine for the era of very large database, and Experiment and evaluation of strategic social services enabled by the database engine” project (Principle

Investigator: Prof. Masaru Kitsuregawa, University of Tokyo and also Director-General, National Institute of Informatics), supported by the Japanese Cabinet Office’s FIRST Program (Funding Program for World-Leading

Innovative R&D on Science and Technology).

*1 A new principle invented by Professor Kitsuregawa and Project Associate Professor Goda (The University of Tokyo).

■ Fully utilizes the hardware (server, storage) resources

■ SQL processing for DB search is automatically divided and executed with a high degree of parallelism

Task Allocation Search Processing I/O Wait Disk I/O

Server

Storage

[Conventional Approach] Sequential Execution

[New Approach] Out-of-Order Execution Principle*1

Search Performance(μs)

Periodic I/O Processing(ms)

【Storage Access Trace in Conventional Approach】

Server

Storage

more than thousands

of tasks

Reduce the processing time dramatically

Fully utilize storage’s performance

0 1000 2000

Processing time (sec)

5×108

4×108

3×108

2×108

1×108Add

ress

(se

ctor

)

【Storage Access Trace in New Approach】


2-2. Performance evaluation

6

110.0

134.3

2.0 3.7

0

20

40

60

80

100

120

140

Query #1 Query #2

Conv. DB

HADB

x 55 x 36

HADB: Hitachi Advanced Database

Query #1

Time series Trend Search

10 minutes, 4 PMU

Query #2

Snapshot Trend Search

5 snapshots, 500 PMU


3-1. Integrated PMU applications

7

Unsupervised data cleansing

• Outlier detection without domain knowledge

• Noise reduction based on data correlation

• Unsupervised, scalable solution


3-2. Integrated PMU applications

8

Automatic abnormality event detection

• Online detection of grid events with measurement-based method

• Historical data mining for similar events

• Updating detection rules from stored historical event data

Start

End

Compute Feature

Vector for PMUs

Event Decision

Update thresholds

• FFT amplitude

• Residue from fitted

complex exponentials

• Spectral density

• Max-min change

• …


4. Visualization

9

Visualization of various applications

• Coordination of information derived from heterogeneous data

• Intuitive display and operator support


Conclusion

10

Conclusion

To utilize both online and historical PMU data for operation…

• Fast DB for acceleration and better efficiency of analysis tasks

• Coordination of various power apps for comprehensive analysis

• Visualization for intuitive awareness with heterogeneous information

Future Plan

• Performance evaluation tests are ongoing on every layers; Database, Applications, and Visualization

• Integration of third party analysis tools to be tested

• Evaluation with real grid data will be planned


Efficient PMU Data Analysis through

High Performance Data Management Platform

10/14/2015

Bo Lucy Yang, Jun Yamazaki, Norifumi Nishikawa,

Hsiu-Khuern Tang, Alex Wang, Anshuman Sahu

END

Hitachi America Ltd.

Big Data Research Laboratory

11

Documents

Efficient PMU Data Analysis through High …...2015/10/14 · Efficient PMU Data Analysis through High Performance Data Management Platform 10/14/2015 Bo Lucy Yang, Jun Yamazaki,