1

Design of Real-time Simulator for Electric Power

Distribution System Based on FPGAs

Zhiying Wang

Xiaopeng Fu

Tianjin University, China

2

Background

Motivation

– Integration of renewables in distribution networks

– Utilization of massive electronics in AC-DC hybrid distribution networks

Application

– Southern Grid of China : flexible DC distribution system pilot prototype verification

– Southern Grid of China : fast RT simulation platform for DC distribution networks

– State Grid of China : RT simulation platform for distribution network with DGs integrated

Components of simulation model

– large-scale distribution networks: high dimension of the model

– power electronics interfaced DG: vast range of time-scales

3

Real-time simulation of distribution network

4

Active Distribution Network FPGA-based RT simulator

2.8 2.9 3 3.1 3.2 3.3 3.4-200

-100

0

100

200

300

400

500

600

t/s

2.993 2.995 2.997 2.999 3.00160

80

100

120

140

PSCAD

FPGA

VP

V1

,c/V

6:00 8:00 10:00 12:00 14:00 16:00 18:00 20:000

100

200

300

400

500

600

700

800

900

t/h

PSCAD

FPGA

S/W

/m2

6:00 8:00 10:00 12:00 14:00 16:00 18:00 20:000

2000

4000

6000

8000

10000

12000

14000

16000

t/h

9:50 10:00 10:10 10:205800

6000

6200

6400

6600

PSCAD

FPGA

PP

B,p

v/W

dynamic behaviors

Overview of Multi-FPGA based Real-time simulator Design

system modelling

numerical algorithm

hardware design

platform design

MG

......

MG

...

MVAC

MVDC

LVDC

5

Simulator Design

6

system decomposition

hardware

design

platform design

coarse-to-fine grained system decomposition

map hardware resource to system modelling

Multi-FPGA system topology, synchronization mechanism, data interaction

7

fillter

transformer

PV

line

+-

+

-

+-

+-+

+PI

PIPI

PI

fillter

transformer

BAT

line

+-

+

-

+-

+-+

+PI

PIPI

PI

inverter control

d-axis electrical system

control system

2

3

4

5

6

11 12

9

10

1

7

8

15 16

13

14

19

20

17

18

22

21

coarsegrained

1

3 4 5

6

9 10 11

2

7 8

17 18 19 20 21

12

13 14 15 16

22

q-axis

d-axis

q-axis

inverter controlfine

grained

Electrical & Control

TLM

fine-grained parallelism

… …

System decomposition at various levels:

Coarse-to-fine grained decomposition of active distribution network

8

FPGA #1 FPGA #2 FPGA #3

Core #1

Core #2

Core #3

Core #5

Core #6

Core #4

Core #7

+-

+

-

+-

+-+

+PI

PIPI

PI

E

C

d

q PI PI

PI PI

PI PIVdc

Is VsIGBT XFM

data transmission

state_number

v(t)

history current source

matrix solving

Electrical system

G-11 G-1

2 G-1N

...

ROM

trigonometric function

exponential function

limiter

adder

compator

transfer function

Control system

supply source

control source

transformer

power electronic device

breaker

meter

electrical elements

RLCG

line

iIF(t-τ)

G-1

v&i

v&i

switch_sig

PWM

iSOURCE

injected node

currents

Bergeron interfacesimpler

FPGA #4 FPGA #5 FPGA #6 FPGA #7 FPGA #8

extendable

module level parallelism

element level parallelism

MANA

ih(t-Δt )

i

system levelparallelism

unit level parallelism

...

Core #...

9

combined mesh and linear array topology for the multi-FPGA system

plug and play

extendable

flexible configuration

maximum utilization of data

transmission channel

Multi-FPGA system topology:

10

· · ·

· · ·

sta_sync_sigFPGA1

(master)

FPGA2

(slave)

· · ·

· · ·

sta_sync_sigFPGA1

(master)

FPGA2

(slave)

· · ·

· · ·

FPGA3

(slave)

· · ·

2t

2t

2t

2t

3t

3t

1t

1t

1t

1t

1t

1t

1t

1t

2t

3t

3t

mint

1 min2t t

2 min3t t

3 min4t t

synchronize the simulation time-step of each FPGA to the same time reference

Single-rate synchronization mechanism Multi-rate Synchronization mechanism

Synchronization mechanism：

11

Multirate Simulation (target power electronics)

12

fast system

physical identification:

Including high-

frequency power

electronic devices

slow system

distribution network

without power

electronic devicesdistribution network

(slow system)

DG

FPGA #1

FPGA #2

FPGA #3

FPGA #...

FPGA #7

FPGA #6FPGA #4

DG(fast system)

v&i

controller

PWM

DG & ESSpower electronic

converters

PMS

M

+-

+

-

+-

+-+

+PI

PIPI

PI

DG

DG

DG

...

DG

...

FPGA #5

DG(fast system)

~~

~

PMSM

~~

......

Controlled signal

Multirate simulation between distributed generation(DG) & distribution network(DN)

13

Multirate simulation between distributed generation(DG) & distribution network(DN)

multirate interfacing method:

slow system Interpolated points

fast system

slow system

fast system

TLM decoupling between DG and DN

interpolation of slow system

averaging of fast system

14

Multirate simulation between electrical system & control system in a DG

multirate interfacing method:

electrical & control system decoupling

directly data interaction

asynchronous multirate simulation

v&i

PWM

DG & ESS(slow system)

PMSM ~

~

~

Controlled signal

power electronic converters(fast system)

controller(slow system)

+-

+

-

+-

+-+

+PI

PIPI

PI

15

Expanding Modeling Capability of Large-scale

Distribution Networks with the Matrix Exponential

Method

Xiaopeng Fu, Lecturer

Tianjin University, China

16

G

k

Ih

k

m

m

Geq

-Geq

-Geq

Geq

-Ih

+Ih

nodal admittance equation

ikm(t)

vk(t) vm(t)

Gu=i

ikm(t)

Ih(t-Δt)

vk(t) vm(t)Geq

RL branch

Electrical System Solution

difference equation using root-matching technique

𝑖km 𝑡 = e−∆𝑡𝑅/𝐿𝑖km(𝑡 − ∆𝑡) +1 − e−∆𝑡𝑅/𝐿

𝑅𝑣km 𝑡

equivalent circuit

etA appears in its Padé-approximant form

17

The Matrix Exponential etA

For arbitrary 𝑨 ∈ ℝ𝑛×𝑛, 𝐞𝑨 is defined as

𝐞𝑨 ∶= 𝑰 + 𝑨 + Τ𝑨2 2! + Τ𝑨3 3! + ⋯

Closely related to state transition of dynamic systems

Solver Integrator Padé Approx. ex ≈ rkm(x)

FE 𝑰 + 𝑡𝑨 𝑟10(𝑥) = 1 + 𝑥

BE 𝑰 − 𝑡𝑨 −1 𝑟01(𝑥) = 1/(1 − 𝑥)

TRAP 2𝑰 − 𝑡𝑨 −𝟏 2𝑰 + 𝑡𝑨 𝑟11(𝑥) = (2 + 𝑥)/(2 − 𝑥)

ARTEMiSArt5

𝐞𝑡𝑨 ≈𝑰 + 2𝑡𝑨/5 + 𝑡𝑨 𝟐/20

𝑰 − 3𝑡𝑨/5 + 3 𝑡𝑨 𝟐/20 − (𝑡𝑨)𝟑/60

LRivk vm

ihist

ivk vm

Geq

Nodal Analysis State Space Analysis

Solver Conductance 𝑮𝒆𝒒 History current source 𝒊𝐡𝐢𝐬𝐭 𝒕

TRAP 𝑅 +2𝐿

Δ𝑡

−1

𝐺eq 𝑣𝑘𝑚 𝑡 − Δ𝑡 − 𝑅 −2𝐿

Δ𝑡𝑖 𝑡 − Δ𝑡

Root Matching*

1 − e−Δ𝑡𝑅/𝐿

𝑅e−Δ𝑡

𝑅𝐿 ∙ 𝑖 𝑡 − Δ𝑡

Linear System Theory

ሶ𝒙 𝑡 = 𝑨𝒙 𝑡 + 𝒃 𝑡 ,has state transition rule:

𝒙 𝑡 = 𝐞𝑡𝑨𝒙 0 + න0

𝑡

𝐞 𝑡−𝜏 𝑨𝒃 𝜏 𝑑𝜏

branch discretizationwith scalar exponential

18

A-stability, Free from numerical oscillation problems, etc.

The matrix exponential formula for nonlinear systems…

Variation-of-constant formula:

𝒙 𝑡 = 𝐞 𝑡−𝑡0 𝑳𝒙0 +න𝑡0

𝑡

𝐞 𝑡−𝜏 𝑳𝑵 𝜏, 𝒙 𝜏 𝑑𝜏

General nonlinear system

ሶ𝒙 𝑡 = 𝑭 𝑡, 𝒙 𝑡 = 𝑳𝒙 𝑡 + 𝑵 𝑡, 𝒙 𝑡linear nonlinear

Matrix Exponential-based Integrators for State-Space Analysis Program

Linear Network RLC branch, PI-section, linear

transformer, etc.

Standard SS equation,

automatically formed

Nonlinear Electric machine,

power converters, dynamic loads,

etc.

Dedicated model library

linear networks

px1 = Ax1+Bu1

y1 = Cx1+Du1

nonlinear injectors

px2 = f (t,x2,u2)

y2 = g(t,x2,u2)

y2

u1

u2

y1

Numerical Discretization at the System Level

Krylov Subspace

For 𝑨 ∈ ℝ𝑛×𝑛 and 𝒗 ∈ ℝ𝑛×𝑛, a q–dimensional Krylov subspace

is defined as

𝑲𝑚 = span 𝒗, 𝑨𝒗,… , 𝑨𝑚−1𝒗 ,

Action of matrix functions: Krylov based approximation

For 𝑲𝑚 𝑨, 𝒗 , Arnoldi iteration gives 𝐕m and 𝐇m = 𝐕m𝐓𝐀𝐕m, which

enables

𝝋𝑝 𝑨 𝒗 ≈ 𝑽𝑚𝝋𝑝 𝑯𝑚 𝑽𝑚T 𝒗

Reduced Order Simulation with Krylov Approximation

19

In the originaldimension 𝑁

In the reduced dimension (𝑚 ≪ 𝑁)

算法 1 Krylov 子空间降阶的指数欧拉仿真算法

𝑡 = 𝑡0;𝒙 = 𝒙0

𝐫𝐞𝐩𝐞𝐚𝐭

compute 𝒇(𝑡,𝒙)

initial guess 𝑚 = 𝑚0

𝐫𝐞𝐩𝐞𝐚𝐭

compute 𝑽𝑚 ,𝑯𝑚 with Arnoldi iteration

construct 𝑯 with 𝑝 = 1

compute 𝐞ℎ𝑯 with scaling & squaring

𝑭 = approximation to ℎ𝝋1(ℎ𝑨)𝒇(𝑡,𝒙)

𝜀 = error estimation 𝑚 = 𝑚new

𝐮𝐧𝐭𝐢𝐥 𝜀 < Tol

𝒙 = 𝒙 + 𝑭 (指数欧拉法)

𝑡 = 𝑡 + ℎ𝐮𝐧𝐭𝐢𝐥 𝑡 = 𝑡end

Algorithm: Exp-Euler with standard Krylov subspace approximation

Matrix Exponential-based Integrators with Krylov Subspace Acceleration

20

Improvement 1: Subspace Reuse

• Continuous linearization => Retaining Jacobian𝒙𝑛+𝑘 = 𝒙𝑛+𝑘−1 + ℎ𝝋1 ℎ𝑱𝑛 𝑭 𝑡𝑛+𝑘−1, 𝒙𝑛+𝑘−1

• Construct formula that exploits Jacobian reuse𝒙𝒏+𝒌 = ෝ𝒙𝒏+𝒌 + 𝒙𝒏+𝒌,ෝ𝒙𝒏+𝒌 = 𝒙𝒏 + 𝑘ℎ𝝋1 𝑘ℎ𝑱𝒏 𝑭 𝒕𝒏, 𝒙𝒏 ,𝒙𝒏+𝒌 = 𝒙𝒏+𝒌−𝟏 + ℎ𝝋1 ℎ𝑱𝒏 𝑱𝒏𝒙𝒏+𝒌−𝟏 + 𝚫𝒈𝒏𝒌

• Adjust Jacobian update rate with error control

Improvement 2: Extended Krylov Subspace

• Standard Krylov subspaces𝐊𝑚 𝑨, 𝒗 = 𝐬𝐩𝐚𝐧 𝒗, 𝑨𝒗,… , 𝑨𝑚−1𝒗

match eigenvalues from largest magnitude

• Extended Krylov subspace𝐊𝛾𝑞,𝑛

𝑨, 𝒗 = 𝐊𝑞 𝑨, 𝒗

⊗ 𝐊𝑛 𝑰 − 𝛾𝑨 −1, 𝒗

More targeted matching of the eigenvalues

polynomial

rational

EI with Kry. Approx. 𝒙n+1 = 𝐞ℎ𝑨𝒙n ≈ 𝑽m𝐞ℎ𝑯m𝑽m

𝐓 𝒙n

ෝ𝒙n = 𝑽m𝐓 𝒙n Order reduction, 𝑽m

𝐓 ∈ ℝ𝑚×𝑁

ෝ𝒙n+1 = 𝐞ℎ𝑯mෝ𝒙n Numerical integration in ℝ𝑚×𝑚

𝒙n+1 = 𝑽mෝ𝒙n+1 Transforming back to ℝ𝑁×𝑁 space

ෝ𝒙n+1 = 𝐞ℎ𝑯mෝ𝒙n

ෝ𝒙n+2 = 𝐞ℎ𝑯mෝ𝒙n+1

⋮

Multiple steps in low dimension

Moment-matching based MOR has 𝐊m 𝑰 − σ𝑨 −𝟏, 𝒃

Different matching criteria

MOR Viewpoint & Further Improvements

21

Efficiency Comparison

The speed up ratios of exponential integrators over other solver are 4.22, 4.55, 9.67, 17.48 for four cases. Computation Time Growth Rate

• SPS, ode23t, ode15s have roughly 𝑁1.7

• Best exponential solver has sublinear growth 𝑁0.66

SolverComputation Time under 10 𝜇s Step Size (s)

Case1: 17 WTGs Case2: 26 WTGs Case3: 66 WTGs Case4: 100 WTGs

SPS 297.31 575.81 3143.61 6036.91

ode23t 269.29 513.80 2663.86 5377.92

ode15s 279.85 537.57 2804.93 6013.67

ExpFix 63.77 171.43 2189.23 5591.23

ExpMRb + RT-Krylov 95.34 112.81 234.16 307.68

Speedup ratio 4.22 ExpFix 4.55 ExpMRb+RT-Krylov 9.67 ExpMRb+RT-Krylov 17.48 ExpMRb+RT-Krylov

Numerical Test of Large-scale Wind Farm

22

Scaling & Squaring Method for the Matrix Exponential

For arbitrary 𝑨 ∈ ℝ𝑛×𝑛, we have 𝐞𝑨 = 𝐞 Τ𝑨 𝜎 𝜎

Scaling: choose 𝜎 = 2𝑠 s.t. Τ𝑨 𝜎 is sufficiently small, use 𝐞 Τ𝑨 𝜎 ≈ 𝐫e Τ𝑨 𝜎

Squaring: 𝐞𝑨 recovered by successive squaring 𝐞𝑨 = 𝐞 Τ𝑨 𝜎 𝜎≈ 𝐫e

𝑨

𝜎

2𝑠

.

0 87654321

𝐞𝑑𝑡8 𝑨 ≈ 𝐫e

𝑑𝑡

8𝑨𝐞

2𝑑𝑡8 𝑨 = 𝐞

𝑑𝑡8 𝑨 ⋅ 𝐞

𝑑𝑡8 𝑨 𝐞

4𝑑𝑡8 𝑨 = 𝐞

2𝑑𝑡8 𝑨 ⋅ 𝐞

2𝑑𝑡8 𝑨 𝐞𝑑𝑡𝑨 = 𝐞

4𝑑𝑡8𝑨 ⋅ 𝐞

4𝑑𝑡8𝑨

Example: use 𝜎 = 23 = 8 to compute 𝐞𝑑𝑡𝑨

Dense Outputs in Large Step Size Integrations

23

Scaling & Squaring Method for the Matrix Exponential

For arbitrary 𝑨 ∈ ℝ𝑛×𝑛, we have 𝐞𝑨 = 𝐞 Τ𝑨 𝜎 𝜎

Scaling: choose 𝜎 = 2𝑠 s.t. Τ𝑨 𝜎 is sufficiently small, use 𝐞 Τ𝑨 𝜎 ≈ 𝐫e Τ𝑨 𝜎

Squaring: 𝐞𝑨 recovered by successive squaring 𝐞𝑨 = 𝐞 Τ𝑨 𝜎 𝜎≈ 𝐫e

𝑨

𝜎

2𝑠

.

Large step-size integration with dense outputs makes multirate interfacing easy!

w/o dense output 4 dense outputs per step 16 dense outputs per step

Dense Outputs in Large Step Size Integrations

24

ሶ𝒙 𝑡 = 𝑨𝒙 𝑡 + 𝒈 𝑡, 𝒙 𝑡 +𝑖=1

𝑁

𝑩𝑖𝒚𝑖 𝑡

𝒚𝑖 𝑡 = 𝒉𝑖 𝒙 𝑡 − 𝜏𝑖 + 𝑪𝑖𝒚𝑖 𝑡 − 𝜏𝑖 , 𝑖 = 1,… ,𝑁.

𝒙 𝑡 = 𝑺 𝑡 , 𝑡 ∈ −max 𝜏𝑖 , 0

𝒚𝑖 𝑡 = 𝑯𝑖 𝑡 , 𝑡 ∈ −𝜏𝑖 , 0

Delayed-Differential-Algebraic Equation (D-DAE)

ቐ

State space formulation

with propagative elements

vk vm

ikm imk

...

Lumped-parameter

system m

Lumped-parameter

system n

With Initial Condition

ቐ

Iterative EI Solution for D-DAE (enables 𝛥𝑡 > 𝜏)

𝒙𝑛+1𝑘

= 𝐞𝛥𝑡𝑨𝒙𝑛 + 𝛥𝑡𝝋1 𝛥𝑡𝑨 𝒈 𝑡𝑛, 𝒙𝑛 +

𝑙=1

𝑝+1

𝛥𝑡𝑙𝝋𝑙 𝛥𝑡𝑨 𝑩𝒂𝑙−1𝑘−1

with polynomial fitting

time[1]

tn-τ

[2]

tn tn+τ tn+2τ tn+(Nτ—1)τ tn+Nττ

时步积分

时滞项计算

基于[tn-τ,tn]的时滞项预测

预测步积分

time

tn-τ

[4]

tn tn+τ tn+2τ tn+(Nτ—1)τ tn+Nττ

基于预测步的时滞项校正(1)

校正步积分(1)

time

tn-τ

[2k+2]

tn tn+τ tn+2τ tn+(Nτ—1)τ tn+Nττ

基于校正步(k-1)的时滞项校正(k)

校正步积分(k)

[2k+1]

[3]

预测步

第1个校正步

第k个校正步

已知时滞项

已知时滞项

𝑰h𝑘

𝑡 =

𝑙=0

𝑝𝒂𝑙

𝑘

𝑙!𝑡 − 𝑡𝑛

𝑙

Exploit model discontinuity property

• Initial wave propagation phenomenon => small step

• Wave propagation dying out => large step

Large Step Size Integration with Propagative Elements

Summary• Expandable architecture with multi-FPGA hardware, each

FPGA is functionally complete

• Suitable step size selection fitting subnetwork timescales, enabled by multirate interfacing techniques

• High-fidelity simulation results for studied system

• Full EMT modeling of external system through Krylovsubspace techniques with multirate interface

25

26

Thanks!Questions?

fuxiaopeng@tju.edu.cn