Hardware Implementation of Hardware Implementation of Ethernet Based DNP3 Data LinkEthernet Based DNP3 Data Link

Esteban PadillaEsteban Padilla

Student Member

2 November 2012

Alben Cárdenas Kodjo Agbossou

Member, IEEE Senior Member

Outline Outline

2

IntroductionIntroduction

• The 4 Technology Layers in the Smart Grid • The Role of Ethernet-based DNP3 over FPGA technology

• Communication Scenario

Distributed Network Protocol (DNP3)Distributed Network Protocol (DNP3)

• Enhanced Performance Architecture (EPA) model• Physical Layer

• Data Link Layer • FT3 frame format for DNP3

TriTri--Mode Ethernet MAC (TEMAC)Mode Ethernet MAC (TEMAC)

• The TEMAC and PHY Overview • Xilinx® Virtex-5® FPGA Features

• Configuration of Virtex-5® FPGA TEMAC

Outline Outline

3

Hardware Implementation of DNP3 DLHardware Implementation of DNP3 DL

• Parallel CRC Generation

• Ethernet-FT3 Transmitter and Receiver

• Primary Data Link Control

Test and ResultsTest and Results

• Synthesis Procedure• Timing Performance

• Ethernet Latency for data transactions in DNP3 traffic• Hardware Implementation

Conclusions and Future WorkConclusions and Future Work

IntroductionIntroduction

4

Communication Communication

InfrastructureInfrastructure

Data need to be propagated

Control directives for devices

Provides always available, secure

and reliable information

dissemination

The 4The 4 Technology Layers Technology Layers

of the Smart Grid of the Smart Grid Decision Intelligence

Communication

Sensors & Actuators

Power Conversion/ Transport/Storage/

Consumption

IEEE 1547.3-2007

Guide for monitoring, information exchange, and

control of DR interconnected with EPS

IEEE 1379-1997

Trial-Use Recommended Practice for data

communications between IED and RTU

Introduction Introduction

The Role of EThe Role of E--DNP3 over FPGADNP3 over FPGA

5

Operator Station/

Programming Terminal

Remote Terminal Unit (RTU)

Interfacing Connections

Power

Supply

Central

Processing

Unit

Memory

Non

volatile

Volatile Analog

Input/output

Modules

Digital

Input/output

Modules

IEDIEDIEDNetwork

Protocol

HV Wiring

LAN

Master

Station

RTU

RTU

RealReal--Time Distributed Time Distributed

IntelligenceIntelligence

Advanced grid-monitoring

Management Optimization

Control Applications for

Grid assets

Calculate power ratings

Dynamically balance

• Loads

• Sources

Protocol Requirements Protocol Requirements

Comprehensive communication

Available, flexible and open

Enough bandwidth

Low implementation complexity

IEEE Standard

Communication Requirements Communication Requirements

Two-ways communications

End-to-end reliable and secure communications

Reliability and speed of data communications

Distance and noise immunity

Standarized cables and connections

ETHERNET (DNP3) ETHERNET (DNP3) ______________________________FPGA FPGA

Communicat ion For Communicat ion For Field Area Smart GridField Area Smart Grid

= =

6

PV

Sys

tem

MTUsFPGA

Virtex-5

LAN Ports UTP Cat. 5

(RJ-45)

RTUsFPGA

Virtex-5

Operator Station/Programming Terminal

Wide Area Network

Private Local Area Network

Server

WT

Sys

tem

Printer

Monitoring Data

Smart µGrid for

Renewable Sources

Introduction Introduction

Communication ScenarioCommunication Scenario

Distributed Network ProtocolDistributed Network Protocol

7

ApplicationLayer

Pseudo-Transport

Layer

Data LinkData LinkLayerLayer

PhysicalPhysicalLayerLayer

Software

Layers

HardwareHardware

LayersLayers

TransportTransport

LayersLayers

High

Layers

User Data

ASDU

APDU

LSDU

APDU

LSDU

DATA + CRC DATA + CRC

TPDU

LPDU

A

H

A

H

T

H

T

H

L

H

L

H

1 249 1 249

10 250 + 32 10 250 + 32

44--Layer Layer Enhanced Performance Architecture (EPA) stack Enhanced Performance Architecture (EPA) stack for DNP3for DNP3

Data Link LayerData Link Layer

Accepts, performs and control

Tx and Rx requests

Data conversion into one data-

flow frame

Frame Sync., Link control and

Error detection.

Enhanced Performance Enhanced Performance

ArchitectureArchitecture

Physical LayerPhysical Layer

RS232-C Standard (Up to 120 Kbps)

Large voltage swings (Pos. and Neg. Sources)

Others communications mediums are

recommended

Send, receive, connect, disconnect and

status services.

8

Distributed Network ProtocolDistributed Network Protocol

FT3 frame format for DNP3FT3 frame format for DNP3

Start

Len Ctrl

Dest. Sour. CRC

0x05 0x64 LSB MSB LSB MSB LSB MSB

Header Block

User Data (16 bytes)

CRC

LSB MSB

Data Block 1

User Data (16 bytes)CRC

LSB MSB

Data Block 2

User Data (1 to 16 bytes)CRC

LSB MSB

Data Block N

7 6 5 4 3 2 1 0

DIR1

PRM0

FCB FCVFUNCTIONCODE

← Pri. to Sec.

← Sec. to Pri.0 DFC

Bit Name Description

DIR Direction1 → from a Master

0 → from a Slave

PRMPrimary

Message

1 → Initial message

0 → Final message

FCBFrame

Count Bit Toggling bit

FCVFrame

Count Valid

1 → FCB is Valid

0 → Ignore FCB

DFCData Flow

Control

1 → Buffer overflow

0 → Buffer available

P0000 Reset Link

P0010 Test Link States

P0011 User data – confirm

P0100 User data – no confirm

P1001 Request Link Status

S0000 Confirm – ACK

S0001 Confirm – NACK

S1011 Link Status

S1111 No SupportedThe FT3 frame format for DNP3 is defined asThe FT3 frame format for DNP3 is defined as::

� 10-byte fixed header block. � Up to 16 data blocks of 16 bytes each one. � Each data block has 16-bit CRC field.� The overall message carry out maximum

250 bytes of data payload.

Function Code Key

Bit description for Control fieldMTU RTU

.

.

.

.

.

.

Outgoing

User Data

Incoming

User Data

Incoming

User DataOutgoing

User Data

Request

Command

Optional

Response

ImprovementImprovement

Ethernet could carry out at least

1400 bytes 1400 bytes

(86 data blocks) (86 data blocks)

of data payload.

TriTri--Mode Ethernet MAC (TEMAC)Mode Ethernet MAC (TEMAC)

9

XC5VLX110T FPGAXC5VLX110T FPGA

Ethernet MACEthernet MAC

MAC

MAC

Control

Gigabit Gigabit

Ethernet Ethernet

TransceiverTransceiver

(88E1111)(88E1111)

RJ-

45

Po

rt

I/O

Bu

ffe

rs

FIF

Os

Em

be

dd

ed

Sys

tem

10

0B

AS

E-T

X

Magnetics

Circuitry

Media Independent

Interface

Media Independent

InterfacePHY SidePHY SideClient SideClient Side

DUP/TX/RX LEDs

Xilinx® VirtexXilinx® Virtex--5® 5® FPGA Features FPGA Features Remote data capture for tes t and measurements wi th advanced ser ia l connec t iv i t y and para l le l I /O techno logy.W ired Networking w i th the Tr iW ired Networking w i th the Tr i --Mode Mode Ethernet Ethernet MAC MAC ((Media Access Contro l lersMedia Access Contro l lers))Rapid deve lopment , h igh log ic in tegra t ion and h igh-speed I /O

s tandards wi th 17280 S l i ces (69120 LUTs and FFs) and 680 I /O User Por ts .

The TEMAC and PHY OverviewThe TEMAC and PHY Overview

ConfigurationConfiguration

Standard physical interface

connect it to IOBs for

10/100/1000 Mbps transfer

rates.

ConfigurationConfiguration

8-bits data wide for Client

Side interface with default

clocking scheme

10

Hardware Implementation of DNP3 DLHardware Implementation of DNP3 DL

Use

r A

pp

lica

tio

n L

aye

r ETH-FT3 Transceiver

ETH_FT3_RX

ETH_FT3_TX

CRC Control Logic

TE

MA

C

Data Link

Control

(Primary/

Secondary)

AXI4 InterfaceAXI4 Interface

Data Buffer

IND

REQ

CON

Important Features

• CRC generated at the same time of TX or RX of the frame.

• Data Link Control works parallel at the TX or RX process.

• No additional latency for transmission or reception frames.

• Accepts, performs and controls TX and RX requests.

• Frame Synchronization, Link Control and Error Detection.

• Up to 1498 bytes for payload.

• VHDL-based Hardware Design Approach

ImprovementImprovement

TX, RX and CRC generation

processes could run at the

same time.

Blocks diagram for Ethernet DNP3 Data Link

ImprovementImprovement

Easy-pin interconnections for

User Application Layer.

ImprovementImprovement

Makes extensive use of the

Advanced eXtensible

Interface 4 Lite (AXI4).

11

CRC Computation

Read

8-bit Data Block

16-bit CRC Block

Initialize

Data. = Data Block

CRC = CRC Block

Counter = 0

Counter < 7

Temp = (CRC XOR Data) AND 0x0001

CRC >>= 1

Data >>= 1

Temp = 0x0001 CRC = CRC XOR 0xA6BC

Return CRC Value

TrueCounter = Counter + 1

False

True

False

CRC Computation

Read

8-bit Data Block

16-bit CRC Block

Initialize

Data. = Data Block

CRC = CRC Block

Counter = 0

Counter < 7

Temp = (CRC XOR Data) AND 0x0001

CRC >>= 1

Data >>= 1

Temp = 0x0001 CRC = CRC XOR 0xA6BC

Return CRC Value

TrueCounter = Counter + 1

False

True

False

Uses shift right registers, XOR/AND logical operators and 3-bit adder

Hardware ImplementationHardware Implementation of DNP3 DLof DNP3 DL

ParallelParallel CRC Generation CRC Generation

Proposed Algorithm Flowchart for

Parallel CRC Generation

12

Hardware ImplementationHardware Implementation of DNP3 DLof DNP3 DL

EthernetEthernet--FT3 TransmitterFT3 Transmitter and Receiverand Receiver

IDLE

WAITCHN

SEND ETH HDR

SEND FT3 HDR

SEND USERDATA

CRCFIELD

CRCFINAL

REQ TX

TIMEOUT

TX ACKFROM MAC

DONEDONE

DONE16TH BYTE

NO USER DATA

LAST DATA BLOCK

DONESTARTUP

IDLE

RX

ETH

HDR

CRC

CHK

RX

USER

DATA

RX

FT3

HDR

DATA VALID

FROM MAC

STARTUP

HEADER

NOT

MATCHED

HEADER

MATCHED

HEADER

MATCHEDHEADER

NOT MATCHED

CRC NOT

MATCHED

CRC

MATCHED

16TH

BYTEDON

E

Proposed State Diagrams forTransmission and Reception Processes

Transmitter Receiver

13

IDLE 1

IDLE 2

RES LINK

WT 1

RES LINK

WT 2

TEST WAIT

DATA WAIT

S0 RX S0 RX

RESET

RETRY OUT/NO S0 RX

RETRY OUT/NO S0 RX

SEND CON

DATA

RESET

RETRY OUT/NO S0 RX

S1 RXDFC = 0

REQ TEST

LINK

S0 RX

S0 RX

SEND CON DATA

S1 RXDFC = 1

NO S0 RXNO S1 RX

RETRY OUT

SEND UC DATA

SEND UC DATA

TIME OUT

TIME OUT

TIME OUT

TIME OUT

STARTUP

Proposed State Diagram for Primary Data Link Control

Hardware ImplementationHardware Implementation of DNP3 DLof DNP3 DL

PrimaryPrimary Data Link ControlData Link Control

Test and ResultsTest and Results

Design

Specification

Behavioral VHDL Model

and Testbench Generation Mixed Level

Simulation

Behavioral and

Logical Synthesis

Prototyping and

Hardware

Configuration

Visualization of

System Operation

Debugging

Design’s

Updates

Devices TEMACDNP3

DLTotal

Slices Registers

429 126 555

Slices LUTs

390 498 888

Total Slices LUTs + Registers 1443

Synthesis procedure for improve the device

utilization

0

10000

20000

30000

40000

50000

60000

70000

80000

XC5VLX110T XC2VP30 XC3S1600E

Total Slices LUT

Used Slices LUT

FPGA Models

In In HDL Synthesis for HDL Synthesis for the FPGA Models the the FPGA Models the devices utilization is devices utilization is

less than 2 %less than 2 %

Device Utilization (estimated values)

Sli

ces

14

The device utilization The device utilization rate is a very rate is a very

important aspect to important aspect to consider, since others consider, since others

embedded systems embedded systems require the a lot of the require the a lot of the physical area in FPGAphysical area in FPGA

15

Test and Results Test and Results

Timing PerformanceTiming Performance

Transmitter Simulation

Receiver Simulation

TTX = 52,92 μs for 250 B Payload Test

TRX = 72,08 μs for 250 B Payload Test

200 400 600 800 1000 1200 1400 16000

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8x 10

-3

Payload (Bytes)Payload (Bytes)Payload (Bytes)Payload (Bytes)

Tim

e (

se

co

nd

s)

Ethernet latency for data transaction in DNP3 Traffic (8-port Switch)

Light network load

Heavy network load

RealReal -- t ime t ime Moni tor ingMoni tor ing, ,

Protect ionProtect ion and and Contro l Contro l

� M icr ogr i d

M oni tor ing and

S chedul i ng .

� P ower osc i l l a t i on

and dam ping .

� V ol tag e and

C ur r ent

Opt im iza t io n .

� I n te l l i g e nt load

and sour ce

ba lanc in g .

� E ven m or e…

16

Test and Results Test and Results

HardwareHardware ImplementationImplementation

Simple Network Test (Master-Slave)

Master and Remote Terminal Units in Network

Conclusions and Future WorkConclusions and Future Work

17

Operator Station/

Programming Terminal

Remote Terminal Unit (RTU)Remote Terminal Unit (RTU)

Interfacing Connections

Power

Supply FPGAFPGA

Memory

Non

volatile

Volatile Analog

Input/output

Modules

Digital

Input/output

Modules

IEDIEDIED

HV Wiring

EthernetEthernet DNP3DNP3ProtocolProtocolInterface Interface

Master Station

RTU

RTU

Enhances processing delay and protocol efficiency.It could be used to support the data communications up to 1 Gbps in Smart µGridsFuture work include application layer definition for management, scheduling and monitoring of a µGrids with Renewable Energy Sources (RES).

The EthernetThe Ethernet--based DNP3 Interface for RTU in Smart based DNP3 Interface for RTU in Smart µµ--GridsGrids was described (VHDL), was described (VHDL), implemented and validated for using Virteximplemented and validated for using Virtex--5 FPGA Technology.5 FPGA Technology.

Thank you!Thank you!