High Availability System White Paper External

Copyright 2011 Rockwell Automation, Inc. All rights reserved.

High Availability

System White Paper

Copyright 2011 Rockwell Automation, Inc. All rights reserved. 2

Outline

Purpose

System Description

Test Data

Summary and Recommendations

Appendix


Purpose

The purpose of this document is to help customers that are implementing a Rockwell High Availability system using Ethernet at multiple levels

(HMI, ControlLogix redundancy and I/O) understand the effect of system

interruptions including loss of power and Ethernet cable breaks.

The High Availability architecture described in this document provides protection for any single point of failure causing a disruption of control of the system

DLR (Device Level Ring) provides protection for media breaks in the I/O network but does not protect against I/O module failure.

If redundant I/O is required then 1715 I/O should be used in place of the Point I/O products


Levels of Redundancy Provided for the Architecture on Next Slide

8 levels of redundancy (see numbers in yellow circles on next slide) 1. Redundant FT data servers: Manage through FactoryTalk Service Platform (FTSP).

2. NIC teaming on servers and client: Configured as load sharing on two physical Ethernet modules on each server and client. Switchover time for cable or NIC loss is vendor dependent.

3. L2 switches (two Cisco 2960): Connected directly to FT data servers and client. Refer to Catalyst 2960 and 2960-S Switches Software Configuration Guide, Cisco IOS Release 12.2(58) SE, April 2011.

4. Router stack (two Cisco 3750G): Configured for stackwise as one logical unit. We are using a stack of 2 physical switches, but more can be added. Refer to Catalyst 3750 Switch Software Configuration

Guide, Cisco IOS Release 12.2(58) SE, April 2011.

5. Dual Ethernet media: From Stratix 8000 to Cisco 3750, from 3750 to 2960, from Cisco 2960 to FT servers. Configured for Etherchannel, load balancing is provided and allows fast recovery if a cable is cut.

Refer to RA Literature Library, Converged Plantwide (CPwE) Design and Implementation Guide, ENET-

TD001E-EN-P, September 2011.

6. L2 switches (two Stratix 8000): If the primary CLGX switch loses power, CLX will switchover and the new primary's Stratix 8000 will provide data. Refer to RA Literature Library, ControlLogix Enhanced

Redundancy System User Manual, 1756-UM001L-EN-P, June 2011.

7. ControlLogix chassis: Two 1756-L75 controllers configured for ControlLogix Redundancy. Refer to RA Literature Library, ControlLogix Enhanced Redundancy System User Manual, 1756-UM001L-EN-P,

June 2011.

8. DLR (Device Level Ring): Quick network recovery time. Refer to RA Literature Library, ENET-AP005D-EN-P, August 2011.


Architecture Configuration for Test System

Server A Server B

FA0/1 FA0/3

G0/1

FA0/3 FA0/2 FA0/2 FA0/1

G2/0/25 G2/0/28

G1/0/28

G2/0/26

G1/0/26 G1/0/25 G1/0/27

G0/2

G2/0/27

Client

NIC teaming on

servers and client

Cisco 3750

192.168.11.1

192.168.20.1

192.168.110.1

5 VLANs configured: 192.168.20.0 HMI

192.168.110.0 Switch management

192.168.11.0 Zone

192.168.100.0 Native

192.168.10.0 I/O

192.168.20.20 192.168.20.21 192.168.20.22

Cisco 2960_A 192.168.110.12

Cisco 2960_B 192.168.110.13

G0/1 G0/2

Redundant

Logix Controllers

EtherNet/IP

DLR

EtherNet/IP

FA1/1 FA1/1

G1/1 G1/2 G1/1 G1/2

CLGX B

192.168.10.10 192.168.10.11

192.168.11.12 192.168.11.13

192.168.10.13 192.168.10.28

Stratix 8000, B 192.168.110.11

CIP, 192.168.11.9

CLGX A

Stratix 8000, A 192.168.110.10

CIP, 192.168.11.8

StackWise

Switch Stack

Cell/Area Zone Levels 0-2

Manufacturing Zone Site Manufacturing

Operations and Control

Level 3

G x/y/z or G y/z or FA y/z: G Gigabit (All ports on C3750 are Gigabit). FA Fast Ethernet. x stack number, we have 2 stacks members.

y module number, Cisco has switches that have multiple modules.

z port number.

16 1734-AENTR with Point I/O

1

2 3

4

5

5

5

6

7

8


Additional Information for the High Availability Architecture Test

The Cell/Area Zone could include many ControlLogix redundancy pairs or standalone Logix controllers, not just one as in our test.

Copper media was used for all testing.

Data interruption at the server will occur when a ControlLogix switchover occurs.

It will take additional time for RSLinx Enterprise (RSLE) to access data in controller memory when a connection is lost, either due to a cable break or a redundancy

switchover. The amount of time depends on the number of tags being read and the

type of tags. During this time, the user is not able to observe data changes at the

client. The measurements made were from first data loss to all data restored.

All tests were single point of failure tests. Only one test was conducted at a time, allowing the system to recover before causing a second failure.


Additional Information for the High Availability Architecture Test contd

Using scattered tags in your controller versus arrayed tags

Scattered tags are defined as tags created individually and as a result are usually scattered throughout memory of the controller

These tags require more time to access data in controller memory when being requested by a data server such as RSLinx Enterprise than arrayed tags

Arrayed tags are defined as tags created in controller memory using the array structure and as a result occupy a contiguous block of memory in the controller,

each element in sequence.

Arrayed tags take significantly less time to access data in controller memory when being requested by a data server such as RSLinx Enterprise than

scattered tags

The High Availability architecture testing that was conducted utilized scattered tags since this is the worst case configuration for accessing

data from the controller


Test Configuration Details

Test environment (Please refer to testing results spreadsheet on the next slide for details):

10,000 scattered tags were present in controller, 5000 scattered tags were on scan

Controller CPU usage at 50% with null time greater than 30%

Null time is defined as available time that the controller CPU has to perform additional functions such as providing additional communications time

Used power removal and cable breaks to cause system interruptions

Wireshark was used to monitor the new primarys 1756-EN2TR module data traffic to verify that all data was being read. This measurement

provided the interval of data interruption for our test results.


Test Configuration Details contd

For each test conducted, we measure system effects in three areas:

Effect on I/O

EN2TR Module Missed packets

I/O connection loss

Effect on ControlLogix Redundancy System

Unexpected ControlLogix switchovers

New secondary controller recovery time

Effect on Data reads by RSLinx Enterprise Test Client

Interval of data interruption (seconds)

Unexpected data server switchovers


Test Data Results

Below is a sample screen capture of the complete test results.

Refer to the PDF file High Availability Testing Results for details.


Test Data Different Tag Types Affect Data Interruption Intervals

Arrayed tags (1 array) vs. scattered tags. (Review slide 7 for additional details)

Keep controller CPU usage at 50%, null time is greater than 30%.

Remove power of primary ControlLogix controller to cause a switchover.

Number of Tags

in Controller

Number of Tags

on Scan

Data Interruption (sec)

1 2 3 Avg.

10,000 scattered 5,000 scattered 42 41 41 41

10,000 arrayed 5,000 arrayed 8 7 7 7

5,000 scattered &

5,000 arrayed

2,500 scattered &

2,500 arrayed 18 17 18 18

Arrayed tags have a lower interval of data interruption when a

switchover occurs than scattered tags.


Summary and Recommendations

All system behavior was as expected based on the design of the system architecture

To reduce the interval of data interruption when a switchover occurs, utilize arrayed tags whenever possible when an RSLinx data server is

requesting data from a Logix controller

The Appendix of this document provides a detailed parts list for the High Availability Architecture that was tested


Appendix

Hardware description and revisions

Two 1756-L75 ControlLogix5575, V19.53

Two 1756-RM/B redundancy module, V3.2

Four 1756-EN2TR 10/100 Mbps Ethernet bridge, V4.3

Sixteen 1734-AENTR, Ethernet adapter, V3.6

Two 1783-MS10T, Stratix8000, V12.2(55) SE

Two Catalyst 2960, Cisco switch, V12.2(58) SE2

Two Catalyst 3750G, Cisco switch, V12.2(58) SE5

Three computers, two for data servers, one for client

Six Broadcom NetXtreme Gigabit Ethernet Card*

ASIC Version: BCM5722 A0

Firmware Version: 5722-V3.08

Management Firmware: ASFIPMI V6.02

* Configured as Smart Load Balancing (TM) and Failover (SLB) for NIC Teaming


Appendix contd

Software description and revisions

RSLogix 5000, V19.01

RSLinx Classic Gateway, V2.57.00.14 CPR 9 SR 3

RSLinx Enterprise, V5.30 CPR 9 SR 3

Data Test Client, V2.300.01.0005

Wireshark, V1.4.3

Broadcom Advanced Control Suite 3, V12.6.13.0

Files provided

ControlLogix ACD files

Switch configuration files (Stratix 8000, C2960, C3750G)

Data Test Client XML files

Documents

High Availability System White Paper External