Ethernet and Determinism by: Michael Roche, Principle Network Applications Engineer, Schneider Electric There have been vigorous debates about Ethernet and determinism when Ethernet is compared with other fieldbus that are widely accepted in industry to be deterministic. Some view Industrial Ethernet systems as not deterministic when compared with those other proprietary Industrial fieldbus systems. We will examine that Industrial Ethernet is in fact, deterministic under some common conditions. First though, we must study the definition of determinism. A deterministic system is regarded as predictable, by producing a calculable, consistent response time between 2 end devices. The application time response (ART) requirement in a deterministic system varies with each application. For example, a wastewater system may be a deterministic system producing a consistent and predictable application response time of 500 ms, whereas a CNC motion system may require an application response time of 1 ms. By the same logic, any annual holiday is deterministic as it falls within a specified calendar day. These previous examples illustrate that determinism is a factor of each particular application with varying customer and process requirements. The essence of determinism is the predictability and consistency of each and all individual operations that suits the application requirements at hand. Many systems boast of determinism, but examined more closely, Ethernet can stack up against the best of them. Consider that of a Distributed I/O system (DIO), where slave I/O control must be controlled by a Master PLC. Loss of communications in a DIO system results in losing control of your process. Any delays due to network modifications that change the expectation of DIO ART performance can cause problems and require unanticipated logic and timing changes just because you added some devices, or another cable segment. While each fieldbus has its baggage, Ethernet offers the best deal in terms of growing your process and factory, with less pain and fewer concessions. Other Fieldbus make you pay the price in very strict terms due to node and device count limitation. Determinism in a DIO system can be considered one of two ways. The first is system determinism which includes the PLC with logic solve and component servicing, the network interface transport and the response from an I/O device. System determinism can also be thought of as Application Response Time (ART). This can be defined as the amount of time it takes for an input on an I/O device to turn on, for the change of state to be detected by the PLC, and for the PLC logic to solve that change of state and write the logical output of the I/O device as shown in Figure 1. ART includes several basic components:

CPU Scan time and backplane module/communications processing Packaging and transport from the PLC network interface Transmission over the network or fieldbus Receipt and processing by the I/O device Response time for the I/O device to present a new condition confirmed value to the

originating PLC and for the PLC logic to process the change

The overall ART can also vary slightly based upon CPU scan drift, due perhaps to online programming activity and burst processing requests from devices such as HMIs or SCADA, process timing in terms of exactly when in the CPU scan cycle, the request was received, and jitter on the network due to device additions/deletions, or periodic burst communications activity. In most cases, it will take more than one CPU scan to resolve the I/O input change and respond with a output message request, regardless of the deterministic fieldbus used. We can conclude therefore, that even the most deterministic systems experience timing drift from one transaction to another based upon very slight deviations. With deterministic fieldbus, this time is typically within the range of 1 or 2 milliseconds. As we will see with conditioned Ethernet as a fieldbus, a one millisecond drift is an unusual deviation due to the speed, precision and evolved features of Ethernet. This article will examine the network transmission component only, and compare the functionality of Ethernet with other fieldbus systems widely regarded as deterministic. After all, response time drift due to non-periodic PLC CPU scan and the other noted criteria is inherent in any automation system, regardless of the fieldbus used. Network Determinism can be defined as the calculable and consistent transport of automation application messages from the interface on one end device to the interface of another end device. Network determinism refers specifically to the transport between these device interfaces. Fieldbus Comparison Most Distributed I/O (DIO) fieldbus systems regarded as deterministic have been logical ring/physical bus token passing networks like Profibus, Modbus Plus, and others. Using Modbus Plus DIO as an example, the time to transmit a message request or response can be calculated, and there is assurance that the PLC request or I/O device response

transmission time will be consistent. Most deterministic networks use token passing as a way of guaranteeing that each device on the network has an opportunity to transmit and receive at a predictable interval. Token passing networks can limit the amount of time any single device can hold onto the token to ensure that access by all devices is granted. The amount of time it takes the token to be passed from an originating point, through all devices and back to the origin is the token rotation time. The token rotation time will grow as the number of devices increase, or the amount of data is increased because the token is passed sequentially for distribution to all devices. In other deterministic networks such as Profibus, the transmission time will increase due to reduced transmission speed as the network distance lengthens beyond certain network length boundaries. Though in these cases, the overall transmission time may increase with the amount of data, devices or distance, the fieldbus itself is considered deterministic due to the calculable and consistent network transport message delivery time between any two given nodes. Because Ethernet was originally a bus contention network by design, it was dismissed as arbitrary, non-deterministic and unsuitable for many Industrial applications. Due to the fact that CSMA/CD Ethernet may cause varying message transmission times due to MAC layer collision backoff algorithm retransmission timers and that possibility that excessive collisions could cause the message to be discarded at the MAC layer, Ethernet relied on higher layer protocols such as TCP or the application to retransmit the message. This was an appropriate obstacle for Ethernet to compete as a deterministic fieldbus with the established benchmark deterministic fieldbus. As Ethernet has evolved however, obstacles such as network access and bus contention have been removed. Since the advent of Ethernet switching introduced by Kalpana in 1995, and IEEE 802.3x full duplex standard, collisions and bus contention have been met with solutions. Operating in full duplex, any device on the Ethernet network can transmit and receive simultaneously at any time without the risk of collisions. In full duplex operation, the Ethernet CSMA/CD collision is not required and thus disabled. Comparing full duplex Ethernet with other deterministic fieldbus variables such as the amount of data, devices and distance in traditional token passing deterministic fieldbus, Ethernet offers several advantages. With regard to the number of devices, Ethernet has no practical limit on the number of end devices via IP subnet masking. For example, a class A network with 24 host bits and an 8 bit subnet mask offers over 16.7 million nodes, which, we can agree is an impracticable subnet size, though it remains a mathematical possibility. Also consider that with point-to-point message distribution each node is able to communicate directly with any other node therefore, the transport of a message query or response is not appreciably affected by the number of devices on the network because there is no sequential message distribution in switched Ethernet as there is in a token passing topology. Logical ring fieldbus using token passing, must send the token sequentially to each device and therefore the transport time increases as the number of devices increases. In the brief comparison above, Ethernet compares very favorably with the established fieldbus, however, there are also threats to Ethernet transport determinism. Next we will

examine the threats, and methods that can be employed to neutralize those threats in a properly articulated Industrial Ethernet network. In a proprietary fieldbus/token passing system, traffic is generally confined to specific message types and sequential circular flow between devices. However, in an Ethernet system, some of the messages which permit the flexibility of free form point-to-point communication require broadcasting to locate the resources necessary to build a complete message request. The Address Resolution Protocol (ARP) is a helper protocol which is used to bind the Ethernet hardware MAC address to the logical software stack IP address. As the two protocols were developed independently of each other and at different times, not in parallel, ARP has become a separate but core protocol for the proper function of IP over Ethernet as the combination of Ethernet and IP have gained acceptance. ARP requests, by design, are broadcast to all devices on the IP subnet or VLAN broadcast domain. However, ARP requests can be disruptive when they are excessive. Broadcast messages such as ARP, and others, are broadcast throughout the Ethernet IP Subnet and are received, then processed by all devices. Processing broadcast requests is an essential function of Ethernet over IP, even if the ARP request is for another end device. Many other common protocols such as NetBIOS or IPX also advertise services which can sometimes generate reciprocal broadcasts from all other NetBIOS hosts on the subnet, as in the case of Microsoft Windows NetBIOS domain or workgroup master browser elections initiatives. Or, a host configured for another network attempting to locate its primary resources. Misconfigured host PCs can be disruptive to some devices by generating 10 or more broadcasts per millisecond in a burst attempt to login/register with unavailable network domain controllers, server shares, and other resources. If ARP or other broadcasts are excessive, these broadcasts can be disruptive by congesting the buffers of all end devices on the subnet which delays or may prevent the routine processing of important Unicast/Multicast automation application messages and legitimate UDP client requests such as BootP/DHCP by servers. Ethernet switches have evolved to combat excessive broadcast traffic with a feature called Broadcast Rate Limiting, which clamps excessive broadcast traffic above the configured level. Choosing managed Industrial Ethernet switches which support Broadcast Rate Limiting allow the switches to protect the end devices from excessive broadcasts and ensures that any disruption from broadcast storms is minimal and unlikely to affect the Industrial application. A configuration guideline is to permit 2 general broadcasts per second on the switch port for each device on the subnet, plus 2 per second for each target device as a conservative number. This is derived from allowing 1 broadcast for each of 2 application services, such as DHCP, and derived from the IP standard broadcast interval of 1 broadcast per second. For example, if a device is communicating with 5 other devices on a subnet with 60 hosts, the Broadcast Rate Limit would be 130 broadcasts per second.

(Subnet devices x 2) + (number of targets x 2) = Rate Limit Consider that if power is restored after an outage, all devices would boot nearly simultaneously, and would be not only probing for duplicate address checks, but seeking to obtain address and configuration information and trying to locate their assigned peers. Broadcasts by clients are typically bound to 1 second intervals, though some devices may alternate frame types within that one second boundary. All clients will as well seek

the MAC addresses of their peers to collect the MAC address information needed to initiate a TCP connection. The Broadcast Rate Limit configured value must be sufficiently democratic to allow all devices to receive advertisements from all connecting clients on the subnet. While the example number of 130 broadcasts per second would be an extraordinary number of broadcasts, it would still not be disruptive to Ethernet. For example, 130 DHCP broadcasts of 512 bytes each would still only equal just over 0.5% utilization on a 100 Mbs Ethernet link. Multicasting is also a useful real time message distribution method but can be disruptive if flooded to all end devices on the subnet. In an unmanaged network, multicasts appear as equal to broadcasts. That is, they are flooded to all host devices, even those end devices that are not intended or required to receive those multicast messages. Multicast filtering is another configurable feature available in managed Industrial Ethernet switches that support GMRP or IGMP multicast filtering. Multicast filtering by the Ethernet switch allows the switch packet engine to distribute those multicasts only to those devices registered to receive them. This prevents flooding and consequently disruption of end devices by buffering unnecessary messages. By operating in full duplex mode and mitigating disruption from excessive broadcasts and multicasts, Ethernet is positioned for deterministic performance. Ethernet typically transmits at 100 Mbs on a switch with modern end devices. With packet sizes for automation protocols typically under 500 bytes, the transmission time to send that 500 byte packet is 0.00004 seconds or 40 microseconds. There are some other factors such as the Normal Velocity of Propagation (NVP), which is essentially the time to transmit a bit down a given length of media. NVP is measured as a percentage of the speed of light. Most Cat 5e cable has an NVP of 0.65-0.70 which will transmit that bit at up to 70% the speed of light. For all practical purposes, this component, even on a 100 Meter cable segment is 477 nanoseconds; so small, that it can be ignored. As noted earlier, there are some variables in traditional fieldbus which deliver deterministic performance, but does affect overall transmission time. For example on Modbus Plus, the number of devices will affect token rotation time. On Profibus, theoverall length of the network will affect the transmission speed. In either case, once the network is established and stabilized, the transmission time will be consistent. For an Ethernet network a similar variable is the number of Ethernet switches in the path between the two end devices. Most Ethernet switches operate in Store and Forward mode, which means that every packet must be completely buffered on the incoming or ingress port, error checked using the Frame Check Sequence (FCS is 32 bit CRC), and forwarded to the outgoing or egress port. This Forwarding Delay takes a small amount of time to complete. Most Ethernet switches have a maximum forwarding delay of under 50 microseconds. The maximum forward delay is generally tied to forwarding delay imposed by a maximum Ethernet frame of 1,518 bytes. As most Industrial automation messages are less than 1/3 of the maximum Ethernet frame size, the forwarding delay in most Industrial Ethernet switches is considerably less than the maximum forwarding delay specification of the switch. Testing performed in our lab has indicated that as more switches are added in the path between two end devices, the total forwarding delay increases in a linear fashion. The

forwarding delay maximum value of 50 microseconds is a worst case value for most switches. Using this value, these tests have shown that a packet would have to traverse 20 switches to equal a forwarding delay of 1 millisecond. Having 20 switches in the path between 2 end devices is not common in a bus or star architecture, although it is certainly possible as a Redundant Ethernet Ring architecture. Because the Redundant Ethernet Ring is a physical ring/logical bus, a device can be just a couple of physical switches away, but packets may have to traverse the backbone of the ring depending on where the bus is terminated. Generally, the physical ring terminates to a bus at the Ring Redundancy Manager switch. Even crossing many switches, and using a very generous forwarding delay factor of 50 microseconds, you can see that the overall transmission time is quite small in Ethernet. There are also Quality of Service/Class of Service traffic shaping technologies that help guarantee priority delivery of message packets, should congestion on the Ethernet backbone become a concern. Adding 3 User Priority bits to the IEEE 802.1p/Q Tag in the MAC frame can establish a priority level of 0-7 for each packet. Priority 7 packets will always be forwarded first and should congestion develop to the point where switches may be forced to drop packets due to buffer overload, only the lower priority packets will be dropped. How packet priority is handled depends on the implementation in the switch. Some switches use strict queuing, which means that if there is even a single high priority packet, it will be plucked from the buffer and forwarded first. If there are many high priority packets, all of them will be forwarded before a lower priority packet is forwarded. Other implementations use a Weighted or Random Early Detection queuing mechanism, where a majority of higher priority packets are forwarded, then a few of the lower priority packets. This is similar to queuing mechanisms used by IP routers. A router will look at the IP Type of Service (TOS) or DiffServ field and interleaves the frame with others depending on the Queuing mechanism configured. This way, all low priority packets. At the MAC layer, a similar feature is accomplished with Industrial Ethernet switches that support IEEE 802.1Q/p. While there are many methods for controlling broadcasts, multicasts and congestion, there is little chance on most Industrial Ethernet networks that this will actually become a problem because of the smaller automation application packet size. Smaller packets take less time to transmit and interleave more easily. In our testing, we transmitted a sample 71 Modbus TCP request packet including the Ethernet MAC overhead (inter-packet gap, preamble, FCS), sent through a series of Ethernet switches operating in full duplex. The results are shown in Figure 2. The same test was repeated with a 325 byte Modbus TCP response shown in Figure 3. As you can see below in Figures 2 and 3, as the number of switches in the path is increased, the transmission time increases correspondingly. Note however, that the actual transmission time, even through several switches, is quite a small value indeed.

This demonstrates that like the proprietary, deterministic fieldbus, once the switch path is established, transmission time reaches a consistent steady state, variable by perhaps a few microseconds. Test Setup The testing was conducted using a variety of managed and unmanaged Industrial Ethernet switches. The packet generator used was the Spirent Smartbits 200 as shown in Figure 4.

Modbus Request and Modbus Response packets were passed through a incremented number of switches on the egress interface and received on the ingress port of the SmartBits 200 to measure the Round Trip Time. Each packet had a nominal 96 bit time Inter Packet Gap (IPG) to simulate a stream of traffic. The elapsed time was measured using the system clock reference on the SmartBits 200. Because the purpose build SmartBits uses specialized ASICs to generate traffic, the stream of traffic was constant and not subject to operating system fluctuations found in software packet generators. Queuing Effect with Prioritization Even with broadcast rate limiting, multicast filtering and prioritization traffic shaping tools configured and in place, it is possible that a maximum size, low priority, Ethernet frame has started buffering at the switch ingress just ahead of a higher priority automation application message, as shown in Figure 5. The maximum Ethernet frame will continued to be buffered, and the automation application message packet will be forced to queue. This is perhaps a rare instance but nonetheless possible. In this scenario, the maximum queuing delay imposed on the automation application packet would be 121 microseconds on a 100 Mbs link. Not enough to disrupt an automation application and well within a reasonable tolerance for determinism.

Designing for Determinism The design of your network can play a role in maintaining a deterministic Ethernet network as well. As mentioned, when operating in full duplex, the only real threat to determinism is disruption by unnecessary protocols or excessive broadcasts/multicasts. If you are deploying a Distributed I/O (DIO) network over Ethernet and want truly deterministic performance, consider isolating the DIO devices on a dedicated PLC communications adapter and dedicated switch as shown in Figure 6.

In Figure 6, the PLC is equipped with 2 Ethernet interface adapters. One adapter services the Distributed I/O for determinism, while the second adapter services all other factory communications including peer PLC comms, SCADA, HMI, MES and other client devices. Having DIO devices on a dedicated Ethernet switch eliminates the propagation of unnecessary or disruptive protocols. Note that with fiber optic interfaces on modern Industrial Ethernet switches, there is no practical limit in terms of distance that other deterministic fieldbus are subject to. Using multi mode fiber media the DIO network could span 2 kilometers between each switch. Using single mode fiber optic media, distances of greater than 20 kilometers between each switch sufficient for virtually any industrial application. In comparison with Profibus, this is accomplished with no degradation of speed. The optional router shown in Figure 6 also allows direct access to the DIO devices from the Factory LAN in a controlled method. The router will not propagate broadcasts from the factory LAN and can be

configured for Access Control such that only authorized personnel may pass through the router to the DIO network. In conclusion, two approaches to Ethernet determinism have been outlined. The first is to employ full duplex operation end-to-end, and use configuration options in Industrial Ethernet switches to mitigate excessive broadcasts, unnecessary multicasts and prioritize automation application traffic. This allows determinism within 1 millisecond on nearly all automation applications. The second option is to use a dedicated DIO LAN for deterministic I/O control over Ethernet. With minimal additional cost, this option allows Ethernet to operate in a closed, controlled environment for deterministic operation. Because of these design and configuration options, Ethernet offers remarkable flexibility compared to other, traditionally deterministic field bus, without the restrictions and performance penalties. As costs decrease and offer selection increases, Ethernet is evolving into the fieldbus of choice. With a little planning and using a few Industrial Ethernet switch configuration features, Ethernet is ready to migrate into areas currently serviced by proprietary fieldbus.