Upload
others
View
15
Download
0
Embed Size (px)
Citation preview
You certainly know that SONET/SDH is an aging technology and that if your
operational network is still based on it then you will need to transition to a packet
switched network. Just as certainly, you know that the Internet Protocol (IP) as used
in IT networks defines packet formats and routing protocols, but does not define a full
networking solution for OT networks. You have probably heard that Carrier Ethernet
(CE) and Multiprotocol Label Switching (MPLS) are alternative solutions, but you may
be unsure which is more suitable for operational networks. You probably have not
seen any recent comparisons of CE and MPLS, and would like to understand the
advantages and disadvantages of each. If all of the above hold, then read on. And if
you want to brush up on packet switched technologies (IP, MPLS, and Ethernet), we
have prepared everything you need to know in an appendix.
A Comparison of
Carrier Ethernet and MPLSfor Critical Infrastructure Operational Networks
White Paper
Critical infrastructure networks, whether circuit or packet
switched, need to utilize guaranteed networking. This is in
contrast with best effort networking which may be good enough
for browsing the Internet and sending emails, but is decidedly
not so for mission critical traffic, such as that of connectivity of
power utilities.
Best effort networking means that there are no guarantees
that information will actually reach its destination, only that the
network will do its best to deliver it. Guaranteed networking
means that that there are a set of service characteristics
that need to be guaranteed, such as availability, Committed
Information Rate (CIR), restoration time, propagation delay, and
information loss rate. These guarantees are enforced in service
provider networks via Service Level Agreements (SLAs) between
the service provider and its customers, and in self-built networks
by pursuing Service Level Objectives (SLOs).
Networks that are guaranteed to meet Service Level
Objectives do so by a combination of:
• Determinism
• Resilience
• Monitoring and diagnostics
• Traffic conditioning
• Security
• and in some cases: timing support
Let’s look at each of these attributes in turn.
Guaranteed Networking
2
White Paper
Circuit switched networks, such as TDM, PDH, and SONET/SDH,
are necessarily deterministic, by which we mean that the path
taken by the information from source to destination is constant
and well known. Even if a fault occurs and protection switching
kicks in, the protection path has been pre-configured, and the
jump from the working path to the protection path is minimally
intrusive. Furthermore, in circuit switched networks bits always
arrive at a known constant rate.
For packet switched networks determinism means that all
packets belonging to the same application are handled the same
way – i.e., follow the same path through the network and are
processed in the same way by the network elements traversed.
Packets needn’t be sent or arrive at a constant rate, so that the
statistical multiplexing advantage is maintained.
In best effort networks packet forwarding paths are stitched
together via the local decisions of routers that learn network
topology using distributed routing protocols. While these local decisions can be influenced by policy, they are mostly up to the whims of individual routers, and indeed change over time. Deterministic packet delivery is accomplished
by having packet forwarding paths configured by Network
Management Systems (NMS) or by Software Defined Networking
(SDN) controllers. These systems are centrally located, maintain
a full view of the topology and usage of the entire network, can
exploit sophisticated algorithms to attain optimal efficiency, and
can be directly managed by human operators.
Determinism
3
White Paper
Note that for non-deterministic networks it is not possible to
reserve resources for a particular service in any given router, since
we can’t be sure that the packets will traverse this particular
router. On the other hand, deterministic networks can be traffic
engineered, that is, resources such as link bandwidth and
network element processing power can be reserved, thus closely
emulating circuit switched networking. Although less celebrated,
we put determinism at the top of our list since it is the heart of guaranteed networking. It is meaningless to speak about Service
Level Objectives if each packet can behave differently from its
predecessor. Similarly, if packets do not arrive or do so tardily,
how could one diagnose the problem if the path taken by the
packets is unknown and in fact indeterminate? Moreover, security
can never be guaranteed if packets are free to traverse arbitrary
and possibly untrusted network elements.
So, how do the various packet switched technologies stack up
regarding determinism?
Routed IP networks are not deterministic, so although modern
applications produce IP packets, mission critical applications
should never rely solely on IP forwarding mechanisms. For the
same reason best effort MPLS (sometimes called IP/MPLS, and
by which we mean any routed – non-traffic engineered flavor
of MPLS) should be ruled out, as it merely expedites the
forwarding of nondeterministic IP.
Mission critical networks can benefit from the determinism of
an underlying Carrier Ethernet or MPLS-TE network. Both are
typically configured by Network Management Systems (or more
recently by SDN).
Determinism
White Paper
4
The most important service characteristic is path continuity,
which takes up or down values. Obviously, the service objective is for continuity to be up 100% of the time, but faults are inevitable, and
can be tolerated if they are rapidly detected and rectified; an
attribute called resilience. We will discuss fault detection in
the next section, here we will consider rectification aspects.
In best effort networks, faults are eventually detected and
processed because affected network elements send status
indications to other network elements, and thus modifying
their forwarding information databases. While these
communications are ongoing packets may be misdirected and
become lost, but eventual consistency implies that eventually
correct packet delivery will be restored.
In guaranteed networking, resilience is attained by carefully
planning protection switching mechanisms, and properly
implementing protection switching protocols. Typical
objectives are under 50 milliseconds from fault detection
to service restoration, and reversion to the original
configuration once slower maintenance operations have
cleared the fault.
So, how do the various packet switched technologies stack
up regarding resilience?
IP networks do not provide protection switching, rather rely
on reconvergence of routing protocols to eventually restore
correct packet delivery (and discard black holed packet in the
meantime based on the TTL field).
Resilience
White Paper
5
Carrier Ethernet provides both linear and ring protection
mechanisms, defined in ITU-T Recommendations G.8031 and
G.8032, respectively. These mechanisms have been widely
implemented and deployed, and are interoperable amongst
all vendors.
MPLS networks, including otherwise best effort ones, may
provide resilience using a mechanism known as Fast ReRoute
(FRR), as described in IETF RFC 4090. Unlike protection
switching, FRR utilizes local detours to achieve very fast
service restoration, at the expense of some reduction in
determinism. MPLS-TP defines linear protection mechanisms
similar to those of Carrier Ethernet. The IETF defined RFC
6378 and the ITU-T standardized Recommendation G.8131.
Unfortunately, these standards are based on conflicting
principles and are not interoperable, and neither has seen
significant adoption. Both IETF and ITU-T worked on ring
protection for MPLS-TP, with the IETF’s RFC 6974 describing
how to achieve ring protection using linear protection
mechanisms. The ITU-T work did not reach fruition.
Resilience
White Paper
6
Guaranteeing communications in a network depends on
continuously monitoring the service level objectives. Even if
planning is carefully carried out, without monitoring one could
never be sure that the objectives are being met. Furthermore, if
some objective is found to be inadequate or deteriorating, a set of
diagnostic tools is needed to find the root cause of the problem.
The monitoring and diagnostic toolset is collectively called
Operations, Administration, and Maintenance (OAM). In circuit
switched networks OAM is added to the bit streams as overhead,
while in packet switched networks special OAM packets are
typically employed. OAM isn’t fully dependable for nondeterministic
packet switched networks, since the results depend on the packet-
forwarding path, which may arbitrarily vary.
Basic continuity is always essential for communications
of any type to succeed, while other specifications (such
as available information rate and propagation delay)
may or may not be important for a given application.
Hence, we conventionally differentiate between Fault Monitoring (FM) – the OAM mechanisms that monitor continuity and Performance Measurement (PM) – the OAM mechanisms for all the other
specifications. FM OAM may be used to trigger
protection switching, and to measure availability – the
percentage of up-time, with numbers such as “five nines”,
i.e., 99.999% availability, being common (five nines translates
to down time of about five and a half minutes per year). PM
OAM is conventionally employed to collect statistics for SLA/
SLO compliance reports.
Monitoring and Diagnostics
White Paper
7
So, how do the various packet switched technologies stack
up regarding OAM?
Pure IP is nondeterministic, and thus can’t fully support
monitoring. A protocol known as Bidirectional Forwarding
Detection (BFD) was developed to test continuity between
adjacent routers, and was later extended to the multi-hop
case. An extensive set of PM tools were produced by the IETF
IPPM working group.
Carrier Ethernet has an extensive OAM toolset, based on
ITU-T Recommendation Y.1731, IEEE 802.1 CFM, and MEF
Implementation Agreements. Extensive interoperability
testing and wide deployment ensures that this toolset is
mature. Furthermore, commissioning testing defined in
Y.1564 allows a service to be fully tested before deployment.
MPLS-TP was specifically developed to achieve OAM
functionality equivalent to that of Carrier Ethernet, and
hence only MPLS-TP has significant OAM capabilities.
Unfortunately, once again the two competing standards
organizations produced two non-interoperable versions of
OAM. The extent of deployment of either of these standards
is unclear.
Monitoring and Diagnostics
White Paper
8
In circuit switched networks, information sources have constant
bit rates, while in packet switched networks information sources
are free to send information, or not, as they see fit. Obviously,
objectives can’t be guaranteed if information sources start
sending at much higher than expected rates, as the required
physical resources would not be available. In such cases, packets
need to be deliberately discarded.
Best effort packet switched networks are often designed to
ensure fairness, meaning that all packets have equal chance of
being delivered. Alternately, packets may be prioritized, so that
lower priority packets are preferentially discarded.
Guaranteed networking employs a more disciplined approach.
When a communications service is configured, it is allocated
a Committed Information Rate (CIR). This does not mean
that the service continually consumes that rate as in circuit
switched networks, merely that the network is planned so
that this rate can be guaranteed. In service provider networks the CIR is directly linked to payment, i.e., the customer pays more for higher CIR. If a source exceeds its committed
information rate, the network defends itself by employing
traffic conditioning, i.e., it limits the information rate to
comply with the CIR.
There are several types of conditioning in use. Simple
policing colors packets green (i.e., allows them into the
network) if the suitably averaged traffic rate does not
exceed the rate for which the customer paid, and colors
them red (and discards them) if the rate is excessive.
More sophisticated policers can color packets yellow and
conditionally allow them to enter the network.
Traffic Conditioning
White Paper
9
An alternative to policing is shaping, whereby packets are
queued when the instantaneous data rate exceeds the CIR,
but afterwards allowed in if the suitably averaged rate is
acceptable. This averaging is conventionally carried out using
a token bucket algorithm.
So, how do the various packet switched technologies stack
up regarding traffic conditioning?
IP packets have DSCP field for indicating packet priority. It is
up to the network administrator to decide if this feature is
utilized.
Carrier Ethernet supports priority-dependent dual-token-
bucket policing and shaping to achieve traffic conditioning
and to protect the network from unfair exploitation.
All flavors of MPLS have a 3-bit Traffic Class (TC) field to
influence queuing or to indicate congestion, but its use is
not enforced by any standardized SLAs.
Of course, since MPLS packets may be transported by
Ethernet, one could envision using Carrier Ethernet
mechanisms to perform the traffic conditioning. However,
this begs the question as to why MPLS is needed in the first
place if Carrier Ethernet is employed.
Traffic Conditioning
White Paper
10
By now, it is well known that network infrastructures can be
hacked in order to illicitly provide information or service, or to
deny information or service from legitimate users. The principal
objectives of network security mechanisms are authorization,
establishing security associations, authentication, and
confidentiality. Authorization means ensuring that a legitimate
user (but only a legitimate user) gains access to its information
or services. Authentication means verifying that a packet claiming
to be from a particular source is indeed from that source.
Confidentiality means denying access to information from those
who should not obtain access.
Sophisticated attackers can compromise both circuit switched and packet switched networks. Circuit switched networks are
often considered secure, but that is merely because their basic
technologies are less understood by the less sophisticated
hacker. Packet switched networks may suffer from many more
attack vectors than circuit switched ones, but have also received
much more attention from the security industry.
So, how do the various packet switched technologies stack up
regarding security?
The IP suite, which was originally designed without security in
mind, has been retrofitted with a number of security features.
IPsec is the fundamental network-layer mechanism, and offers
both user-to-user and tunneling modes. Internet Key Exchange
(IKE) handles authorization for IPsec. The IP suite also contains
security mechanisms at higher layers, including SSH, TLS, and
HTTPS. IP routing protocols have limited security features, but
may themselves be protected using the above.
Security
White Paper
11
Similar to IPsec Carrier Ethernet defines MACsec (802.1AE)
for authentication and confidentiality, and a mechanism for
authorization and establishing security associations called
802.1X. Ethernet management communications have their own
security mechanisms, such as those defined in SNMPv3.
MPLS was designed for core networks, under the assumption
that such networks are walled gardens, meaning that although
defense could be applied at the perimeter, no attacks were
envisioned from inside the network.
For this reason, MPLS packets, of any flavor, forgo source
identifiers. Without a source identifier there is no possibility
of authenticating a packet as coming from any given source.
Furthermore, a MPLS packet has no real destination address -
merely a short label. Hence it is relatively simple for an insider
to fabricate a packet that is technically legitimate. Finally, MPLS
defines no security mechanism of its own, so that confidentiality
can’t be enforced.
Furthermore, the MPLS control plane employs soft state
protocols. A soft-state protocol is one that requires periodic
keep-alive messages to maintain state. Thus by maliciously
discarding a few packets, an attacker can lead to massive
denial of service.
Of course, since MPLS is a thin layer sitting between
Ethernet and IP, an MPLS packet could avail itself of Ethernet
and/or IP security mechanisms. There are two problems
with this approach. First, according to modern thinking,
every network layer requires security mechanisms. Note
that although IPsec may be used, a denial of service attack
at the MPLS layer will still disrupt service. Second, although
multiple layers may be employed, the important question is
what functionality is deployed at each such layer. If a so-
called MPLS network uses mostly non-MPLS mechanisms,
the question is whether it is an MPLS network at all, and for
what purpose the MPLS label was inserted.
Security
White Paper
12
Many modern applications require some level of timing/synchronization
support, with the stringency of the requirements increasing over time.
For power utilities, teleprotection systems may require accuracies on
the order of microseconds, modern synchrophasors need to maintain
1 microsecond time accuracy even when they lose GPS reception, and
smart grid applications have been targeting 1 microsecond absolute
time accuracy as well. The time accuracies required by other critical
infrastructures, such as water distribution networks, railway and
maritime transportation networks and Air Traffic Control systems, are
becoming more stringent as well.
When communicating using synchronous circuit switched technologies,
the network elements themselves must maintain highly accurate
frequency lock in order to extract bits from the received bit-stream.
This inherent mechanism of frequency distribution can be used to
provide high accuracy frequency information to applications that
need it, or to maintain highly accurate time of day for applications
that require time or phase information. Transferring highly accurate frequency and time of day information over asynchronous packet switched networks is much more challenging.The Network Time Protocol used in IP networks can attain accuracies
of 1 millisecond on small networks, and tens of milliseconds on larger
ones. The Precision Time Protocol, known by its moniker IEEE 1588,
uses on-path support to attain much higher accuracies, typically sub-
microsecond.
So, how do the various packet switched technologies stack up
regarding timing?
As we have mentioned, IP supports NTP for low accuracy timing needs.
Timing Support
White Paper
13
Since Ethernet (unlike MPLS and IP) defines a physical layer,
and for high-rate point-to-point links this physical layer is
constant bit rate, all that is required to support distribution of
frequency to user applications is to lock the source frequency
to a suitable frequency reference and to ensure traceability of
delivered frequency to this source. This technique is known as
Synchronous Ethernet or SyncE.
Carrier Ethernet can avail itself of SyncE and IEEE 1588 to
distribute highly accurate frequency and time information to
applications that need it. No similar mechanisms are available
for MPLS (of any flavor), simply because MPLS does not provide
a physical layer, and MPLS standards do not define the requisite
on-path support elements.
Of course, when MPLS is transported over Ethernet, the
underlying Ethernet layer may be used for timing distribution,
as long as the MPLS mechanisms do not impede the required
functions. However, MPLS routers do not universally support
SyncE functions, and MPLS forwarding may decide to forward
packets along paths without on-path support, thus disrupting
the Ethernet timing functionalities. On the other hand, if the
Ethernet layer is given free rein in forwarding decisions, then, as
before, the MPLS layer serves no purpose.
Timing Support
White Paper
14
We have reviewed the requirements for guaranteed networking,
and the fundamental behavior of the various types of IP,
Ethernet, and MPLS networks. We inevitably reach the conclusion
that no version of IP is by itself satisfactory for mission critical
applications. On the other hand, Carrier Ethernet networks fulfil
all of our requirements, and MPLS can fulfil most of them if we
use the proper combination of flavors.
One needs to be careful regarding MPLS, as vendors and service
providers are often unclear as to whether they are deploying
best-effort routed MPLS or completely deterministic traffic
engineered MPLS. Even in the latter case, the MPLS-TP’s OAM and
resilience features are not widely deployed, and their absence
should be seen as a red flag for any mission critical network.
The most significant differences between Carrier Ethernet and
deterministic and monitored MPLS remain the timing and security
aspects. No form of MPLS can provide physical layer frequency
support (as in SyncE) and any time distribution protocol will be
either proprietary or at a non-MPLS layer (either below at the
Ethernet layer or above at the IP layer). Regarding security, MPLS
was and remains a protocol for unattackable walled-garden
networks. While mechanisms can be deployed at IP and Ethernet
layers to overcome some portion of the attack spectrum, a full
security solution for MPLS remains elusive.
We summarize our conclusions in the following table:
Wrap-Up
White Paper
Pure IP
Carrier Ethernet
Best Effort MPLS
MPLS-TE without TP
MPLS-TP without TE
TE+TP
Determinism Resilience Monitoring and diagnostics
Traffic conditioning Security Timing
support
Partial compliance Full compliance No compliance
The driving force behind the transition from circuit switched
networks to packet switched ones is statistical multiplexing.
Networks based on E1s or T1s or SONET/SDH reserve bandwidth
for the worst-case messaging rate, and transmit idle patterns
when there is no traffic to send. Thus, when PDH or SONET/
SDH multiplex together multiple information sources, each
source always occupies its maximum information rate. On
the other hand packet switched networks are not required to
reserve resources (bandwidth, switching capability, memory).
If there are no messages from one source of information,
then network resources are automatically available to carry
information from another source. This characteristic, known as
statistical multiplexing, makes providing service guarantees more
challenging.
The advantage of statistical multiplexing may not seem very
important for a conventional massively overprovisioned OT
network, but the savings are huge in most modern applications,
and hence packet switched network technologies have replaced
TDM-based networks except for in the lowest layers of high-
data-rate transport (such as Optical Transport Networks – OTN).
Hence, if you are going to build a new network, or completely
upgrade an existing one, you are almost certainly going to deploy
a packet switched network. However, you still have considerable
leeway, as there are three different packet switched network
technologies in common use, and each of these has several
flavors that need to be understood before deciding on which
suits your needs.
APPENDIX: Packet Switched Networks
White Paper
16
The first technology is pure IP which comes in two flavors:
IPv4 and IPv6. The second is Ethernet, which comes in three
flavors: basic Ethernet, Carrier Ethernet (CE; which itself has two
categories, namely CE-1.0 and CE-2.0), and Industrial Ethernet.
The third is MPLS (Multiprotocol Label Switching), which presently
comes in four flavors: best effort MPLS, MPLS for L3VPN services,
MPLS-TE, and MPLS-TP.
These three technologies are often used in conjunction, in fact,
a single packet can be simultaneously IP, MPLS and Ethernet. The
essential issue is which functionalities are performed by which
technology. For example, most modern applications utilize IP as
their basic packet format, but this IP packet may be encapsulated
in Ethernet to deliver it from its source to the first router, and
may then be encapsulated in MPLS to be transported from that
router to the next router and the next, and finally encapsulated in
Ethernet for the last hop to its destination.
The next three sections provide a brief overview of each of the
three packet switched network technologies.
IPIP, or Internet Protocol, is a large suite of protocols used by both
the public Internet and in many private networks. The original
version, called IP version 4 (IPv4), is constrained by having far
too few addresses for the public Internet (only about 4.3 billion
devices can be simultaneously connected) and is thus being
replaced by IP version 6 (IPv6).
The IP suite is not a complete protocol stack, starting above
the second layer of the OSI 7-layer model, and thus requires
physical layer and link layer protocols to transport it (for example
– Ethernet). On the other hand the IP suite is highly developed
in many ways, for example, it has optional delivery reliability
mechanisms (e.g., TCP), security features (IPsec, SSH, and TLS),
APPENDIX: Packet Switched Networks
White Paper
17
and support for many applications (e.g., web browsing, email,
and video streaming).
Anyone designing a critical infrastructure network should be
aware that IP is a nondeterministic best effort technology. For
this reason, a higher layer protocol (usually TCP) must be used to
retransmit packets that were lost along the way. This, of course,
introduces additional delay and delay variation.
Ethernet and Carrier Ethernet Ethernet is debatably the most successful packet switched
technology, with hundreds of millions of ports in use. Today
Ethernet handles data rates from 2 Mbps to 100 Gbps, travels
over DSL, dedicated copper wiring, fiber, and radio. Although
it started as a broadcast-domain technology with a specific
congestion avoidance mechanism, high-speed today’s Ethernet
based on full-duplex links connected by switches can support
arbitrary topologies.
Having started in local area networks operated by the customer,
basic Ethernet provides absolutely no service assurances and
is decidedly not carrier-grade. However, Ethernet has been
upgraded to “Carrier Ethernet” by adding mechanisms such as
network management, OAM and QoS, protection switching,
timing, and security. Carrier Ethernet is today a multi-billion dollar
market with huge-scale deployments by all major communications
service providers.
On the other hand, Ethernet has also been adapted to industrial
automation applications by supplementing it with ruggedized
connectors and extended temperature switches, as well as
protocols that provide determinism, resilience and real-time
control. Industrial Ethernet has also found use in power utilities
substations.
APPENDIX: Packet Switched Networks
White Paper
18
MPLSMPLS is a thin layer, most often residing between the Ethernet
and IP layers, originally designed to accelerate the forwarding
of IP packets in large networks. MPLS packets do not contain
addresses, but instead a stack of labels that serve to locally
direct forwarding.
MPLS now has four distinguishable flavors (differing in control
protocols):
1. Best effort MPLS (sometimes called IP/MPLS), as used in the
core of the Internet, is a best effort technology which relies
on the IP routing system, the MPLS layer serving solely to
accelerate forwarding;
2. MPLS for L3VPN services is a popular offering to deliver VPN
services to businesses by providing service isolation;
3. MPLS-TE is MPLS augmented with traffic engineering and
resource reservation, and is used when true service level
guarantees are required;
4. MPLS-TP supplements MPLS with network management, OAM
and QoS, and protection switching, in order to offer a drop-in
replacement for Carrier Ethernet. MPLS-TP forwarding may be
determined by IP routing or may be entirely configured by a
centralized management system. MPLS-TP functionality may
be combined with MPLS-TE.
APPENDIX: Packet Switched Networks
White Paper
Specifications are subject to change without prior notification. The RAD name, logo and logotype are registered trademarks of RAD Data Communications Ltd. ©2017 RAD Data Communications. All rights reserved. Version 6/17
Your Network’s Edge
For more information visit www.rad.com