A LOGICAL NETWORK END-GAME - 1-4-5.netdmm/NGN/End-Game_External_2_01.doc · Web viewSequence Numbers have been included in the control-word used in the various ... SDH Synchronous

Neil Harrison Version 0.1 External 2 20/03/03

AN ALTERNATIVE NETWORK SERVICES END-AN ALTERNATIVE NETWORK SERVICES END-GAMEGAME

Neil HarrisonBT/Ignite Carrier & Networks/CTO/MPLS & GMPLS technologyEmail: [email protected]

1 THE MAJOR NETWORKING PROBLEMS THAT MUST BE SOLVED.......................................2

1.1 FUNCTIONAL CONVERGENCE.......................................................................................................21.2 FAULT-MANAGEMENT OF MPLS...................................................................................................4

1.2.1 Does MPLS use a cnls or co pkt-sw forwarding mode?...................................................41.2.2 The effect of topology on fault-management complexity..................................................61.2.3 Other MPLS architectural oddities....................................................................................71.2.4 The IETF solution to MPLS fault-management (LSP-Ping)............................................10

1.3 HOW TO GUARANTEE QOS AND CHARGE FOR IT.........................................................................111.3.1 Public services case....the general QoS problem...........................................................121.3.2 Private services case....a specific QoS problem.............................................................131.3.3 Private services solutions...............................................................................................141.3.4 The general public services solution..............................................................................14

DOCUMENT HISTORY....................................................................................................................... 17

GLOSSARY......................................................................................................................................... 17

REFERENCES.................................................................................................................................... 18

Page 1 of 18/tt/file_convert/5abaa81e7f8b9ab1118c1ba4/document.doc


1 The major networking problems that must be solvedOperators face many networking problems, but there are two that are of fundamental importance. These simply have to be addressed and sensible solutions found to them. They are: Functional Convergence......or to put this another way: ‘how to avoid endless

‘technology=service’ stovepipes and/or complex/costly user-plane and control-plane gateways’. How to guarantee QoS and charge for it with IP.......this also factors-in what will happen to the

PSTN and the impact of increasing BB penetration.

When I look at the current network solutions that are being proffered mainly by router vendors in the IETF, it is very clear to me that these problems will not get solved satisfactorily. In fact, it is more likely that they will get worse with the ‘cost of ownership’ invariably pushed on the operators. So let’s have a closer look at these two problems.

1.1 Functional Convergence

I have previously explored this topic in detail in [1]. However, here is a summary of the main points:

1 A key observation is that we need all three networking modes - that is: cnls, co pkt-sw and co cct-sw. We cannot deliver all the products/services required by our customers via one mode alone. All technologies map to 1 of these modes, eg cnls {IP, IPX}, co pkt-sw {FR, ATM, MPLS}, co cct-sw {nx64k, PDH, SDH, OTN}.

2 There are common functional components that networks require, and without which they won't scale/work. Some obvious functional components are addressing, routing, signalling, OAM, QoS metrics and traffic descriptors. There can be more than one *user-plane* network per technology depending on the degree of logical/physical separation between traffic and control (and/or management) user-plane networks. For example, in a cnls mode everything is congruent and goes via a common user-plane network, but as we move towards the ‘duct’ through co pkt-sw and co cct-sw technologies, there is increasing logical/physical separation. Path holding times must increase as we move towards the duct (many reasons, see [Error: Reference source not found]), and we have many legacy client/server relationships that we have to carefully consider in any migration/convergence solutions.

3 There are some people who believe certain functions (especially addressing and routing) can be crunched across all layers. These are mainly those pushing GMPLS in IETF and various start-up vendors (especially those working on OTNs). We *do not* agree with this view for many sound commercial and technical reasons (see [Error: Reference source not found]).

4 However, we do believe in function convergence for all technologies that belong to the *same* networking mode; the most difficult/contentious mode here is co pkt-sw. To date, all standards bodies (eg ITU, IETF) and industry forums that have sprung-up (eg FRF, MPLSF, ATMF) have specified functions in isolation. This tends to breed technology religion/competition, as each set

1 “Strategic Network Architecture Development Based on Functional Convergence”. Neil Harrison. 16 September 2002.



of technology champions claim that ‘my technology is better than yours’. In turn, this creates a problem/cost for operators in any future interworking/migration. The effect of this is either (i) to foster the familiar stove-pipe set of disjoint 'technology=service' solutions, or (ii) require complex/costly (if indeed possible) gateways to interwork the functional components of both the user-plane and the control-plane. However, the problem here is not really the technology per se (ie the user-plane PDU structure) but the fact that the functional components are very different even though they are addressing the same functional requirement.{Aside – As this paper was being written there has been an announcement that the FRF and MPLSF are to merge. This is good first step in recognizing our industry needs to stop re-inventing the (functional) wheel for technologies that belong to the same networking mode. However, this is not sufficient. ATMF must be included in such a merger}.

5 Besides the obvious capex/opex savings that would accrue from insisting on functional convergence of technologies belonging to a common networking mode, there are also benefits in reducing the uncertainty of product forecasts on network build costs. That is, if we offer several network products that have a specific characteristic access presentation but can be carried in a common way across the network core (ie we have some form of edge interworking function), then this will reduce the overall variance of the forecast uncertainty across the common mode product set. For example, being able to offer native FR, ATM and MPLS access presentations and using MPLS also as the common core technology for the co pkt-sw networking mode.

The above sets a general strategic network architecture philosophy we should work towards. For example, if MPLS is to become the common co pkt-sw mode technology (and all vendors do indeed show this) it must be ‘carrier-strength’ such that we feel confident we can migrate existing technologies like ATM and FR on to it.

The first step here is to be able to fault-manage MPLS properly. This is precisely why we need Y.1710 (requirements/principles) [2] and Y.1711 (OAM mechanisms) [3]. Although these Recommendations are targeted at MPLS, the approach taken is generic and could be applied to any co pkt-sw technology, ie they were written with the principle of functional convergence in mind.

The next step is to get sensible service-interworking standards for MPLS/ATM, and more general network-interworking standards for XoverMPLS.

However, MPLS is different things to different people and comes in several different flavours. Some types of MPLS are good whilst others are architecturally wrong and/or have poor or missing functional components.

MPLS appears superficially to be a simple network technology that looks a bit like a cross between FR and ATM. In many respects this is true. However, this superficial resemblance hides a significant degree of complexity and problems that only become apparent when one gets deeper into the protocols themselves, considers the implications of the topological constructs that can be created, and understands the consequences of using MPLS for specific applications, eg L3 VPNs or XoverMPLS.



In what follows I have had to assume some level of understanding of what MPLS is, how/why it came into being and the various flavours of MPLS. For those requiring a grounding in MPLS please see [4].

1.2 Fault-management of MPLS1.2.1 Does MPLS use a cnls or co pkt-sw forwarding mode?

Some argue that MPLS is based on a cnls forwarding paradigm. This is simply not true as MPLS uses a co pkt-sw forwarding paradigm. This is not just an issue of ‘academic interest’, since it has important implications wrt to the topological constructs that make sense, and especially the fault-management and QoS/traffic-management of such constructs. We will now briefly prove that MPLS uses a co pkt-sw forwarding paradigm and contrast the defect behaviour of such a mode with a cnls mode.

{Aside - I am going to home-in on the *essential* characteristic that differentiates, and thus defines, these modes. I am not going to consider the larger range of possible secondary characteristics that could be used (eg reliable transfer, use of signalling, etc), because when one considers all the possible characteristics that *could* be used to characterise these modes, then one sees all shades of grey and no black or white definitions of either 'pure co' or 'pure cnls'. For an example of a good discussion of what I mean here wrt secondary characteristics see Chapter 5 of 'Interconnections' by Radia Perlman.}

1.2.1.1 CNLS case

In a cnls mode each packet *must* contain all the information required for it to be routed correctly at *any* network node. In other words, one can pick-up a packet from any point in a cnls network and drop it into some other point in that network and it will find its way to the correct target destination (assuming no TTL exhaust). Some people refer to this property of cnls networks as ‘memoryless routing’. This is achieved by ensuring each cnls packet carries the full network-unique addresses (both source and destination, and although it is only the latter that defines unicast routing the former is required to know ‘who’ sent the packet). The fact that each/every cnls packet contains the full network-unique addresses and thus can find its way to the right destination from any location in the network is the essential defining characteristic of the cnls mode.

1.2.1.2 CO case

In a co pkt-sw mode only the network access points possess the network-unique address semantic. The links between nodes use link-connection identifiers, which are only required to be unique for the link between a pair of adjacent nodes. These link-connection identifiers only possess networking currency when they are allocated to a parent connection entity at it's creation time. Similarly, the link-connection identifiers lose their networking currency when their parent connection entity is destroyed.

A specific link-connection identifier can be used: (i) on different links in the same connection, or (ii) on any links in other connections. This means one cannot take a packet out of a given



connection at any point and drop it into the network at some other point and expect it to find it's way to the correct destination. There are 2 possible outcomes from doing this:

1 The packet will black-hole at the next receiving node if the link-connection identifier has no currency at that node, ie it is not allocated to some parent connection; or,

2 The packet will be forwarded at the next receiving node if the link-connection identifier has currency at that node (and thereafter forwarded on all subsequent links to the connection sink point), ie it is allocated to some parent connection. Note also that this case can be *recursive* if there are nested client/server layer network relationships....so the black-holing or further forwarding behaviour can continue upwards in the affected client connections.

Any failures that result in packets being misdirected raise traffic integrity/security concerns. So not only must such defects be automatically detectable, but they must also automatically elicit the right type of consequent actions, eg in this case squelch the traffic.

So the essential defining characteristic of the co pkt-sw mode is that it uses link-connection identifiers that are only link-unique, and unlike the cnls mode one cannot pick up a co pkt-sw packet from one connection and drop it anywhere in the network and expect it find its way to the right destination.

ATM, FR and MPLS technologies all use link-connection identifiers....VPI/VCI, DLCI and labels respectively.

Based on the preceding arguments it is obvious that *all* methods of instantiating MPLS LSPs result in a co pkt-sw networking mode in its most fundamental sense. Even LDP. LDP per se is a form of signalling (noting that signalling is often cited as a *secondary* defining characteristic of a co mode, but this is not the essential defining characteristic). The fact that mp2p connections are created in LDP and can vary their connectivity at will under the influence of an IGP does *not* mean this yields a cnls mode. To make this point crystal clear note that: one can run a cnls network without an IGP, or one can run a co pkt-sw mode with an IGP (eg PNNI is specified to allow a distributed routing

protocol to be used) and flex the connectivity at will if one wants, or one can even run a co cct-sw mode with an IGP (eg GMPLS) and flex the connectivity at will if

one wants.

Now whether doing some of these things is sensible or not is not the issue: The point being made here is that having or not-having an IGP tells one nothing about the nature of the network mode one is dealing with.

Given that it is clear that MPLS uses a forwarding mode based on link-unique-only identifiers, and thus has the essential defining characteristic of a co pkt-sw technology, this has important ramifications for it's fault-management. By definition it is subject to defects that *cannot* occur in a cnls mode, eg swapped LSPs and mismerged LSPs. This observation is critically important when MPLS is carrying arbitrary clients, ie XoverMPLS, since some client layer networks (and the



operators running such networks, who may be external customers of the MPLS network provider) will demand a certain fault-handling behaviour from the MPLS server layer network.

1.2.2 The effect of topology on fault-management complexity

A cnls network requires no a priori connections before traffic is sent. Thus ‘any-to-any ad hoc connectivity’ is what cnls modes do by definition. Co networks on the other hand do require the a priori establishment of connections before traffic can be sent.

As we noted in the previous section, the co pkt-sw mode is subject to failure modes that simply cannot occur in a cnls mode. So besides the obvious simple break failures (either arising below or within the MPLS network), we can also get swapped LSPs and mismerged LSPs (where the offending traffic may or may not be affected itself).

Now the ease with which one can deal with all these failure modes is a function of what sort of connection topologies are allowed to be constructed. All the major forwarding defects are very easy to detect/handle correctly in p2p constructs. For further details of the fault-management of MPLS p2p LSPs, including the measurement of the dependent near-end and far-end availability metrics, please see the ITU Recommendations Y.1710 [Error: Reference source not found] and Y.1711 [Error: Reference source not found].

The only topological constructions that make any sense in a co pkt-sw mode are p2p and p2mp. mp2p or mp2mp topological constructs however make no sense. The reason for this is that p2p and p2mp constructions preserve the identify of the *source*, whereas mp2p or mp2mp constructions do not since they ‘merge’ traffic.

Merging is an extremely bizarre thing to do in a co pkt-sw forwarding mode. For one thing it demands we also must have some method of 'demerging' so that we can identify the source. For example consider the two applications of L3 rfc2547 VPNs and the various Martini drafts for XoverMPLS when they are used with the MPLS mode of LDP DU (which creates mp2p constructs):

In the L3 VPN case the demerging happens in 2 stages. Firstly there are p2p 1-hop LSPs that are always installed above the mp2p constructs (these are created automatically by BGP4). These however only resolve to the VPN level and not the specific PE source, ie a given PE issues the *same* VPN-identifying label to all other PEs that are part of the same VPN. When this p2p label is popped in the PE you now need to use the IPv4 addresses that are exposed for onward forwarding within the VPN, and since IP packets contain both source/destination network-unique addresses then full resolution (of the source) is obtained.

In the XoverMPLS case (Martini), where X can be any client of the MPLS layer network (eg FR, ATM), we again see that 1-hop p2p LSPs are created above the mp2p LSPs to facilitate demerging. In this case however, the 1-hop p2p label issued by a given PE must carry the semantics of source resolution wrt to a given PE destination point, ie it has to uniquely identify the source directly here. Note also that targeted LDP is used here to signal the 1-hop p2p LSPs and not BGP4 as in the rfc2547 case.



So one thing we can immediately note is that any mp2p constructions must *always* have p2p constructions above them. This requirement is a mandatory consequence of merging in a co pkt-sw mode. However, you will not find this fact explained in any of the literature on MPLS.

{Aside: One often hears the claim that MPLS solves the so-called N2/2 meshing problem of traditional L2 technologies like ATM and FR when used to provide VPNs. In one sense this is true but in another sense it is not true. There are actually two N2/2 meshing problems: (i) routing adjacencies and (ii) forwarding connectivity.

It is BGP4 that solves the routing adjacency problem since it acts as a ‘common isolator’ in the VPN IGPs, ie CEs only peer with PEs and there are no CE to CE direct routing peerings. This is actually the N2/2 problem that is of most concern to the router vendors.

BGP4 also sort-of solves the forwarding connectivity problem in that it *automatically* creates the full-mesh of p2p LSPs that are required.......but note that they are still required. If A wants to talk to B then *something* has to connect A and B in the forwarding sense, this is simply unavoidable.

In fact, since core VPN topologies are generally long-holding with low churn, then what one really wants here (even from the traditional L2 case) is some means to automatically configure the required p2p connectivity.....BGP4 does this automatic configuration in the case of rfc2547 VPNs. Further, because VPN topologies are indeed long-holding one needs to ensure that the user-plane topology is not affected by any failures of (any of the) control-plane protocols. Putting this observation in a different and more general way.....a design goal should be to remove common-mode failure mechanisms.}

Unfortunately merging creates almost intractable fault-management and QoS/traffic-management problems. It’s very hard to tell when things go wrong in mp2p constructs, eg how much of a mp2p tree has to be broken to say its all broken? Further, in a co pkt-sw forwarding mode 'breaks' are not the only defects that have to be considered. We have to also consider swapped LSPs and mismerged LSPs, eg packets leaking from one LSP into another LSP. It’s also not just a problem of *detecting* these defects, we also need to take appropriate consequent actions. For example informing higher layer clients (eg ATM say when considering the Martini stuff) that the defect is in a lower layer and not to raise alarms, and squelching traffic if swapped or mismerging defects are detected (as this could lead to a VPN integrity/security compromise). These requirements are very important.

If we can’t easily define and detect/handle defects in a mp2p construct then there is no chance of defining meaningful/measurable availability SLAs. And if we can’t do this then there is also no chance of defining meaningful/measurable QoS SLAs.

1.2.3 Other MPLS architectural oddities

The fault-detection/handling problems of mp2p constructs are exacerbated when ECMP, PHP and TTL-copying between layer networks are considered. MPLS also allows variable LSP source identifiers to be used whose format is dependent on which signalling protocol was used to create the LSP. Whilst not a major problem in its own right, the user-plane source identifiers of LSPs



really ought to be the same irrespective of how instantiated. Finally, there is a further architectural oddity in the case of XoverMPLS where (in the various so-called Martini drafts) Sequence Numbers have been added to detect misordered (or lost) packets.

So let’s briefly explain what these are and their implications for fault-detection/handling.

ECMP (Equal Cost Multi-Path) is a proprietary means of load sharing based on some hash function of the traffic label stack (and maybe the payload, eg IP SA/DA). The objective of ECMP is to always send traffic from a given same end-system/application source over the same load-balancing leg in order to avoid packet misordering of a given traffic flow. The reason ECMP is required is because LDP LSPs track the IGP SPF routing, and so one can get focussed overload on some routes whilst having a low/zero load on other routes. If one of the load-balancing legs fail however, but other load-balancing legs are working fine say, then you can get the situation that the traffic from some locations is working perfectly whilst other traffic sources are experiencing a complete break. ECMP therefore creates a further level of fault-management problems that have to be addressed when LDP is used. This is not a trivial issue to address as we will see when we later consider the IETF solution for this known as ‘LSP-Ping’.

PHP (Penultimate Hop Popping) removes the label 1-hop prior to the true end node of the LSP. It was introduced by some vendors to avoid having to do multiple label look-up/manipulation at the end node of an LSP. So it’s basically a cost-saving exercise by those vendors. Sadly this makes the trail termination point a movable feast. Some functions, like defect detection and consequent actions, need to be applied at the trail termination point in a unidirectional sense. PHP makes this impossible. Note also that PHP also raises security concerns since it allows spoofing.

TTL-copying between layers. TTL is a technique used in cnls networks to mitigate against the damage caused by routing loops. TTL does not stop loops forming, it simply limits the damage they can do when they occur. It is not an essential requirement in a co pkt-sw technology, as there are other ways to prevent routing loops using signalling techniques. TTL has been specified for MPLS by IETF however. In MPLS there are two models for TTL handling in nested client/server LSP relationships: the so-called ‘uniform’ model and the ‘pipe’ model. The uniform model copies the TTL from the client (IP or LSP) to the server LSP at the server ingress, and copies the TTL back to the client at the server egress. And all nodes that are sequentially passed decrement the TTL. In the pipe model the TTL of each LSP, whether client or server, is treated independently and there is no copying of the TTL between them. So TTL decrementing in the pipe model is done independently in each nested LSP. The pipe model accurately reflects the required client/server layered network relationships defined in G.805. However, even though the pipe model is the architecturally correct mode, it causes problems for the trace-route mode of LSP-Ping as will be discussed later.

LSP source identifiers. The IETF have allowed the source identifier of an LSP (which is essentially the LSP access-point address) to be a function of the signalling protocol that created it. So, for example, an LDP signalled LSP uses a different source identifier to an RSVP-TE signalled LSP. This is an architecturally strange thing to have done as the source identifier of an LSP is only a function of the user-plane and it should be consistent irrespective of which



signalling protocol created it (including none if statically provisioned). This flaw has not been made in Y.1711 where the LSP source identifier is denoted by the 20 octet TTSI (Trail Termination Source Identifier) and this is specifically stated as decoupled from the control-plane (this was previously defined as a requirement in Y.1710). In Y.1711 a CV flow (this is a periodic/deterministic ‘Connectivity Verification’ OAM packet) carries the TTSI, and so it is trivially easy to detect and identify the offending source of any swapped/mismerged LSP defects (in addition to detecting the simple ‘break’ defect).

Sequence Numbers have been included in the control-word used in the various XoverMPLS Martini drafts that are being produced in the PWE3 WG. The stated rationale is to detect (and maybe correct?) out-of-order delivered MPLS packets. MPLS networks (like any co pkt-sw forwarding mode) should never, under normal operation, misorder packets. However, it is known that one router vendor has an implementation that does misorder packets when operating normally.

However, there is a deeper concern here: We require standards that specify client/server layer network relationships which are functionally independent, ie both the client layer network and the server layer network must be functionally self-sufficient and decoupled. This is an obvious requirement to allow flexible/arbitrary client/server relationships to exist, and to allow layer network technologies to be developed/sunsetted independently.

However, PWE3 are on the road to creating new layer networks that lie somewhere between the client (X) and the server (MPLS) layer networks. The client/server relationship here should just be simple adaptation (see G.805), but if functions like Sequence Numbers are added then these will requirement management. There has also been recent discussion on the PWE3 list of adding a congestion management capability between the client and the server layers and this too will need management. Further, a large router vendor is intending to propose adding OAM functions to the in-between layer to manage the ‘pseudo-wire’.....this one is particularly bad from an architectural perspective.

Adding such functions will make matters significantly worse. The right answer is to be able to manage the client layer network and the server layer network independently. So if there are functional deficiencies in either the client or server layer technologies they should be addressed in those layers directly, and not in a kludged ‘in-between’ layer.

Just considering the Sequence Numbers, if these stay then they really need properly specifying in terms of some type of defect state (entry/exit criteria and consequent actions). However, IETF have purposely left this open-ended and up to the vendor to interpret their use. Two specific technical points to note here are as follows:- the Sequence Number field is 16 bits. This should not wrap at the highest packet-rate

envisaged before the unavailable state is declared (because QoS metrics are still valid for collection until then); or, as a minimum, until some defect state as been declared, and signalled upwards, from the MPLS server layer (which of course assumes this is using decent OAM such as Y.1711);

- when the client layer is a packet network (eg FR or ATM) then one cannot tell the difference between a quiescent user (or a low-rate user with low packet-loss) and a dead connection (or



a high-rate user with high packet-loss). Which again is why for co pkt-sw networks we need a function like the CV OAM packet as defined in Y.1711 which is deterministic/independent of the user traffic behaviour.

I have questioned various people in IETF about the above aspects many times and it seems no thought has been given to any of these matters (or if there has, this has been done ‘behind closed doors’ and left vague on purpose in the documents).

1.2.4 The IETF solution to MPLS fault-management (LSP-Ping)

The current IETF solution for the fault-management of MPLS is to use a tool called LSP-Ping [5] which has been under development (in ever changing guises) in the IETF for at least 2 years now.

LSP-Ping is loosely based on the classic IP Ping (ICMP echo request/response) OAM function, however it has had to be significantly modified (and in it latest guise is very complex) to address the mp2p co pkt-sw mode of LDP.

LSP-Ping essentially comprises two components: A trace route component and an end-point Ping component. The latest form of the proposals suggest that an LSP is first traced out and then the end point is pinged periodically.

This requires (in very simplified terms) sending out a ‘MPLS echo request’ to a given FEC with increasing TTL and a UDP (destination port 3503) IP payload with DA chosen at random from the 127/8 address-space (this is hoped to exercise any ECMP paths). The MPLS echo request contains downstream FEC mapping TLVs. When the TTL exhausts it is supposed to return (using one of the optional return modes specified) the next hop label mappings for the FEC. This information is used in subsequent MPLS echo requests with the TTL increased by 1, and the process repeats until all the paths (including ECMP subnetworks) are hopefully traced out.

Once the LSP is fully traced out then the end point can be periodically pinged to check for continuity. It is also recommended that the trace route function be periodically run just in case routings have changed. How often this is should be done is left open-ended.

As noted previously, fault management of mp2p constructs is problematic per se, and it gets even more problematic when ECMP, PHP and TTL-copying are added.....these also impact whether LSP-Ping will work or not. There are many problems with LSP-Ping itself, and the following list is far from exhaustive: it’s now a complex protocol and its not clear anyone really understands how it works under all

circumstances (based on a recent spate of mail on the draft); no specific defects are defined anywhere....so how one could ensure consistency of

specification (whether from a customers perspective or an operators perspective, say if interworking with another operator) is not obvious;

it will not detect all types of defects that could occur, and this varies depending on the usage mode, eg a request for a return via the (RSVP) control-plane won’t work at all in case of mismerged traffic when the MPLS echo request ends up at some arbitrary location;

for those defects it can detect, since the detection periods are left open-ended and no consequent actions are specified it is more likely the customer will be the first to detect



problems (which hardly sits easy with any operators ambition to drive down customer dissatisfaction and any desire to offer consistent and easily measurable SLAs);

it requires a ‘reliable’ return path to the source.......this is a weakness that is acknowledged in the draft, ie faults are not easy to resolve as to outgoing or return directions when bi-directional protocols are used. Fault detection/handling really needs to be done on a unidirectional basis as required in Y.1710 and specified in Y.1711.

its invokes no consequent actions at the sink of the LSP, eg suppress traffic, tell higher layers to suppress alarms, etc. This is a serious problem for the XoverMPLS cases, and especially so if the client layer (X) belongs to a different operator/customer domain;

the use of the 127/8 IP DA in the LSP-Ping payload is contentious in its own right (these addresses are not supposed to exist outside a host), and it is not certain or deterministic wrt exercising all ECMP paths using these addresses.....since there are more than 16 million such addresses then one would have to cycle them all to be sure all ECMP paths were exercised, and this is clearly not feasible. Any nested use of ECMP seems unfathomable;

TTL copying or not between different nested LSPs can result in unpredictable behaviour in the case of mismerged defects;

in the case of a trace-request entering a pipe model LSP, it seems the trace request won’t trace the LSP and it will just get routed to the egress;

PHP can give rise to unpredictable behaviour in the case of rfc2547 VPNs when the inner label is used to route traffic directly to a CE attachment circuit, ie ping/trace packets seem likely to get sent into the customer’s private network; And PHP is something we see as both a (i) security concern in its own right and (ii) a problem for ensuring the required consequent actions for a given failure are carried out at the right location, ie the true path sink node;

it is a wholly inappropriate technology to use (assuming they eventually get it to work in some fashion) for the XoverMPLS case, and especially the important ATM/MPLS service-interworking case. Here only Y.1711 as currently specified makes any sense.

Of course, if we did not have mp2p constructs in MPLS there would no real need for LSP-Ping at all. I make one final remark in closing this section:

LDP was originally conceived to provide faster forwarding than IP cnls forwarding, ie by only doing full-address look-up at the ingress, and thereafter using simpler label-swapping on pre-calculated LSP paths. However, advances in address processing speeds have nullified this advantage. To the author’s knowledge, mp2p LSPs solve no network problem or provide any significant advantage that cannot be better solved using either proper cnls IP transport (instead of LDP mp2p) when ‘soft SLAs’ are sufficient in L3 VPNs say, or using p2p LSPs when ‘hard SLAs’ (or XoverMPLS) is required. All LDP seems to provide are lots of fault-management and traffic/QoS-management problems, but this should not be surprising given that a mp2p/merging construction is an unnatural thing to do in a co pkt-sw forwarding mode anyway.

1.3 How to guarantee QoS and charge for it

A really hard problem all operators will face, especially as BB access increases in penetration and new applications emerge, is how to deliver QoS in IP and charge for it.



The IETF/router-vendor's DiffServ (DS) solution to this problem is indeed no solution....it is not in their interests to address this for a raft of political, ideological or commercial/legacy reasons. As an aside, we should note that the IETF are strictly only worried about protecting the Internet (big ‘I’) and not the commercial exploitation of the IP protocol by operators. Now since the Internet (and the routers it is based on) can only operate in a cnls mode, and since this mode alone is not sufficient to provide all the services we need, then the IETF will never provide overall/optimal solutions for traditional operators anyway. Operators need to understand this basic fact.

There are several facets to this problem space, and we initially need to split it between public services (which is actually the general problem case) and private services, ie VPNs (which is a specific subset of the general problem case).

1.3.1 Public services case....the general QoS problem

Fundamentally, any scheme that allows BW sharing/borrowing between users who are supposed to be getting differential treatment is flawed. Many years ago BT gave a work-package to Professor Frank Kelly (statistical maths labs of Cambridge University) to analyse DiffServ. The conclusions of this work simply confirmed something we engineers knew to be intuitively true (without the maths), ie DS has a tiny load/operating region where class differentiation is observable.......below that load point all classes look alike and the network is available to all, and above that load point all classes again look alike and the network is broken for all. This is typical shared network behaviour, just take a trip up you favourite motorway at midnight and the rush hour to see this in practical action.

It is not conceivable that anyone could run their networks in a stable manner at a consistent loading level to see DS exhibiting a true 'differential QoS behaviour'.......DS will provide a prioritised 'congestion behaviour' under resource shortfalls, eg failures. So what is the proposed tactic to deal with this? Well, today most operators simply over-engineer their core networks and push the hard BW partitioning issue to the edges where it cannot be avoided.

However, this ‘throw BW at the problem till it goes away’ idea still does not really address the QoS/changing problem as: actual class load distributions need to be controlled in the core to avoid focussed class overload

(not trivial when just using a SPF IGP); we can still not offer any hard guarantees to the customer anyway; it does not address the issue when one is a transit service-provider, ie remove the access case; we are still forced to hard-BW partition in the access (which is why ATM still dominates here); it makes pricing of different classes almost impossible.....customers will quickly realise that they

can get the performance they want from the lowest class for a very high % of time, and so there is little (if any) incentive to buy a higher class.

So what have we got now? Well, we have a network where DS is doing nothing useful at all anywhere, ie either in the core or the access. Put simply, a DS solution only works well when it does not have to work at all.

However, its a great model for the router vendors because they can still sell operators very expensive core boxes and we take the cost for these plus the continued 'over-engineer and pray'



attitude for the core. So this, coupled with the stove-pipe 'technology=>service' facet mentioned previously, has been most operator’s modus operandi to date.

But are these the most profitable/sustainable ways to carry our business forward in the future? My engineering instincts tell me not, and there has to be a better way which I will propose shortly. However, as a reminder, we have already addressed how to solve the stove-pipe case by proposing that we strive for functional convergence *within* each of the cnls, co pkt-sw and co cct-sw modes (but not *across* them all as some in IETF would have us believe is sensible, ie those pushing GMPLS).

1.3.2 Private services case....a specific QoS problem

This introduces a new and very important facet, which is this:

There is no relationship between a packet's DS codepoint (supposed to be required for up-state QoS forwarding treatment) and its survivability importance vis-à-vis any other packet....be it with the *same or different* DS codepoint.

Let me give a couple of examples of what I mean by this.

Imagine a VPN situation where voice may be mission critical or not, and similarly data may be mission critical or not. For example, a CEO cutting a critical business deal via a voice call and an office cleaner calling his/her mate for a gossip. Both have to go EF class (to meet delay/jitter transfer requirements for voice), but clearly one of these calls should survive over the other during resource shortfall. However it's worse than this, because if there is mission critical AF class traffic this will get displaced in favour of non-mission critical EF class traffic......and this applies *across* all VPNs in the region affected by the resource shortfall. So here we have the case that the survivability of a given VPN is a function of both its own traffic and all the traffic in other VPNs (and maybe public service traffic too if this is also merged in the same network based purely on a DS packet-level approach to QoS).....and, further, this relationship changes epoch by epoch as the traffic changes.

A further example in the case of a public network scenario would be how to distinguish between a really important voice call to the emergency services (in the UK ‘999’) and an ordinary voice call? Both have to go EF class (because this is what voice demands) but clearly the former must have a significantly higher claim on resource than the latter.

So as we can see, simple classification of QoS at the packet level alone is insufficient as we cannot distinguish ‘urgency’ from ‘importance’.

So what is today's solution? We over-engineer and pray again.

These observations raise a really important point - IETF do not grasp (and don’t want to grasp) the importance, both technically but especially commercially, of being able to manage 'connections' where this is necessary. And as a consequence they are producing flawed architectures and poor standards for MPLS as I have noted previously.



1.3.3 Private services solutions

So what is the right answer for the private services L3 VPN case? Well, I think we need two complimentary solutions using either a cnls IP server layer or a co pkt-sw p2p MPLS server layer since these both produce logical/manageable network modes.

Where soft SLAs are acceptable to customers it would seem more sensible to consider the use IP instead of mp2p. You can still do rfc2547 VPNs by using MPLSoverIP as proposed in the recent draft from Eric Rosen of Cisco [6]. So we keep the top-level 1 hop p2p LSPs (which are automatically created by BGP4, and which are effectively what rfc2547 is really all about), but use IP instead of mp2p LSPs underneath these. This gives a similar SLA/performance behaviour to using mp2p as is done today but without all the problems noted earlier, and we also get some additional benefits, ie: cannot have the same defects as mp2p, since as IP uses full SA/DA information for switching

and not link-connection identifiers....this has to be a major advantage; because of the above we don't need LSP-Ping, even if they do get it working one day.....this too

must be a significant advantage; there are no PHP or TTL-copying problems to worry about...... both of these are architectural

errors of judgement in their own right and also impact the efficacy of LSP-Ping; PHP also raises security concerns;

we don't even have to consider *any* label scaling problems in the P routers....the issue simply no longer exists!;

we can run it over any IP network.

It is understood Cisco have already released a solution for rfc2547 over IP (GRE tunnels). The only concern here is if there are any security issues raised by using GRE tunnels, ie brute force DOS-like attack on the 4 octet key.

Now where customers demand hard SLAs, proper traffic isolation between VPNs and robust fault-management, then there is no choice except to do this via p2p solutions. Note that this solution would not be for *all* VPN customers.....maybe the majority would take the soft-SLA IP option noted above? So it is only there for when it’s needed for those customers who demand it.

Further, a p2p solution will be vital to get things like ATM/MPLS service-interworking sorted. And a further benefit here is that we can run Y.1711 to get some simple/powerful fault-management tools in place.

Note also that when p2p solutions are used there is strictly no need for two levels of LSP as advocated in the various Martini drafts.......and this is simply a general truth for a client/server relationship between any two layer network technologies. Two levels of LSP are always forced however when mp2p is used, simply to get rid of the merging as I noted earlier. However, two levels of p2p LSPs could be used to improve hierarchical scaling benefits.

6 “Encapsulating MPLS in IP or GRE”. draft-rosen-mpls-in-ip-or-gre-00.txt



1.3.4 The general public services solution

There are two public networks which scale to massive size, are simple to manage, work very well for their intended services and have proofs by existence. One is the PSTN and the other is the Internet (big ‘I’). Why is this? Perhaps the most important reason is they only offer a single class of service.....and this is actually a more profound observation than it might appear at first glance.

I noted earlier that if we are ever going to crack the 'how to charge for QoS in IP' problem we will have to recognise the need to do two basic things: move away from any BW sharing/borrowing model; introduce management of application-interfacing protocols at a connection-level as well as at a

packet-level for certain types of service.

Once we accept these are fundamental ‘must-have’ requirements, then the only thing that is left to understand is how to implement a solution that brings these requirements together in a logical/harmonised manner.

The above points were recognised and elaborated on in a paper I wrote about 3 years ago on co/cnls unification [7]. This paper argued that it would be impossible to efficiently solve the full range of service/QoS requirements unless we could somehow offer end-systems/applications the choice of a cnls mode or a co pkt-sw mode. My paper also explained why IP and ATM could never converge to realise this vision (most fundamental of which is because they use different addressing families), and suggested that MPLS may hold out that promise *if* developed correctly.

However, we have quite recently come into contact with a company, who I shall call Vendor X (because we have a NDA with them), who seem to have gone down exactly the same line of reasoning as I did in the above referenced co/cnls unification paper. We have already held a couple of technical sessions with them, and their solution does hold-out great promise of delivering the co/cnls unification vision and, as a consequence, cracking the ‘IP QoS and how to charge for it’ problem.

In essence, the Vendor X proposal recognises the fundamental networking truth that any notion of BW sharing where differentiation only occurs at the packet level simply does not work, and that if operators are to be able to offer guaranteed QoS services, and charge accordingly for these, they will need to introduce a connection management function. They too have also noticed that the PSTN has many of the essential conceptual/functional properties that are required and is therefore a good role model. However, because of the importance of IP as an application interfacing protocol (which when considered at a PDU level has nothing to do with its use in a cnls network mode), then instead of the PSTN’s 64kbit/s co cct-sw mode, Vendor X is suggesting using an IP 'traffic quanta' (with suitable session-signalled QoS/traffic descriptors) in a co pkt-sw mode. This makes a huge amount of sense when you work through what it actually means....however, as we have a NDA with this company I cannot give any further details here.

The Vendor X solution does not address the 'traditional' cnls traffic however, so this is the final piece of the alternative architecture that needs adding. We will still need router-based networks to handle this. In my original co/cnls unification proposal I suggested we should look to the Internet



for the answer, with the key observation being (i) single class only per network and (ii) no BW sharing between different single-class networks.

So we require several *logically* hard-BW partitioned single-class cnls public networks running over simple/traditional routers. True QoS differentiation here accrues from running different user-densities per BW-unit in these networks and we charge accordingly for this (most likely on a flat-subscription basis as long as the total information sent in some agreed period does not exceed a certain threshold).

One of the single-class cnls networks could address the SME/teleworker market and another the residential market. We may need other cnls single-class partitioned networks, but two would seem a good starting point. Further, L3 rfc2547 VPNs could be provided off any of these single-class cnls networks (using IP over GRE say in place of the problematic mp2p LSP as discussed earlier), with the user-density factor controlling the delivered QoS. In all public cases (including the Vendor X case) the traffic would be groomed to the appropriate core network partition at some edge device.

In conclusion, this paper has shown that there is alternative way to provide network solutions that will truly address an operator’s real requirements, but that the current proposals coming from IETF and router vendors will not. Further, most of the functional components and standards are either already in-place or being worked-on to achieve this alternative vision. So it’s only a matter of time before we can start to migrate to this alternative vision.



Document HistoryVersion Issue date Comments

Draft 0.1 20/02/03 Initial draft

GlossaryATM Asynchronous Transfer ModeBB BroadBandBGP4 Border Gateway Protocol (version 4)BW BandWidthCCAMP Common Control And Management Protocolscnls Connection-Lessco Connection-Orientedco cct-sw Connection-Oriented Circuit-Switchedco pkt-sw Connection-Oriented Packet-SwitchedDLCI Data Link Connection IdentifierDOS Denial of ServiceDS Differentiated Services (DiffServ)DU Downstream UnsolicitedECMP Equal Cost Multi-PathFEC Forwarding Equivalence ClassFR Frame RelayGMPLS Generalised Multi-Protocol Label SwitchingGRE Generic Rate EncapsulationICMP Internet Control and Management ProtocolIETF Internet Engineering Task ForceIGP Interior Gateway ProtocolIP Internet ProtocolIPX Internetwork Packet eXchangeITU International Telecommunications UnionLDP Label Distribution ProtocolLSP Label Switched Pathmp2mp Multi-point-to-multi-pointmp2p Multi-point-to-pointMPLS Multi-Protocol Label SwitchingOAM Operations Administration and MaintenanceOTN Optical Transport Networkp2mp Point-to-multi-pointp2p Point-to-pointPDH Plesiochronous Digital HierarchyPDU Protocol Data UnitPHP Penultimate Hop Poppingpkt packet



PNNI Private Network to Network InterfacePPVPN Provider Provisioned Virtual Private NetworksPSTN Public Switched Telephone NetworkPWE3 Pseudo Wire Emulation Edge-to-EdgeQoS Quality of ServiceROI Return On InvestmentRSVP-TE ReSource reserVation Protocol – Traffic EngineeredSA/DA Source Address/Destination Address (in an IP packet)SDH Synchronous Digital HierarchySLA Service Level AgreementSPF Shortest Path Firstsw switchTDM Time Division MultiplexingTTL Time To LiveUDP User Datagram ProtocolVCI Virtual Channel IdentifierVPC Virtual Path IdentifierVPN Virtual Private NetworkWG Working Groupwrt with respect to

References

2 Y.1710 "Requirements for OAM functionality in for MPLS networks".

3 Y.1711 "OAM mechanism for MPLS networks".

4 “MPLS – Technology and applications. Bruce Davie and Yakov Rekhter. Morgan Kaufmann publishers. ISBN 1-55860-656-4

5 “Detecting MPLS Data Plane Liveness”. draft-ietf-mpls-lsp-ping-01.txt

7 “A new perspective on IP QoS from the unification of CO and CNLS transfer modes under a common addressing scheme”. Neil Harrison. 5 February 2000. [BT internal paper only]


Documents

A LOGICAL NETWORK END-GAME - 1-4-5.netdmm/NGN/End-Game_External_2_01.doc · Web viewSequence Numbers have been included in the control-word used in the various ... SDH Synchronous