DMVPN Explained _ CCIE Blog

Blog Home | INE Home | Members | Contact Us

Free Resources

View Archives

Training Products

CCIE Bloggers

Aug02

DMVPN Explained

48 CommentsPosted by Petr Lapukhov, 4xCCIE/CCDE in VPN

DMVPN stands for Dynamic Multipoint VPN and it is an effective solution for dynamicsecure overlay networks. In short, DMVPN is combination of the followingtechnologies:

1) Multipoint GRE (mGRE)2) Next-Hop Resolution Protocol (NHRP)4) Dynamic Routing Protocol (EIGRP, RIP, OSPF, BGP)3) Dynamic IPsec encryption5) Cisco Express Forwarding (CEF)

Assuming that reader has a general understanding of what DMVPN is and a solidunderstanding of IPsec/CEF, we are going to describe the role and function of eachcomponent in details. In this post we are going to illustrate two major phases of DMVPNevolution:

1) Phase 1 – Hub and Spoke (mGRE hub, p2p GRE spokes)2) Phase 2 – Hub and Spoke with Spoke-to-Spoke tunnels (mGRE everywhere)

As for DMVPN Phase 3 – “Scalable Infrastructure”, a separate post is required to cover thesubject. This is due to the significant changes made to NHRP resolution logic (NHRPredirects and shortcuts), which are better being illustrated when a reader has goodunderstanding of first two phases. However, some hints about Phase 3 will be also providedin this post.

DMVPN Explained | CCIE Blog http://blog.ine.com/2008/08/02/dmvpn-explained/

1 of 47 1/5/2011 12:40 AM

Note: Before we start, I would like to thank my friend Alexander Kitaev, for taking time toreview the post and providing me with useful feedback.

Multipoint GRE

Let us start with the most basic building component of DMVPN – multipoint GRE tunnel.Classic GRE tunnel is point-to-point, but mGRE generalizes this idea by allowing a tunnel tohave “multiple” destinations.

This may seem natural if the tunnel destination address is multicast (e.g. 239.1.1.1). Thetunnel could be used to effectively distribute the same information (e.g. video stream) tomultiple destinations on top of a multicast-enabled network. Actually, this is how mGRE isused for Multicast VPN implementation in Cisco IOS. However, if tunnel endpoints need toexchange unicast packets, special “glue” is needed to map tunnel IP addresses to “physical”or “real” IP addresses, used by endpoint routers. As we’ll see later, this glue is called NHRP.

Note, that if you source multiple mGRE tunnels off the same interface (e.g. Loopback0) of asingle router, then GRE can use special “multiplexor” field the tunnel header to differentiatethem. This field is known as “tunnel key” and you can define it under tunnel configuration. Asa matter of fact, up to IOS 12.3(14)T or 12.3(11)T3 the use of “tunnel key” was mandatory –mGRE tunnel would not come up, until the key is configured. Since the mentioned versions,you may configure a tunnel without the key. There were two reasons to remove therequirement. First, hardware ASICs of 6500 and 7600 platforms do not support mGRE


2 of 47 1/5/2011 12:40 AM

tunnel-key processing, and thus the optimal switching performance on those platforms ispenalized when you configure the tunnel key. Second, as we’ll see later, DMVPN Phase 3allows interoperation between different mGRE tunnels sharing the same NHRP network-idonly when they have the same tunnel-key or have no tunnel-key at all (since this allowssending packets “between” tunnels).

Generic NHRP

Now let’s move to the component that makes DMVPN truly dynamic – NHRP. The protocolhas been defined quite some time ago in RFC 2332 (year 1998) to create a routingoptimization scheme inside NBMA (non-broadcast multiple-access) networks, such as ATM,Frame-Relay and SMDS (anybody remembers this one nowadays? The general idea wasto use SVC (switched virtual circuits) to create temporary shortcuts in non-fully meshedNBMA cloud. Consider the following schematic illustration, where IP subnet 10.0.0.0/24overlays partial-meshed NBMA cloud. NHRP is similar in function to ARP, allowing resolvingL3 to L2 addresses, but does that in more efficient manner, suitable for partially meshedNBMA clouds supporting dynamic layer 2 connections.

The following is simplified and schematic illustration of NHRP process. In the above topology,in order for R1 to reach R4, it must send packets over PVCs between R1-R2, R2-R3 andfinally R3-R4. Suppose the NMBA cloud allows using SVC (Switched virtual circuits, dynamicpaths) – then it would be more reasonable for R1 to establish SVC directly with R4 and sendpackets over the optimal way. However, this requires R1 to know NMBA address (e.g. ATMNSAP) associated with R4 to “place a call”. Preferably, it would be better to make R1 learnR4 IP address to NSAP (NBMA address) mapping dynamically.

Now assume we enable NHRP on all NBMA interfaces in the network. Each router in topologyacts as either NHC (Next-Hop Client) or NHS (Next-Hop Server). One of the functions of NHCis to register with NHS its IP address mapped to NBMA Layer 2 address (e.g. ATM NSAP


3 of 47 1/5/2011 12:40 AM

address). To make registration possible, you configure each NHC with the IP address of atleast one NHS. In turn, NHS acts as a database agent, storing all registered mappings, andreplying to NHC queries. If NHS does not have a requested entry in its database, it canforward packet to another NHS to see if it has the requested association. Note that a routermay act as a Next-Hop server and client at the same time. Back to the diagram, assume thatR2 and R3 are NHSes, R1 and R4 are NHCs. Further, assume R4 is NHC and registers its IPto NBMA address mapping with R4 and R1 thinks R2 is the NHS. Both R2 and R3 treat eachother as NHS. When R1 wants to send traffic to R4 (next-hop 10.0.0.4), it tries to resolve10.0.0.4 by sending NHRP resolution request to R2 – the configured NHS. In turn, R2 willforward request to R3, since it has no local information.

Obviously, modern networks tend not to use ATM/SMDS and Frame-Relay SVC too much,but one can adopt NHRP to work with “simulated NBMA” networks, such as mGRE tunnels.The NBMA layer maps to “physical” underlying network while mGRE VPN is the “logical”network (tunnel internal IP addressing). In this case, mGRE uses NHRP for mapping “logical”or “tunnel inside” IP addresses to “physical” or real IP addresses. Effectively, NHRP performthe “glue” function described above, allowing mGRE endpoints discovering each other’s realIP address. Since NHRP defines a server role, it’s natural to have mGRE topology lay out inHub-and-Spoke manner (or combination of hubs and spokes, in more advanced cases). Let’ssee some particular scenarios to illustrate NHRP functionality with mGRE.

NHRP Phase 1

With NHRP Phase 1 mGRE uses NHRP to inform the hub about dynamically appearingspokes. Initially, you configure every spoke with the IP address of the hub as the NHS server.However, the spoke’s tunnel mode is GRE (regular point-to-point) tunnel with a fixeddestination IP that equals to the physical address of the hub. The spokes can only reach thehub and only get to other spoke networks across the hub. The benefit of Phase 1 is simplifiedhub router configuration, which does not require static NHRP mapping for every new spoke.


4 of 47 1/5/2011 12:40 AM

As all packets go across the hub, almost any dynamic routing protocol would help withattaining reachability. The hub just needs to advertise a default route to spokes, while spokesshould advertise their subnets dynamically to the hub. Probably it makes sense to run EIGRPand summarize all subnets to 0.0.0.0/0 on the hub, effectively sending a default route to allspokes (if the spokes do not use any other default route, e.g. from their ISPs). Configurespokes as EIGRP stubs and advertise their respective connected networks. RIP could be setup in similar manner, by simply configuring GRE tunnels on spokes as passive interfaces.Both EIGRP and RIP require split-horizon disabled on the hub mGRE interface in order toexchange subnets spoke to spoke. As for OSPF, the optimal choice would be using point-to-multipoint network type on all GRE and mGRE interfaces. In addition to that, configure ipospf database filter-all out on the hub and set up static default routes via tunnel interfaceson the spokes (or static specific routes for corporate networks).

Here is a sample configuration. The detailed explanation of NHRP commands and “show”commands output follows the example.

mGRE + NHRP Phase 1 + EIGRP

R1:!! Hub router!router eigrp 123 no auto-summary network 10.0.0.0 0.255.255.255!! Tunnel source!interface Loopback0 ip address 150.1.1.1 255.255.255.0!! VPN network!interface Loopback 1 ip address 10.0.1.1 255.255.255.0!! mGRE tunnel!interface Tunnel0 ip address 10.0.0.1 255.255.255.0 no ip redirects ip nhrp authentication cisco ip nhrp map multicast dynamic ip nhrp network-id 123 no ip split-horizon eigrp 123 ip summary-address eigrp 123 0.0.0.0 0.0.0.0 5 tunnel source Loopback0 tunnel mode gre multipoint tunnel key 123

R2:!! Spoke Router!router eigrp 123 no auto-summary network 10.0.0.0 0.255.255.255 eigrp stub connected!


5 of 47 1/5/2011 12:40 AM

interface Loopback0 ip address 150.1.2.2 255.255.255.0!interface Loopback 1 ip address 10.0.2.2 255.255.255.0!! GRE tunnel!interface Tunnel0 ip address 10.0.0.2 255.255.255.0 ip nhrp authentication cisco ip nhrp map multicast 150.1.1.1 ip nhrp map 10.0.0.1 150.1.1.1 ip nhrp nhs 10.0.0.1 ip nhrp network-id 123 ip nhrp registration timeout 30 ip nhrp holdtime 60 tunnel source Loopback0 tunnel destination 150.1.1.1 tunnel key 123

R3:!! Spoke Router!router eigrp 123 no auto-summary network 10.0.0.0 0.255.255.255 eigrp stub connected!interface Loopback0 ip address 150.1.3.3 255.255.255.0!interface Loopback 1 ip address 10.0.3.3 255.255.255.0!interface Tunnel0 ip address 10.0.0.3 255.255.255.0 ip nhrp authentication cisco ip nhrp map multicast 150.1.1.1 ip nhrp map 10.0.0.1 150.1.1.1 ip nhrp nhs 10.0.0.1 ip nhrp network-id 123 ip nhrp registration timeout 30 ip nhrp holdtime 60 tunnel source Loopback0 tunnel destination 150.1.1.1 tunnel key 123

Note that only the hub tunnel uses mGRE encapsulation, and spokes use regular point-to-point GRE tunnels. Now, let’s look at the NHRP commands used in the example above.The most basic command ip nhrp map [Logical IP] [NBMA IP] – creates a static bindingbetween a logical IP address and NBMA IP address. Since mGRE is treated by NHRP asNMBA medium, logical IP corresponds to the IP address “inside” a tunnel (“inner”) and theNBMA IP address corresponds to the “outer” IP address (the IP address used to source atunnel). (From now on, we are going to call “inner” IP address and simply “IP address” or“logical IP address” and the “outer” IP address as “NBMA address” or “physical IP address”).The use of static NHRP mappings is to “bootstrap” information for the spokes to reach thelogical IP address of the hub. The next command is ip nhrp map multicast dynamic|[StaticIP] and its purpose is the same as “frame-relay mapH broadcast”. The command


6 of 47 1/5/2011 12:40 AM

specifies the list of destination that will receive the multicast/broadcast traffic originated fromthis router. Spokes map multicasts to the static NBMA IP address of the hub, but hub mapsmulticast packets to the “dynamic” mappings – that is, the hub replicates multicast packets toall spokes registered via NHRP. Mapping multicasts is important in order to make dynamicrouting protocol establish adjacencies and exchange update packets. The ip nhrp nhs[ServerIP] command configures NHRP client with the IP address of its NHRP server. Notethe “ServerIP” is the logical IP address of the hub (inside the tunnel) and therefore spokesneed the static NHRP mappings in order to reach it. The spokes use the NHS to register theirlogical IP to NBMA IP associations and send NHRP resolution request. (However, in thisparticular scenarios, the spokes will not send any NHRP Resolutions Requests, since theyuse directed GRE tunnels – only registration requests will be sent). The commands ip nhrpnetwork-id and ip nhrp authentication [Key] identify and authenticate the logical NHRPnetwork. The [ID] and the [Key] must match on all routers sharing the same GRE tunnel. It ispossible to split an NBMA medium into multiple NHRP networks, but this is for advancedscenarios. As for the authentication, it’s a simple plain-text key sent in all NHRP messages.While the “network-id” is mandatory in order for NHRP to work, you may omit theauthentication. Next command is ip nhrp holdtime that specifies the hold-time value set inNHRP registration requests. The NHS will keep the registration request cached for theduration of the hold-time, and then, if no registration update is receive, will time it out. TheNHS will also send the same hold-time in NHRP resolution responses, if queried for therespective NHRP association. Note that you configure the ip nhrp holdtime command onspokes, and spoke will send registration requests every 1/3 of the hold-time seconds.However, if you also configure the ip nhrp registration timeout [Timeout] on a spoke, theNHRP registration requests will be sent every [Timeout] sends, not 1/3 of the configuredhold-time. The hold-time value sent in NHRP Registration Requests will remain the same, ofcourse.

Now let’s move to the show commands. Since it’s only the hub that uses the NHRP dynamicmappings to resolve the spokes NBMA addresses, it is useful to observe R1 NHRP cache:

Rack1R1#show ip nhrp detail10.0.0.2/32 via 10.0.0.2, Tunnel0 created 00:16:59, expire 00:00:30 Type: dynamic, Flags: authoritative unique registered used NBMA address: 150.1.2.210.0.0.3/32 via 10.0.0.3, Tunnel0 created 00:11:34, expire 00:00:55 Type: dynamic, Flags: authoritative unique registered used NBMA address: 150.1.3.3

As we can see, the logical IP “10.0.0.2” maps to NBMA address “150.1.2.2” and the logical IP10.0.0.3 maps to NBMA address 150.1.3.3. The “authoritative” flag means that the NHS haslearned about the NHRP mapping directly from a registration request (the NHS “serves” theparticular NHC). The “unique” flag means that the NHRP registration request had the same“unique” flag set. The use of this flag is to prevent duplicate NHRP mappings in cache. Ifunique mapping for a particular logical IP is already in the NHRP cache and another NHCtries to register the same logical IP with the NHS, the server will reject the registration, untilthe unique entry expires. Note that by default IOS routers set this flag in registration request,and this can be disabled by using ip nhrp registration no-unique command on a spoke.Sometimes this may be needed when spoke change its NBMA IP address often and needs tore-register a new mapping with the hub. The last flag, called “used” flag, means that the


7 of 47 1/5/2011 12:40 AM

router uses the NHRP entry to switch IP packets. We will discuss the meaning of this flag inNRHP process switching section below. Also, note the “expires” field, which is a countdowntimer, started from the “holdtime” specified in the Registration Request packet.

Let’s see the NHRP registration and reply process flows on the NHS.

Rack1R1#debug nhrpNHRP protocol debugging is on

Rack1R1#debug nhrp packetNHRP activity debugging is on

First, R3 tries to register its Logical IP to NBMA IP mapping with the hub. Note the specificNHRP packet format, split in three parts.

1) (F) – fixed part. Specifies the version, address family (afn) and protocol type (type) forresolution, as well as subnetwork layer (NBMA) type and length (shtl and sstl). Note that “shtl”equals 4, which is the length of IPv4 address in bytes, and “sstl” is for “subaddress” fieldwhich is not used with IPv4.

2) (M) – mandatory header part. Specifies some flags, like “unique” flag and the “Request ID”,which is used to track request/responses. Also includes are the source NBMA address(tunnel source in GRE/mGRE) and the source/destination protocol IP addresses. DestinationIP address is the logical IP address of the hub and the source IP address is the logical IPaddress of the spoke. Using this information hub may populate the spoke logical IP addressto NBMA IP address mapping.

3) (C-1) – CIE 1, which stands for “Client Information Element” field. While it’s not used in thepackets below, in more advanced scenarios explored later, we’ll see this filed containing theinformation about networks connected to requesting/responding routers.

Also note the NAT-check output, which is Cisco’s extension used to make NHRP work forrouters that tunnel from behind the NAT.

NHRP: Receive Registration Request via Tunnel0 vrf 0, packet size: 81 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "unique", reqid: 26 src NBMA: 150.1.3.3 src protocol: 10.0.0.3, dst protocol: 10.0.0.1 (C-1) code: no error(0) prefix: 255, mtu: 1514, hd_time: 60 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0NHRP: netid_in = 123, to_us = 1NHRP: NAT-check: matched destination address 150.1.3.3NHRP: Tu0: Found and skipping dynamic multicast mapping NBMA: 150.1.3.3NHRP: Attempting to send packet via DEST 10.0.0.3NHRP: Encapsulation succeeded. Tunnel IP addr 150.1.3.3

After processing the request, the router responds with NHRP Registration Reply. Note thatthe (M) header did not change, just the source and destination logical IP address of thepacket are reversed. (R1->R3)

NHRP: Send Registration Reply via Tunnel0 vrf 0, packet size: 101


8 of 47 1/5/2011 12:40 AM

src: 10.0.0.1, dst: 10.0.0.3 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "unique", reqid: 26 src NBMA: 150.1.3.3 src protocol: 10.0.0.3, dst protocol: 10.0.0.1 (C-1) code: no error(0) prefix: 255, mtu: 1514, hd_time: 60 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0NHRP: 101 bytes out Tunnel0

Now the NHS receives the Registration Request from R2, and adds the corresponding entryin its NHRP cache

NHRP: Receive Registration Request via Tunnel0 vrf 0, packet size: 81 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "unique", reqid: 38 src NBMA: 150.1.2.2 src protocol: 10.0.0.2, dst protocol: 10.0.0.1 (C-1) code: no error(0) prefix: 255, mtu: 1514, hd_time: 60 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0NHRP: netid_in = 123, to_us = 1NHRP: NAT-check: matched destination address 150.1.2.2NHRP: Tu0: Found and skipping dynamic multicast mapping NBMA: 150.1.2.2NHRP: Attempting to send packet via DEST 10.0.0.2NHRP: Encapsulation succeeded. Tunnel IP addr 150.1.2.2

NHRP: Send Registration Reply via Tunnel0 vrf 0, packet size: 101 src: 10.0.0.1, dst: 10.0.0.2 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "unique", reqid: 38 src NBMA: 150.1.2.2 src protocol: 10.0.0.2, dst protocol: 10.0.0.1 (C-1) code: no error(0) prefix: 255, mtu: 1514, hd_time: 60 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0NHRP: 101 bytes out Tunnel0

We see how NRHP Phase 1 works now. The spokes register their associations with the hubvia NHRP and the hub learns their NBMA addresses dynamically. At the same time, spokesuse point-to-point tunnels to speak to the hub and reach each other. Note that EIGRP is notthe only protocol suitable for use with NHRP Phase 1. OSPF is also a viable solution, thank topoint-to-multipoint network type and database filter-all out command. See the examplebelow for OSPF configuration with NHRP Phase 1:

mGRE + NHRP Phase 1 + OSPF

R1:!! Hub router!router ospf 123 router-id 10.0.0.1 network 10.0.0.0 0.255.255.255 area 0!interface Loopback0


9 of 47 1/5/2011 12:40 AM

ip address 150.1.1.1 255.255.255.0!interface Loopback 1 ip address 10.0.1.1 255.255.255.0!interface Tunnel0 ip address 10.0.0.1 255.255.255.0 no ip redirects ip nhrp authentication cisco ip nhrp map multicast dynamic ip nhrp network-id 123 tunnel source Loopback0 tunnel mode gre multipoint tunnel key 123 ip ospf network point-to-multipoint ip ospf database-filter all out

R2:!! Spoke Router!router ospf 123 network 10.0.0.0 0.255.255.255 area 0 router-id 10.0.0.2!interface Loopback0 ip address 150.1.2.2 255.255.255.0!interface Loopback 1 ip address 10.0.2.2 255.255.255.0!interface Tunnel0 ip address 10.0.0.2 255.255.255.0 ip nhrp authentication cisco ip nhrp map multicast 150.1.1.1 ip nhrp map 10.0.0.1 150.1.1.1 ip nhrp nhs 10.0.0.1 ip nhrp network-id 123 ip nhrp registration timeout 30 ip nhrp holdtime 60 tunnel source Loopback0 tunnel destination 150.1.1.1 tunnel key 123 ip ospf network point-to-multipoint!ip route 0.0.0.0 0.0.0.0 Tunnel0

R3:!! Spoke Router!router ospf 123 network 10.0.0.0 0.255.255.255 area 0 router-id 10.0.0.3!interface Loopback0 ip address 150.1.3.3 255.255.255.0!interface Loopback 1 ip address 10.0.3.3 255.255.255.0!interface Tunnel0 ip address 10.0.0.3 255.255.255.0 ip nhrp authentication cisco ip nhrp map multicast 150.1.1.1


10 of 47 1/5/2011 12:40 AM

ip nhrp map 10.0.0.1 150.1.1.1 ip nhrp nhs 10.0.0.1 ip nhrp network-id 123 ip nhrp registration timeout 30 ip nhrp holdtime 60 tunnel source Loopback0 tunnel destination 150.1.1.1 tunnel key 123 ip ospf network point-to-multipoint!ip route 0.0.0.0 0.0.0.0 Tunnel0

As we said, the main benefit of using NHRP Phase 1 is simplified configuration on the hubrouter. Additionally, spoke routers receive minimal routing information (it’s either summarizedor filtered on the hub) and are configured in uniform manner. In most simple case, spokerouters could be configured without any NHRP, by simply using point-to-point GRE tunnels.This scenario requires the hub to create a static NHRP mapping for every spoke. Forexample:

mGRE + NHRP Phase 1 + OSPF + Static NHRP mappings

R1:!! Hub router!router ospf 123 router-id 10.0.0.1 network 10.0.0.0 0.255.255.255 area 0!interface Loopback0 ip address 150.1.1.1 255.255.255.0!interface Loopback 1 ip address 10.0.1.1 255.255.255.0!interface Tunnel0 ip address 10.0.0.1 255.255.255.0 no ip redirects ip nhrp authentication cisco ip nhrp map 10.0.0.2 150.1.2.2 ip nhrp map 10.0.0.3 150.1.3.3 ip nhrp map multicast 150.1.2.2 ip nhrp map multicast 150.1.3.3 ip nhrp network-id 123 tunnel source Loopback0 tunnel mode gre multipoint tunnel key 123 ip ospf network point-to-multipoint ip ospf database-filter all out

R2:!! Spoke Router!router ospf 123 network 10.0.0.0 0.255.255.255 area 0 router-id 10.0.0.2!interface Loopback0 ip address 150.1.2.2 255.255.255.0!


11 of 47 1/5/2011 12:40 AM

interface Loopback 1 ip address 10.0.2.2 255.255.255.0!interface Tunnel0 ip address 10.0.0.2 255.255.255.0 tunnel source Loopback0 tunnel destination 150.1.1.1 tunnel key 123 ip ospf network point-to-multipoint!ip route 0.0.0.0 0.0.0.0 Tunnel0

R3:!! Spoke Router!router ospf 123 network 10.0.0.0 0.255.255.255 area 0 router-id 10.0.0.3!interface Loopback0 ip address 150.1.3.3 255.255.255.0!interface Loopback 1 ip address 10.0.3.3 255.255.255.0!interface Tunnel0 ip address 10.0.0.3 255.255.255.0 tunnel source Loopback0 tunnel destination 150.1.1.1 tunnel key 123 ip ospf network point-to-multipoint!ip route 0.0.0.0 0.0.0.0 Tunnel0

The disadvantage of NHRP Phase 1 is the inability to establish spoke-to-spoke shortcuttunnels. NHRP Phase 2 resolves this issue and allows for spoke-to-spoke tunnels. To betterunderstand the second phase, we first need to find out how NHRP interacts with CEF – thenow default IP switching method on most Cisco routers. Consider the topology and exampleconfiguration that follows. See the detailed breakdown after the configuration.

mGRE + NHRP Phase 2 + EIGRP


12 of 47 1/5/2011 12:40 AM

R1:!! Hub router!router eigrp 123 no auto-summary network 10.0.0.0 0.255.255.255!interface Loopback0 ip address 150.1.1.1 255.255.255.0!interface Loopback 1 ip address 10.0.1.1 255.255.255.0!interface Tunnel0 ip address 10.0.0.1 255.255.255.0 no ip redirects ip nhrp authentication cisco ip nhrp map multicast dynamic ip nhrp network-id 123 no ip split-horizon eigrp 123 no ip next-hop-self eigrp 123 tunnel source Loopback0 tunnel mode gre multipoint tunnel key 123

R2:!! Spoke Router!router eigrp 123 no auto-summary network 10.0.0.0 0.255.255.255 eigrp stub connected!interface Loopback0 ip address 150.1.2.2 255.255.255.0!interface Loopback 1 ip address 10.0.2.2 255.255.255.0!


13 of 47 1/5/2011 12:40 AM

interface Tunnel0 ip address 10.0.0.2 255.255.255.0 ip nhrp authentication cisco ip nhrp map multicast 150.1.1.1 ip nhrp map 10.0.0.1 150.1.1.1 ip nhrp nhs 10.0.0.1 ip nhrp network-id 123 ip nhrp registration timeout 30 ip nhrp holdtime 60 tunnel source Loopback0 tunnel mode gre multipoint tunnel key 123

R3:!! Spoke Router!router eigrp 123 no auto-summary network 10.0.0.0 0.255.255.255 eigrp stub connected!interface Loopback0 ip address 150.1.3.3 255.255.255.0!interface Loopback 1 ip address 10.0.3.3 255.255.255.0!interface Tunnel0 ip address 10.0.0.3 255.255.255.0 ip nhrp authentication cisco ip nhrp map multicast 150.1.1.1 ip nhrp map 10.0.0.1 150.1.1.1 ip nhrp nhs 10.0.0.1 ip nhrp network-id 123 ip nhrp registration timeout 30 ip nhrp holdtime 60 tunnel source Loopback0 tunnel mode gre multipoint tunnel key 123

Note that both spokes use mGRE tunnel encapsulation mode, and the hub sets theoriginating router next-hop IP address in “reflected” EIGRP updates (by default EIGRP setsthe next-hop field to “0.0.0.0” – that is, to self). By the virtue of the EIGRP configuration, thesubnet “10.0.2.0/24” (attached to R2) reaches to R3 with the next-hop IP address of “10.0.0.2”(R2). It is important that R3 learns “10.0.2.0/24” with the next hop of R2 logical IP address. Aswe see later, this is the key to trigger CEF next-hop resolution. The mGRE encapsulationused on spokes will trigger NHRP resolutions since now this is NBMA medium. Now,assuming that traffic to 10.0.2.0/24 does not flow yet, check the routing table entry for 10.0.2.2and the CEF entries for the route and its next-hop:

Rack1R3#show ip route 10.0.2.2Routing entry for 10.0.2.0/24 Known via "eigrp 123", distance 90, metric 310172416, type internal Redistributing via eigrp 123 Last update from 10.0.0.2 on Tunnel0, 00:09:55 ago Routing Descriptor Blocks: * 10.0.0.2, from 10.0.0.1, 00:09:55 ago, via Tunnel0 Route metric is 310172416, traffic share count is 1 Total delay is 1005000 microseconds, minimum bandwidth is 9 Kbit Reliability 255/255, minimum MTU 1472 bytes


14 of 47 1/5/2011 12:40 AM

Loading 1/255, Hops 2

Rack1R3#show ip cef 10.0.2.210.0.2.0/24, version 48, epoch 00 packets, 0 bytes via 10.0.0.2, Tunnel0, 0 dependencies next hop 10.0.0.2, Tunnel0 invalid adjacency

Rack1R3#show ip cef 10.0.0.210.0.0.0/24, version 50, epoch 0, attached, connected0 packets, 0 bytes via Tunnel0, 0 dependencies valid glean adjacency

Note that CEF prefix for “10.0.2.0/24” is invalid (but not “glean”), since “10.0.0.2” has not yetbeen resolved. The CEF prefix for “10.0.0.2” has “glean” adjacency, which means the routerneeds to send an NHRP resolution request to map the logical IP to NBMA address.Therefore, with CEF switching, NHRP resolution requests are only sent for “next-hop” IPaddresses, and never for the networks (e.g. 10.0.2.0/24) themselves (the process-switchingdoes resolve any prefix as we’ll see later). Go ahead and ping from R3 to “10.0.3.3” andobserve the process:

Rack1R3#ping 10.0.2.2

Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 10.0.2.2, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 36/80/180 ms

Check the mappings on the hub router. The only two entries registered are the VPN IPaddresses of R2 and R3, together with the respective NBMA IP addresses. Note the “expire”field, which, as mentioned above, counts the time for the entry to expire based on the“holdtime” settings of the registering router’s interface. Later we will see how CEF uses thiscountdown timer to refresh or delete CEF entries for the next-hop IP address

Rack1R1#show ip nhrp10.0.0.2/32 via 10.0.0.2, Tunnel0 created 00:16:33, expire 00:00:43 Type: dynamic, Flags: authoritative unique registered NBMA address: 150.1.2.2 Requester: 10.0.0.3 Request ID: 79810.0.0.3/32 via 10.0.0.3, Tunnel0 created 00:16:34, expire 00:00:51 Type: dynamic, Flags: authoritative unique registered NBMA address: 150.1.3.3 Requester: 10.0.0.2 Request ID: 813

Check the mappings on R2 (note that R2 now has mapping for R3’s

next-hop associated with its NBMA IP address)

Rack1R2#show ip nhrp10.0.0.1/32 via 10.0.0.1, Tunnel0 created 00:14:52, never expire Type: static, Flags: authoritative used NBMA address: 150.1.1.110.0.0.2/32 via 10.0.0.2, Tunnel0 created 00:05:49, expire 00:00:10 Type: dynamic, Flags: router authoritative unique local NBMA address: 150.1.2.2 (no-socket)10.0.0.3/32 via 10.0.0.3, Tunnel0 created 00:00:30, expire 00:00:29 Type: dynamic, Flags: router used


15 of 47 1/5/2011 12:40 AM

NBMA address: 150.1.3.3

The same command output on R3 is symmetric to the output on R2:

Rack1R3#show ip nhrp10.0.0.1/32 via 10.0.0.1, Tunnel0 created 00:14:00, never expire Type: static, Flags: authoritative used NBMA address: 150.1.1.110.0.0.2/32 via 10.0.0.2, Tunnel0 created 00:00:05, expire 00:00:54 Type: dynamic, Flags: router NBMA address: 150.1.2.210.0.0.3/32 via 10.0.0.3, Tunnel0 created 00:01:46, expire 00:00:13 Type: dynamic, Flags: router authoritative unique local NBMA address: 150.1.3.3 (no-socket)

Now check the CEF entry for R2’s next-hop IP address on R3:

Rack1R3#sh ip cef 10.0.0.210.0.0.2/32, version 65, epoch 0, connected0 packets, 0 bytes via 10.0.0.2, Tunnel0, 0 dependencies next hop 10.0.0.2, Tunnel0 valid adjacency

The CEF entry for “10.0.0.2” is now valid, since NHRP mapping entry is present. If thenext-hop for the prefix “10.0.2.0/24” was pointing toward the hub (R1) (e.g. if the hub wasusing the default ip next-hop-self eigrp 123) then the NHRP lookup will not be triggered,and cut-through NHRP entry will not be installed. Let’s see the debugging command outputon R1, R2 and R3 to observe how the routers collectively resolve the next-hop IP addresseswhen R3 pings R1:

Rack1R1#debug nhrpNHRP protocol debugging is onRack1R1#debug nhrp packetNHRP activity debugging is on



It all starts when R3 tries to route a packet to “10.0.2.2” and finds out it has “glean” adjacencyfor its next-hop of “10.0.0.2”. Then, R3 attempt to send NHRP resolution request directly toR2, but fails since R2 NMBA address is unknown. At the same time, the original data packet(ICMP echo) follows to R2 across the hub (R1).

Rack1R3#NHRP: MACADDR: if_in null netid-in 0 if_out Tunnel0 netid-out 123NHRP: Checking for delayed event 0.0.0.0/10.0.0.2 on list (Tunnel0).NHRP: No node found.NHRP: Sending packet to NHS 10.0.0.1 on Tunnel0NHRP: Checking for delayed event 0.0.0.0/10.0.0.2 on list (Tunnel0).NHRP: No node found.


16 of 47 1/5/2011 12:40 AM

NHRP: Attempting to send packet via DEST 10.0.0.2NHRP: Send Resolution Request via Tunnel0 vrf 0, packet size: 81 src: 10.0.0.3, dst: 10.0.0.2 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth src-stable", reqid: 994 src NBMA: 150.1.3.3 src protocol: 10.0.0.3, dst protocol: 10.0.0.2 (C-1) code: no error(0) prefix: 0, mtu: 1514, hd_time: 360 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0NHRP: Encapsulation failed for destination 10.0.0.2 out Tunnel0

Next, R3 tries to send resolution request to the NHS, which is R1. The resolution requestcontains information about source NBMA address of R3, and source protocol (logical IP)addresses of R3 and R2.

Rack1R3#NHRP: Attempting to send packet via NHS 10.0.0.1NHRP: Encapsulation succeeded. Tunnel IP addr 150.1.1.1NHRP: Send Resolution Request via Tunnel0 vrf 0, packet size: 81 src: 10.0.0.3, dst: 10.0.0.1 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth src-stable", reqid: 994 src NBMA: 150.1.3.3 src protocol: 10.0.0.3, dst protocol: 10.0.0.2 (C-1) code: no error(0) prefix: 0, mtu: 1514, hd_time: 360 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0NHRP: 81 bytes out Tunnel0

The Resolution Request from R3 arrives to NHS. In essence, R3 tries to resolve the “glean”CEF adjacency using NHRP the same way it uses ARP on Ethernet. Note that request onlymentions logical IP addresses of R3 (“10.0.0.3”) and R2 (“10.0.0.2”) and NBMA address ofR3.

Rack1R1#NHRP: Receive Resolution Request via Tunnel0 vrf 0, packet size: 81 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth src-stable", reqid: 994 src NBMA: 150.1.3.3 src protocol: 10.0.0.3, dst protocol: 10.0.0.2 (C-1) code: no error(0) prefix: 0, mtu: 1514, hd_time: 360 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0NHRP: netid_in = 123, to_us = 0NHRP: NAT-check: matched destination address 150.1.3.3NHRP: nhrp_rtlookup yielded Tunnel0NHRP: Tu0: Found and skipping dynamic multicast mapping NBMA: 150.1.3.3NHRP: netid_out 123, netid_in 123NHRP: nhrp_cache_lookup_comp returned 0x855C7B90NHRP: Attempting to send packet via DEST 10.0.0.3NHRP: Encapsulation succeeded. Tunnel IP addr 150.1.3.3

The NHS has the NHRP mapping for “10.0.0.2” in its NHRP cache – R2 registered thisassociating with R1. The NHS may immediately reply to the client. Note the “(C-1)” – CIEheader in the NHRP reply packet. While the “(M)” (mandatory) header contains the same


17 of 47 1/5/2011 12:40 AM

information received in request packet from R3, the CIE header contains the actual NHRPreply, with the mapping information for R2. This is because the NHS considers R2 to be the“client” of it, and therefore it sends the actual information in CIE header. Note the “prefix”length of 32 – this means the reply is just for one host logical IP address.

Rack1R1#NHRP: Send Resolution Reply via Tunnel0 vrf 0, packet size: 109 src: 10.0.0.1, dst: 10.0.0.3 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth dst-stable unique src-stable", reqid: 994 src NBMA: 150.1.3.3 src protocol: 10.0.0.3, dst protocol: 10.0.0.2 (C-1) code: no error(0) prefix: 32, mtu: 1514, hd_time: 342 addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 0 client NBMA: 150.1.2.2 client protocol: 10.0.0.2NHRP: 109 bytes out Tunnel0

At this point, R2 receives the original data packet from R3 (ICMP echo) and tries to send aresponse back. The problem is that the destination IP address for the echo reply is “10.0.3.3”and the next-hop is “10.0.0.3”, which has “glean” CEF adjacency. Again, R2 replies backacross the hub and send a Resolution Request packet: first, directly R3 – this attempt fails –then it sends the resolution request to the NHS.

Rack1R2#NHRP: MACADDR: if_in null netid-in 0 if_out Tunnel0 netid-out 123NHRP: Checking for delayed event 0.0.0.0/10.0.0.3 on list (Tunnel0).NHRP: No node found.NHRP: Sending packet to NHS 10.0.0.1 on Tunnel0NHRP: Checking for delayed event 0.0.0.0/10.0.0.3 on list (Tunnel0).NHRP: No node found.NHRP: Attempting to send packet via DEST 10.0.0.3NHRP: Send Resolution Request via Tunnel0 vrf 0, packet size: 81 src: 10.0.0.2, dst: 10.0.0.3 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth src-stable", reqid: 1012 src NBMA: 150.1.2.2 src protocol: 10.0.0.2, dst protocol: 10.0.0.3 (C-1) code: no error(0) prefix: 0, mtu: 1514, hd_time: 360 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0NHRP: Encapsulation failed for destination 10.0.0.3 out Tunnel0NHRP: Attempting to send packet via NHS 10.0.0.1NHRP: Encapsulation succeeded. Tunnel IP addr 150.1.1.1

Rack1R2#NHRP: Send Resolution Request via Tunnel0 vrf 0, packet size: 81 src: 10.0.0.2, dst: 10.0.0.1 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth src-stable", reqid: 1012 src NBMA: 150.1.2.2 src protocol: 10.0.0.2, dst protocol: 10.0.0.3 (C-1) code: no error(0) prefix: 0, mtu: 1514, hd_time: 360 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0NHRP: 81 bytes out Tunnel0


18 of 47 1/5/2011 12:40 AM

NHRP: MACADDR: if_in null netid-in 0 if_out Tunnel0 netid-out 123NHRP: Checking for delayed event 0.0.0.0/10.0.0.3 on list (Tunnel0).NHRP: No node found.NHRP: Sending packet to NHS 10.0.0.1 on Tunnel0

R3 finally receive the Resolution Reply from the NHS, and now it may complete the CEFadjacency for “10.0.0.2”. Since that moment, it switches all packets to “10.0.2.2” directly viaR2, not across R1.

Rack1R3#NHRP: Receive Resolution Reply via Tunnel0 vrf 0, packet size: 109 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth dst-stable unique src-stable", reqid: 994 src NBMA: 150.1.3.3 src protocol: 10.0.0.3, dst protocol: 10.0.0.2 (C-1) code: no error(0) prefix: 32, mtu: 1514, hd_time: 342 addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 0 client NBMA: 150.1.2.2 client protocol: 10.0.0.2NHRP: netid_in = 0, to_us = 1NHRP: Checking for delayed event 150.1.2.2/10.0.0.2 on list (Tunnel0).NHRP: No node found.NHRP: No need to delay processing of resolution event nbma src:150.1.3.3 nbma dst:150.1.2.2

The resolution request that R2 sent before in attempted to resolve the NBMA address for“10.0.0.3” arrives to R1. Since the NHS has all the information in its local cache (R3registered its IP to NBMA address mapping) it immediately replies to R2. Note the CIE headerin the NHRP reply packet, which contains the actual mapping information.

Rack1R1#NHRP: Receive Resolution Request via Tunnel0 vrf 0, packet size: 81 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth src-stable", reqid: 1012 src NBMA: 150.1.2.2 src protocol: 10.0.0.2, dst protocol: 10.0.0.3 (C-1) code: no error(0) prefix: 0, mtu: 1514, hd_time: 360 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0NHRP: netid_in = 123, to_us = 0NHRP: NAT-check: matched destination address 150.1.2.2NHRP: nhrp_rtlookup yielded Tunnel0NHRP: Tu0: Found and skipping dynamic multicast mapping NBMA: 150.1.2.2NHRP: netid_out 123, netid_in 123NHRP: nhrp_cache_lookup_comp returned 0x848EF9E8NHRP: Attempting to send packet via DEST 10.0.0.2NHRP: Encapsulation succeeded. Tunnel IP addr 150.1.2.2

Rack1R1#NHRP: Send Resolution Reply via Tunnel0 vrf 0, packet size: 109 src: 10.0.0.1, dst: 10.0.0.2 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth dst-stable unique src-stable", reqid: 1012 src NBMA: 150.1.2.2 src protocol: 10.0.0.2, dst protocol: 10.0.0.3 (C-1) code: no error(0) prefix: 32, mtu: 1514, hd_time: 242 addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 0


19 of 47 1/5/2011 12:40 AM

client NBMA: 150.1.3.3 bclient protocol: 10.0.0.3NHRP: 109 bytes out Tunnel0

At last, R2 receive the reply to its original request, and now it has all the information tocomplete the CEF entry for “10.0.0.3” and switch packets across the optimal path to R3. Atthis moment both spokes have symmetric information to reach each other

Rack1R2#NHRP: Receive Resolution Reply via Tunnel0 vrf 0, packet size: 109 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth dst-stable unique src-stable", reqid: 1012 src NBMA: 150.1.2.2 src protocol: 10.0.0.2, dst protocol: 10.0.0.3 (C-1) code: no error(0) prefix: 32, mtu: 1514, hd_time: 242 addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 0 client NBMA: 150.1.3.3 client protocol: 10.0.0.3NHRP: netid_in = 0, to_us = 1NHRP: Checking for delayed event 150.1.3.3/10.0.0.3 on list (Tunnel0).NHRP: No node found.NHRP: No need to delay processing of resolution event nbma src:150.1.2.2 nbma dst:150.1.3.3

Timing out NHRP entries

Now that we know that CEF resolves the next-hop information via NHRP, how does it time-outthe unused cut-through tunnel? As we remember, each NHRP entry has countdown expiretimer, initialized from the registration hold-time. Every 60 seconds global NHRP process runson a router and checks the expire timer on all NHRP entries. If the expire timer for an NHRPentry is greater than 120 seconds, nothing is done to the corresponding CEF entry. If thetimer is less than 120 seconds, the NHRP process marks the corresponding CEF entry as“stale” but still usable. As soon as the router switches an IP packet using the “stale” entry, ittriggers new NHRP resolution request, and eventually refreshes the corresponding NHRPentry as well as CEF entry itself. If no packet hits the “stale” CEF entry, the NHRP mappingwill eventually time-out (since the router does not send any “refreshing” requests) and thecorresponding CEF entry will become invalid. This will effectively tear down the spoke-to-spoke tunnel.

NHRP Phase 2 Conclusions

Let us quickly recap what we learned so far about NHRP Phase 2 and CEF. Firstly, this moderequires all the spokes to have complete routing information with the next-hop preserved.This may limit scalability in large networks, since not all spokes may accept full load ofrouting updates. Secondly, CEF only resolve the next-hop information via NHRP, not the fullrouting prefixes. Actually, the second feature directly implies the first limitation. As we noted,the no ip next-hop-self eigrp 123 command is required to make spoke-to-spoke tunnelswork with CEF. However, they added the command only in IOS version 12.3. Is there a way tomake spoke-to-spoke tunnels work when the next-hop is set to “self” (the default) in EIGRPupdates? Actually, there are few ways. First and the best one – do not use old IOS images toimplement DMVPN Actually, it is better to use the latest 12.4T train images with DMVPNPhase 3 for the deployment – but then again those images are from the “T”-train! OK, so the


20 of 47 1/5/2011 12:40 AM

other option is get rid of EIGRP and use OSPF, with the network type “broadcast”. OSPF is alink-state protocol – it does not hide topology information and does not mask the next-hop inany way (well, at least when the network-type is “broadcast”). However, the limitation is thatthe corresponding OSPF topology may have just two redundant hubs – corresponding toOSPF DR and BDR for a segment. This is because every hub must form OSPF adjacencieswith all spokes. Such limitation is not acceptable in large installations, but still works fine insmaller deployments. However, there is one final workaround, which is probably the one youmay want to use in the current CCIE lab exam – disable CEF on spokes. This is a veryinteresting case per se, and we are going to see now NHRP works with process switching.

NHRP Phase 2 + EIGRP next-hop-self + no CEF

In this scenario, EIGRP next-hop self is enabled on R1 (the hub). Now R3 sees 10.0.2.0/24with the next hop of R1. Disable CEF on R2 and R3, and try pinging 10.0.2.2 off R3 loopback1interface.

R3 sees the route behind R2 as reachable via R1

Rack1R3#show ip route 10.0.2.2Routing entry for 10.0.2.0/24 Known via "eigrp 123", distance 90, metric 310172416, type internal Redistributing via eigrp 123 Last update from 10.0.0.1 on Tunnel0, 00:09:55 ago Routing Descriptor Blocks: * 10.0.0.1, from 10.0.0.1, 00:09:55 ago, via Tunnel0 Route metric is 310172416, traffic share count is 1 Total delay is 1005000 microseconds, minimum bandwidth is 9 Kbit Reliability 255/255, minimum MTU 1472 bytes Loading 1/255, Hops 2

R3 pings “10.0.2.2”, sourcing packet off “10.0.3.3”. Since CEF is disabled, the systemperforms NHRP lookup to find the NBMA address for “10.0.2.2”. This is opposed to CEFbehavior that would only resolve the next-hop for “10.0.2.2″ entry. Naturally, the routerforwards NHRP request to R3’s NHS, which is R1. At the same time, R3 forwards the datapacket (ICMP echo) via its current next-hop – “10.0.0.1”, that is via the hub.

Rack1R3#NHRP: MACADDR: if_in null netid-in 0 if_out Tunnel0 netid-out 123NHRP: Checking for delayed event 0.0.0.0/10.0.2.2 on list (Tunnel0).NHRP: No node found.NHRP: Sending packet to NHS 10.0.0.1 on Tunnel0NHRP: Checking for delayed event 0.0.0.0/10.0.2.2 on list (Tunnel0).NHRP: No node found.NHRP: Attempting to send packet via DEST 10.0.2.2NHRP: Encapsulation succeeded. Tunnel IP addr 150.1.1.1NHRP: Send Resolution Request via Tunnel0 vrf 0, packet size: 81 src: 10.0.0.3, dst: 10.0.2.2 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth src-stable", reqid: 900 src NBMA: 150.1.3.3 src protocol: 10.0.0.3, dst protocol: 10.0.2.2 (C-1) code: no error(0) prefix: 0, mtu: 1514, hd_time: 360 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0NHRP: 81 bytes out Tunnel0


21 of 47 1/5/2011 12:40 AM

NHRP: MACADDR: if_in null netid-in 0 if_out Tunnel0 netid-out 123NHRP: Checking for delayed event 0.0.0.0/10.0.2.2 on list (Tunnel0).NHRP: No node found.NHRP: Sending packet to NHS 10.0.0.1 on Tunnel0

Resolution Request arrives to R1 (the NHS). Since R1 has no mapping for “10.0.2.2” (R2 onlyregisters the IP address 10.0.0.2 – its own next-hop IP address), the NHS looks up intorouting table, to find the next-hop towards 10.0.2.2. Since it happens to be R2’s IP “10.0.0.2”,the NHS then tries to forward the resolution request towards the next router on the path tothe network requested in resolution message – to R2. Thanks to R2’s NHRP registration withR1, the NHS now knows R2’s NBMA address, and successfully encapsulates the packet. Inaddition, R1 forwards the data packet from R1 to R2, using its routing table. Obviously, thedata packet will arrive to R2 a little bit faster, since NHRP requires more time to process andforward the request.

Rack1R1#NHRP: Receive Resolution Request via Tunnel0 vrf 0, packet size: 81 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth src-stable", reqid: 900 src NBMA: 150.1.3.3 src protocol: 10.0.0.3, dst protocol: 10.0.2.2 (C-1) code: no error(0) prefix: 0, mtu: 1514, hd_time: 360 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0NHRP: netid_in = 123, to_us = 0NHRP: NAT-check: matched destination address 150.1.3.3NHRP: nhrp_rtlookup yielded Tunnel0NHRP: Tu0: Found and skipping dynamic multicast mapping NBMA: 150.1.3.3NHRP: netid_out 123, netid_in 123NHRP: nhrp_cache_lookup_comp returned 0x0NHRP: Attempting to send packet via DEST 10.0.2.2NHRP: Encapsulation succeeded. Tunnel IP addr 150.1.2.2

NHRP: Forwarding Resolution Request via Tunnel0 vrf 0, packet size: 101 src: 10.0.0.1, dst: 10.0.2.2 (F) afn: IPv4(1), type: IP(800), hop: 254, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth src-stable", reqid: 900 src NBMA: 150.1.3.3 src protocol: 10.0.0.3, dst protocol: 10.0.2.2 (C-1) code: no error(0) prefix: 0, mtu: 1514, hd_time: 360 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0NHRP: 101 bytes out Tunnel0

Now the data packet (ICMP echo) has arrived to R2. R2 generates the response (ICMP –echo reply from “10.0.2.2” to “10.0.3.3”) and now R2 needs the NMBA address of “10.0.3.3”(CEF is disabled on R2). As usual, R2 generates a resolutions request to its NHS (R1). At thesame time, R2 sends the response packet to R3’s request across the hub, since it does notknow the NBMA address of R3.

Rack1R2#NHRP: Send Resolution Request via Tunnel0 vrf 0, packet size: 81 src: 10.0.0.2, dst: 10.0.3.3 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth src-stable", reqid: 919


22 of 47 1/5/2011 12:40 AM

src NBMA: 150.1.2.2 src protocol: 10.0.0.2, dst protocol: 10.0.3.3 (C-1) code: no error(0) prefix: 0, mtu: 1514, hd_time: 360 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0NHRP: 81 bytes out Tunnel0

Soon after the data packet arrived, R2 receives the Resolution Request from R3 forwarded byR1. Since R2 is the egress router on NBMA segment for the network “10.0.2.2”, it may replyto the request.

Rack1R2#NHRP: Receive Resolution Request via Tunnel0 vrf 0, packet size: 101 (F) afn: IPv4(1), type: IP(800), hop: 254, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth src-stable", reqid: 900 src NBMA: 150.1.3.3 src protocol: 10.0.0.3, dst protocol: 10.0.2.2 (C-1) code: no error(0) prefix: 0, mtu: 1514, hd_time: 360 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0NHRP: netid_in = 123, to_us = 0NHRP: nhrp_rtlookup yielded Loopback1NHRP: netid_out 0, netid_in 123NHRP: We are egress router for target 10.0.2.2, recevied via Tunnel0NHRP: Redist mask now 1NHRP: Attempting to send packet via DEST 10.0.0.3NHRP: Encapsulation succeeded. Tunnel IP addr 150.1.3.3

Note that R2 replies with the full prefix found in its routing table – “10.0.2.0/24”, not just singlehost “10.0.2.2/32” (this feature is critical for DMVPN Phase 3). This information isencapsulated inside “(C-1)” part of the NHRP reply packet (Client Information Element 1)which describes a client – network connected to the router (R2). The “prefix” field is “/24”which is exactly the value taken from the routing table.

Also note, that R2 learned R3’s NBMA address from the Resolution Request, and now repliesdirectly to R3, bypassing R1. The “stable” flag means that the querying/replying routerdirectly knows the source or destination IP address in the resolution request/reply.

Rack1R2#NHRP: Send Resolution Reply via Tunnel0 vrf 0, packet size: 129 src: 10.0.0.2, dst: 10.0.0.3 <-- NBMA addresses of R2/R3 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth dst-stable unique src-stable", reqid: 900 src NBMA: 150.1.3.3 src protocol: 10.0.0.3, dst protocol: 10.0.2.2 (C-1) code: no error(0) prefix: 24, mtu: 1514, hd_time: 360 addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 0 client NBMA: 150.1.2.2 client protocol: 10.0.2.2NHRP: 129 bytes out Tunnel0

At this moment, Resolution Request from R2 for network “10.0.3.3″ reaches R1 – the NHS.Since the NHS has no information on “10.0.3.3″, it forwards the request to R3 – the next-hopfound via the routing table on path to “10.0.3.3″.


23 of 47 1/5/2011 12:40 AM

Rack1R1#NHRP: Receive Resolution Request via Tunnel0 vrf 0, packet size: 81 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth src-stable", reqid: 919 src NBMA: 150.1.2.2 src protocol: 10.0.0.2, dst protocol: 10.0.3.3 (C-1) code: no error(0) prefix: 0, mtu: 1514, hd_time: 360 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0NHRP: netid_in = 123, to_us = 0NHRP: NAT-check: matched destination address 150.1.2.2NHRP: nhrp_rtlookup yielded Tunnel0NHRP: Tu0: Found and skipping dynamic multicast mapping NBMA: 150.1.2.2NHRP: netid_out 123, netid_in 123NHRP: nhrp_cache_lookup_comp returned 0x0NHRP: Attempting to send packet via DEST 10.0.3.3NHRP: Encapsulation succeeded. Tunnel IP addr 150.1.3.3

NHRP: Forwarding Resolution Request via Tunnel0 vrf 0, packet size: 101 src: 10.0.0.1, dst: 10.0.3.3 (F) afn: IPv4(1), type: IP(800), hop: 254, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth src-stable", reqid: 919 src NBMA: 150.1.2.2 src protocol: 10.0.0.2, dst protocol: 10.0.3.3 (C-1) code: no error(0) prefix: 0, mtu: 1514, hd_time: 360 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0NHRP: 101 bytes out Tunnel0

Back to R3. At this point, it received the ICMP reply for the original ICMP echo packet. NowR3 receives the NHRP Resolution Reply to its original Resolution Request directly from R2.This allows R3 to learn that “10.0.2.0/24” is reachable via NMBA IP address “150.1.2.2”. Notethat CIE field “(C-1)” in the reply packet, which tells R3 about the whole “10.0.2.0/24” network– the “prefix” is set to “24”.

Rack1R3#NHRP: Receive Resolution Reply via Tunnel0 vrf 0, packet size: 129 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth dst-stable unique src-stable", reqid: 900 src NBMA: 150.1.3.3 src protocol: 10.0.0.3, dst protocol: 10.0.2.2 (C-1) code: no error(0) prefix: 24, mtu: 1514, hd_time: 360 addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 0 client NBMA: 150.1.2.2 client protocol: 10.0.2.2NHRP: netid_in = 0, to_us = 1NHRP: NAT-check: matched destination address 150.1.2.2NHRP: Checking for delayed event 150.1.2.2/10.0.2.2 on list (Tunnel0).NHRP: No node found.NHRP: No need to delay processing of resolution event nbma src:150.1.3.3 nbma dst:150.1.2.2NHRP: Checking for delayed event 0.0.0.0/10.0.2.2 on list (Tunnel0).NHRP: No node found.

Finally, the Resolution Request from R2, forwarded by R1 (the NHS) arrives to R3. The localrouter performs lookup for 10.0.3.3 and finds this to be directly connected network, with theprefix of /24. Therefore, R3 generates a Resolution Reply packet and sends it directly to R2,


24 of 47 1/5/2011 12:40 AM

bypassing R1. This packet tells R2 to map logical IP “10.0.3.0/24” to NBMA address“150.1.3.3”.

Rack1R3#NHRP: Receive Resolution Request via Tunnel0 vrf 0, packet size: 101 (F) afn: IPv4(1), type: IP(800), hop: 254, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth src-stable", reqid: 919 src NBMA: 150.1.2.2 src protocol: 10.0.0.2, dst protocol: 10.0.3.3 (C-1) code: no error(0) prefix: 0, mtu: 1514, hd_time: 360 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 0NHRP: netid_in = 123, to_us = 0NHRP: nhrp_rtlookup yielded Loopback1NHRP: netid_out 0, netid_in 123NHRP: We are egress router for target 10.0.3.3, recevied via Tunnel0NHRP: Redist mask now 1NHRP: Attempting to send packet via DEST 10.0.0.2NHRP: Encapsulation succeeded. Tunnel IP addr 150.1.2.2

NHRP: Send Resolution Reply via Tunnel0 vrf 0, packet size: 129 src: 10.0.0.3, dst: 10.0.0.2 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth dst-stable unique src-stable", reqid: 919 src NBMA: 150.1.2.2 src protocol: 10.0.0.2, dst protocol: 10.0.3.3 (C-1) code: no error(0) prefix: 24, mtu: 1514, hd_time: 360 addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 0 client NBMA: 150.1.3.3 client protocol: 10.0.3.3NHRP: 129 bytes out Tunnel0

At last, R2 receives the response to its Resolution Request, and everything is stable now. R2and R3 know how to reach “10.0.3.0/24” and “10.0.2.0/24” respectively.

Rack1R2#NHRP: Receive Resolution Reply via Tunnel0 vrf 0, packet size: 129 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "router auth dst-stable unique src-stable", reqid: 919 src NBMA: 150.1.2.2 src protocol: 10.0.0.2, dst protocol: 10.0.3.3 (C-1) code: no error(0) prefix: 24, mtu: 1514, hd_time: 360 addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 0 client NBMA: 150.1.3.3 client protocol: 10.0.3.3NHRP: netid_in = 0, to_us = 1NHRP: NAT-check: matched destination address 150.1.3.3NHRP: Checking for delayed event 150.1.3.3/10.0.3.3 on list (Tunnel0).NHRP: No node found.NHRP: No need to delay processing of resolution event nbma src:150.1.2.2 nbma dst:150.1.3.3NHRP: Checking for delayed event 0.0.0.0/10.0.3.3 on list (Tunnel0).NHRP: No node found.

Now let’s look at NHRP caches of all three routers:

Rack1R1#show ip nhrp


25 of 47 1/5/2011 12:40 AM

10.0.0.2/32 via 10.0.0.2, Tunnel0 created 01:00:47, expire 00:04:02 Type: dynamic, Flags: authoritative unique registered NBMA address: 150.1.2.210.0.0.3/32 via 10.0.0.3, Tunnel0 created 01:00:47, expire 00:04:23 Type: dynamic, Flags: authoritative unique registered NBMA address: 150.1.3.3

Rack1R2#show ip nhrp10.0.0.1/32 via 10.0.0.1, Tunnel0 created 01:56:30, never expire Type: static, Flags: authoritative used NBMA address: 150.1.1.110.0.0.3/32 via 10.0.0.3, Tunnel0 created 00:00:24, expire 00:05:35 Type: dynamic, Flags: router implicit NBMA address: 150.1.3.3 10.0.2.0/24 via 10.0.2.2, Tunnel0 created 00:00:24, expire 00:05:35 Type: dynamic, Flags: router authoritative unique local NBMA address: 150.1.2.2 (no-socket)10.0.3.0/24 via 10.0.3.3, Tunnel0 created 00:00:24, expire 00:05:35 Type: dynamic, Flags: router NBMA address: 150.1.3.3

Rack1R3#show ip nhrp10.0.0.1/32 via 10.0.0.1, Tunnel0 created 01:56:00, never expire Type: static, Flags: authoritative used NBMA address: 150.1.1.110.0.0.2/32 via 10.0.0.2, Tunnel0 created 00:00:02, expire 00:05:57 Type: dynamic, Flags: router implicit used NBMA address: 150.1.2.210.0.2.0/24 via 10.0.2.2, Tunnel0 created 00:00:02, expire 00:05:57 Type: dynamic, Flags: router used NBMA address: 150.1.2.210.0.3.0/24 via 10.0.3.3, Tunnel0 created 00:00:02, expire 00:05:57 Type: dynamic, Flags: router authoritative unique local NBMA address: 150.1.3.3 (no-socket)

The “implicit” flag means that the router learned mapping without explicit request, as a part ofother router’s reply or request. The “router” flag means that the mapping is either for theremote router or for a network behind the router. The “(no-socket)” flag means that the localrouter will not use this entry and trigger IPSec socket creation. The “local” flag means themapping is for the network directly connected to the local router. The router uses thosemappings when it loses connection to the local network, so that the NHC may send a purgerequest to all other clients, telling that the network has gone and they must remove theirmappings.

Here is an example. Ensure R3 has the above-mentioned mappings, and then shut down theLoopback1 interface, observing the debugging command output on R3 and R2. R3 sendspurge request directly to R2, since it knows R2 requested that mapping.

Rack1R3#NHRP: Redist callback: 10.0.3.0/24NHRP: Invalidating map tables for prefix 10.0.3.0/24 via Tunnel0NHRP: Checking for delayed event 150.1.3.3/10.0.3.3 on list (Tunnel0).NHRP: No node found.NHRP: Attempting to send packet via DEST 10.0.0.2NHRP: Encapsulation succeeded. Tunnel IP addr 150.1.2.2NHRP: Send Purge Request via Tunnel0 vrf 0, packet size: 73 src: 10.0.0.3, dst: 10.0.0.2


26 of 47 1/5/2011 12:40 AM

(F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "reply required", reqid: 36 src NBMA: 150.1.3.3 src protocol: 10.0.0.3, dst protocol: 10.0.0.2 (C-1) code: no error(0) prefix: 0, mtu: 1514, hd_time: 0 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 0 client protocol: 10.0.3.3NHRP: 73 bytes out Tunnel0

R2 receives Purge Request from R3. Note that the “reply required” flag is set. Hence, R2must confirm that it deleted the mapping with a Purge Reply packet. R2 will erase thecorresponding mapping learned via “10.0.0.3” and generate a response packet

Rack1R2#NHRP: Receive Purge Request via Tunnel0 vrf 0, packet size: 73 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "reply required", reqid: 36 src NBMA: 150.1.3.3 src protocol: 10.0.0.3, dst protocol: 10.0.0.2 (C-1) code: no error(0) prefix: 0, mtu: 1514, hd_time: 0 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 0 client protocol: 10.0.3.3NHRP: netid_in = 123, to_us = 1NHRP: Attempting to send packet via DEST 10.0.0.3NHRP: Encapsulation succeeded. Tunnel IP addr 150.1.3.3

R2 first tries to send the Purge Reply to R3 directly, using the NBMA address of R3. Note thatCIE header mentions the network erased from the local mappings list

Rack1R2#NHRP: Send Purge Reply via Tunnel0 vrf 0, packet size: 73 src: 10.0.0.2, dst: 10.0.0.3 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "reply required", reqid: 36 src NBMA: 150.1.3.3 src protocol: 10.0.0.3, dst protocol: 10.0.0.2 (C-1) code: no error(0) prefix: 0, mtu: 1514, hd_time: 0 addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 0 client protocol: 10.0.3.3NHRP: 73 bytes out Tunnel0NHRP: Invalidating map tables for prefix 10.0.3.0/24 via Tunnel0NHRP: Attempting to send packet via DEST 10.0.0.1NHRP: Encapsulation succeeded. Tunnel IP addr 150.1.1.1

R3 receives the reply to its purge request and now it knows that R2 is consistent.

Rack1R3#NHRP: Receive Purge Reply via Tunnel0 vrf 0, packet size: 73 (F) afn: IPv4(1), type: IP(800), hop: 255, ver: 1 shtl: 4(NSAP), sstl: 0(NSAP) (M) flags: "reply required", reqid: 36 src NBMA: 150.1.3.3 src protocol: 10.0.0.3, dst protocol: 10.0.0.2 (C-1) code: no error(0) prefix: 0, mtu: 1514, hd_time: 0


27 of 47 1/5/2011 12:40 AM

addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 0 client protocol: 10.0.3.3NHRP: netid_in = 0, to_us = 1

Timing out NHRP entries with Process-Switching

The last question is how NHRP times out unused entries in case of process-switching mode.Recall the “used” flag set for NHRP mapping. Every time a packet is process-switched usingthe respective NHRP entry, it is marked as “used”. The background NHRP process runsevery 60 seconds, and check the expire timers for each NHRP entry. If the “used” flag is setand expire timer for the entry is greater than 120 seconds then the process clears the flag(and every new packet will refresh it). If the timer is less than 120 seconds and the flag is set,IOS generates a refreshing NHRP request. However, if the flag is not set, the system allowsthe entry to expire, unless another packet hits it and makes active.

The above-described behavior of NHRP with process switching allows for one interestingfeature. The hub router may now summarize all information sent down to spokes say into onedefault route. This will not affect the spokes, for they will continue querying next-hopinformation for every destination prefix sent over the mGRE tunnel interface, and learning theoptimal next-hop. It would be great to combine this “summarization” feature with theperformance of CEF switching. This is exactly what they implemented with DMVPN Phase 3.However, Phase 3 is subject to a separate discussion.

Integrating IPsec

Haven’t we forgotten something for DMVPN Phase 1/Phase 2? That was IPsec, thecomponents that provides confidentiality and integrity checking to mGRE/NHRP. Now,compared with the complexity of NHRP operations, IPsec integration is straightforward.

First, the hub needs to know how to authentication all the spokes using IKE. The mostscalable way is to use X.509 certificates and PKI, but for the simplicity, we will just use thesame pre-shared key on all routers. Note that we need to configure the routers with awild-card pre-shared key, in order to accept IKE negotiation requests from any other dynamicpeer.

As for IPsec Phase 2, we need dynamic crypto maps there, since the hub has no idea of theconnecting peer IP addresses. Fortunately, Cisco IOS has a cute feature called IPsecprofiles, designed for use with tunnel interfaces. The profile attaches to a tunnel interface andautomatically considers all traffic going out of the tunnel as triggering the IPsec Phase 2. TheIPsec phase proxy identities used by the IPsec profile are the source and destination host IPaddresses of the tunnel. It makes sense to use IPSec transport mode with mGRE as thelatter already provides tunnel encapsulation. Besides, IOS supports some features, like NATtraversal only with IPSec transport mode.

Let’s review an example below and explain how it works.

mGRE + NHRP Phase 2 + Spoke-to-spoke tunnels + IPsec

R1:crypto isakmp policy 10 encryption 3des


28 of 47 1/5/2011 12:40 AM

authentication pre-share hash md5 group 2!crypto isakmp key 0 CISCO address 0.0.0.0 0.0.0.0!crypto ipsec transform-set 3DES_MD5 esp-3des esp-md5-hmac mode transport!crypto ipsec profile DMVPN set transform-set 3DES_MD5!interface Tunnel 0 tunnel protection ipsec profile DMVPN

R2 & R3:

crypto isakmp policy 10 encryption 3des authentication pre-share hash md5 group 2!crypto isakmp key 0 CISCO address 0.0.0.0 0.0.0.0!crypto ipsec transform-set 3DES_MD5 esp-3des esp-md5-hmac mode transport!crypto ipsec profile DMVPN set transform-set 3DES_MD5!interface Tunnel 0 tunnel protection ipsec profile DMVPN

Start with any spoke, e.g. R3. Since the router uses EIGRP on Tunnel 0 interface, a multicastpacket will eventually be send out of the tunnel interface. Thanks to the static NHRP multicastmapping, mGRE will encapsulate the EIGRP packet towards the hub router. The IPsec profilewill see GRE traffic going from “150.1.3.3” to “150.1.1.1”. Automatically, ISAKMP negotiationwill start with R1, and authentication will use pre-shared keys. Eventually both R1 and R3 willcreate IPsec SAs for GRE traffic between “150.1.3.3” and “150.1.1.1”. Now R3 may sendNHRP resolution request. As soon as R3 tries to send traffic to a network behind R2, it willresolve next-hop “10.0.0.2” to the IP address of 150.1.2.2. This new NHRP entry will triggerISAKMP negotiation with NBMA address 150.1.2.2 as soon as router tries to use it for packetforwarding. IKE negotiation between R3 and R2 will start and result in formation of new SAscorresponding to IP address pair “150.1.2.2 and 150.1.3.3” and GRE protocol. As soon as therouters complete IPsec Phase 2, packets may flow between R2 and R3 across the shortcutpath.

When an unused NHRP entry times out, it will signal the ISAKMP process to terminate therespective IPsec connection. We described the process for timing out NHRP entries before,and as you remember, it depends on the “hold-time” value set by the routers. Additionally, thesystems may expire ISAKMP/IPsec connections due to IPsec timeouts.

This is the crypto system status on the hub from the example with NHRP Phase 2 andprocess-switching:


29 of 47 1/5/2011 12:40 AM

IPsec Phase 1 has been established with both spokes

Rack1R1#show crypto isakmp sadst src state conn-id slot status150.1.1.1 150.1.2.2 QM_IDLE 1 0 ACTIVE150.1.1.1 150.1.3.3 QM_IDLE 3 0 ACTIVE

IPsec Phase 2 SA entries for both protected connections to R2 and R3 follows. Note that SAsare for GRE traffic between the loopback.

Rack1R1#show crypto ipsec sa

interface: Tunnel0 Crypto map tag: Tunnel0-head-0, local addr 150.1.1.1

protected vrf: (none) local ident (addr/mask/prot/port): (150.1.1.1/255.255.255.255/47/0) remote ident (addr/mask/prot/port): (150.1.2.2/255.255.255.255/47/0) current_peer 150.1.2.2 port 500 PERMIT, flags={origin_is_acl,} #pkts encaps: 230, #pkts encrypt: 230, #pkts digest: 230 #pkts decaps: 227, #pkts decrypt: 227, #pkts verify: 227 #pkts compressed: 0, #pkts decompressed: 0 #pkts not compressed: 0, #pkts compr. failed: 0 #pkts not decompressed: 0, #pkts decompress failed: 0 #send errors 12, #recv errors 0

local crypto endpt.: 150.1.1.1, remote crypto endpt.: 150.1.2.2 path mtu 1514, ip mtu 1514, ip mtu idb Loopback0 current outbound spi: 0x88261BA3(2284198819)

inbound esp sas: spi: 0xE279A1EE(3799622126) transform: esp-3des esp-md5-hmac , in use settings ={Transport, } conn id: 2001, flow_id: SW:1, crypto map: Tunnel0-head-0 sa timing: remaining key lifetime (k/sec): (4472116/2632) IV size: 8 bytes replay detection support: Y Status: ACTIVE spi: 0xB4F6A9E5(3036064229) transform: esp-3des esp-md5-hmac , in use settings ={Transport, } conn id: 2003, flow_id: SW:3, crypto map: Tunnel0-head-0 sa timing: remaining key lifetime (k/sec): (4596176/2630) IV size: 8 bytes replay detection support: Y Status: ACTIVE spi: 0x1492E4D0(345171152) transform: esp-3des esp-md5-hmac , in use settings ={Transport, } conn id: 2005, flow_id: SW:5, crypto map: Tunnel0-head-0 sa timing: remaining key lifetime (k/sec): (4525264/2630) IV size: 8 bytes replay detection support: Y Status: ACTIVE

inbound ah sas:

inbound pcp sas:

outbound esp sas: spi: 0x81949874(2173999220) transform: esp-3des esp-md5-hmac ,


30 of 47 1/5/2011 12:40 AM

in use settings ={Transport, } conn id: 2002, flow_id: SW:2, crypto map: Tunnel0-head-0 sa timing: remaining key lifetime (k/sec): (4472116/2626) IV size: 8 bytes replay detection support: Y Status: ACTIVE spi: 0xAA5D21A7(2858230183) transform: esp-3des esp-md5-hmac , in use settings ={Transport, } conn id: 2004, flow_id: SW:4, crypto map: Tunnel0-head-0 sa timing: remaining key lifetime (k/sec): (4596176/2627) IV size: 8 bytes replay detection support: Y Status: ACTIVE spi: 0x88261BA3(2284198819) transform: esp-3des esp-md5-hmac , in use settings ={Transport, } conn id: 2006, flow_id: SW:6, crypto map: Tunnel0-head-0 sa timing: remaining key lifetime (k/sec): (4525265/2627) IV size: 8 bytes replay detection support: Y Status: ACTIVE

outbound ah sas:

outbound pcp sas:


local crypto endpt.: 150.1.1.1, remote crypto endpt.: 150.1.3.3 path mtu 1514, ip mtu 1514, ip mtu idb Loopback0 current outbound spi: 0xBEB1D9CE(3199326670)

inbound esp sas: spi: 0x10B44B31(280251185) transform: esp-3des esp-md5-hmac , in use settings ={Transport, } conn id: 2007, flow_id: SW:7, crypto map: Tunnel0-head-0 sa timing: remaining key lifetime (k/sec): (4436422/2627) IV size: 8 bytes replay detection support: Y Status: ACTIVE

inbound ah sas:

inbound pcp sas:

outbound esp sas: spi: 0xBEB1D9CE(3199326670) transform: esp-3des esp-md5-hmac , in use settings ={Transport, } conn id: 2008, flow_id: SW:8, crypto map: Tunnel0-head-0 sa timing: remaining key lifetime (k/sec): (4436424/2627) IV size: 8 bytes replay detection support: Y


31 of 47 1/5/2011 12:40 AM

Status: ACTIVE

outbound ah sas:

outbound pcp sas:

Now let’s see how a spoke router establishes a spoke-to-spoke IPsec tunnel:

No NHRP mapping for spoke’s network first

Rack1R3#sh ip nhrp10.0.0.1/32 via 10.0.0.1, Tunnel0 created 02:02:42, never expire Type: static, Flags: authoritative used NBMA address: 150.1.1.1

ISAKMP negotiated just with R1

Rack1R3#sh crypto isakmp sadst src state conn-id slot status150.1.1.1 150.1.3.3 QM_IDLE 1 0 ACTIVE

Generate traffic to network behind R2. Note that the first ping passes through, since it’srouted across the hub, but the second packet is sent directly to R2 and is missed, since IPsecPhase 2 has not yet been established

Rack1R3#ping 10.0.2.2

Type escape sequence to abort.Sending 5, 100-byte ICMP Echos to 10.0.2.2, timeout is 2 seconds:!.!!!Success rate is 80 percent (4/5), round-trip min/avg/max = 52/121/324 ms

Notice the new NHRP mappings. Note that the tunnel will expire in about 3

minutes, if no new traffic is going to be generated

Rack1R3#sh ip nhrp 10.0.0.1/32 via 10.0.0.1, Tunnel0 created 02:05:38, never expire Type: static, Flags: authoritative used NBMA address: 150.1.1.110.0.2.0/24 via 10.0.2.2, Tunnel0 created 00:02:44, expire 00:03:15 Type: dynamic, Flags: router NBMA address: 150.1.2.2

IOS create IPsec Phase 2 SAs for tunnels between R2-R3 and R1-R3. The tunnel between 2and R3 is dynamic and is used to send only the data traffic.

Rack1R3#show crypto isakmp sadst src state conn-id slot status150.1.1.1 150.1.3.3 QM_IDLE 1 0 ACTIVE150.1.3.3 150.1.2.2 QM_IDLE 2 0 ACTIVE

Rack1R3#show crypto ipsec sa

interface: Tunnel0


32 of 47 1/5/2011 12:40 AM

Crypto map tag: Tunnel0-head-0, local addr 150.1.3.3


local crypto endpt.: 150.1.3.3, remote crypto endpt.: 150.1.1.1 path mtu 1514, ip mtu 1514, ip mtu idb Loopback0 current outbound spi: 0x10B44B31(280251185)

inbound esp sas: spi: 0xBEB1D9CE(3199326670) transform: esp-3des esp-md5-hmac , in use settings ={Transport, } conn id: 2001, flow_id: SW:1, crypto map: Tunnel0-head-0 sa timing: remaining key lifetime (k/sec): (4526856/2383) IV size: 8 bytes replay detection support: Y Status: ACTIVE

inbound ah sas:

inbound pcp sas:

outbound esp sas: spi: 0x10B44B31(280251185) transform: esp-3des esp-md5-hmac , in use settings ={Transport, } conn id: 2002, flow_id: SW:2, crypto map: Tunnel0-head-0 sa timing: remaining key lifetime (k/sec): (4526853/2381) IV size: 8 bytes replay detection support: Y Status: ACTIVE

outbound ah sas:

outbound pcp sas:


local crypto endpt.: 150.1.3.3, remote crypto endpt.: 150.1.2.2 path mtu 1514, ip mtu 1514, ip mtu idb Loopback0 current outbound spi: 0x847D8EEC(2222821100)

inbound esp sas: spi: 0xA6851754(2793740116) transform: esp-3des esp-md5-hmac ,


33 of 47 1/5/2011 12:40 AM

in use settings ={Transport, } conn id: 2004, flow_id: SW:4, crypto map: Tunnel0-head-0 sa timing: remaining key lifetime (k/sec): (4602306/3572) IV size: 8 bytes replay detection support: Y Status: ACTIVE

inbound ah sas:

inbound pcp sas:

outbound esp sas: spi: 0x847D8EEC(2222821100) transform: esp-3des esp-md5-hmac , in use settings ={Transport, } conn id: 2003, flow_id: SW:3, crypto map: Tunnel0-head-0 sa timing: remaining key lifetime (k/sec): (4602306/3572) IV size: 8 bytes replay detection support: Y Status: ACTIVE

outbound ah sas:

outbound pcp sas:

Now you see how all the component of DMVPN work together. We have not covered someother major topics like NAT traversal with NHRP and DMVPN redundancy with multiple hubs.Those advanced topics probably require a separate post, since this one has grown too bigalready

Tags: ccie, cef, dmvpn, dynamic, eigrp, ipsec, multipoint, NHRP, ospf

Download this page as a PDF

About Petr Lapukhov, 4xCCIE/CCDE:

Petr Lapukhov's career in IT begain in 1988 with a focus on computer programming, andprogressed into networking with his first exposure to Novell NetWare in 1991. Initially involvedwith Kazan State University's campus network support and UNIX system administration, hewent through the path of becoming a networking consultant, taking part in many networkdeployment projects. Petr currently has over 12 years of experience working in the Cisconetworking field, and is the only person in the world to have obtained four CCIEs in under twoyears, passing each on his first attempt. Petr is an exceptional case in that he has been


34 of 47 1/5/2011 12:40 AM

working with all of the technologies covered in his four CCIE tracks (R&S, Security, SP, andVoice) on a daily basis for many years. When not actively teaching classes, developingself-paced products, studying for the CCDE Practical & the CCIE Storage Lab Exam, andcompleting his PhD in Applied Mathematics.

Find all posts by petr | Visit Website

You can leave a response, or trackback from your own site.

48 Responses to “DMVPN Explained”

DMVPN Explained � says:August 2, 2008 at 5:49 am

[...] News » News News DMVPN Explained2008-08-02 08:49:46Have adjacencies withall spokes. Such limitation is that the limitation is [...]

1.

agalang says:August 2, 2008 at 7:27 am

fantastic explanation..been waiting for this for a long time

2.

annon says:August 2, 2008 at 7:57 am

It wasH brilliant!!!

3.

DMVPN Explained � says:August 2, 2008 at 9:21 am

[...] & RAS Servers News » News News DMVPN Explained2008-08-0212:21:49Endpoints endpoints need to exchange unicast packets, special “glue” is used[...]

4.

zeus says:August 2, 2008 at 1:24 pm

Petr is freaking awesome!!!!! Brians how did u find this guy!!!

5.

ccieat21 says:August 2, 2008 at 8:45 pm

Great Post .. looking forward for advanced DMVPN posts

6.


35 of 47 1/5/2011 12:40 AM

Vijay Tyagi says:August 2, 2008 at 8:53 pm

You Rocks!!! Petr.

I think IT industry will remember you forever for your great contributions.

Regards

7.

joseph brunnerr says:August 3, 2008 at 3:02 pm

I take issue with part of the above Petr!

I quote:

“Since all packets are routed across the hub, almost any dynamic routing protocol wouldhelp with attaining reachability. The hub just needs to advertise a default route to spokes,while spokes should advertise their subnets dynamically to the hub. Probably the bestchoice is to run EIGRP and summarize all subnets to 0.0.0.0/0 on the hub, effectivelysending a default route to all spokes.”

As dmvpn is commonly used in a hq/branch office scenario, typically the spokes have adefault route towards their ISP; this is used to reach websites locally, and one or morehub public ip addressesH If the spokes learn a default route via a dynamic routingprotocol from the hub, they will not use it. They will most likely use the default static routetowards the local isp next hop address/outgoing serial interface. Also if then they are notusing the default route they will not learn any of the hub’s subnets, or the other spokessubnets for DMVPN phase 1 (no spoke to spoke routing) via the DMVPN.

Could you please clarify this part in DMVPN Phase 1 regarding having the hub advertisea default route to the spokesH i have never done it or seen it done this way.

thanks,

Joe#19366

8.

Brandon Carroll says:August 3, 2008 at 6:32 pm

I Love it! Perfect timing too. I would however like the ability to grab a PDF of this

9.

Petr Lapukhov, CCIE #16379 says:August 3, 2008 at 8:35 pm

10.


36 of 47 1/5/2011 12:40 AM

To: Joseph Brunner

Joe, naturally if you use a form of “split-tunneling” on spokes, you would not like toadvertise default route fom the hub. Thanks for the point, I will add a few wordscommentions on that in the post.

However, my idea is that with Phase 1, it does not matter whether you send detailed orsummarized information from the hub – spoke to spoke tunnels are not used. Sendingdefault route is just an extremal form of summarization, and you may resort to sendingsay 2-3 routes describing the corporate subnets.

It’s the same idea as with ezVPN split-tunneling – you either tunnel everything or tunneljust specifics.


To: Brandon Caroll

Yeah, a few other people also asked for PDF version I’m thinking on uploading thePDF paper and providing a link in the post.

11.

no ip split-horizon/. says:August 3, 2008 at 11:06 pm

[...] If you’re new here, you may want to subscribe to my RSS feed. Thanks for visiting!Ihad to take 2 minutes and blog this. Last night I set up DMVPN. It all looked ok but Icould not get the spoke to spoke connectivity. Then today I find Petr Lupakhovs –DMVPN Explained blog post on CCIE Blog at Internetwork expert. (You can find it here)[...]

12.

JC says:August 7, 2008 at 7:25 am

Outstanding and thorough explanation! I recently set up a fairly complex DMVPN solutionand thought I understood the technology pretty well. I am humbled. It would be great ifyou could expand this to include Phase III and multiple hub redundancy. That way thiscould be the definitive guide to NHRP/DMVPN!

13.

MBR says:August 11, 2008 at 9:54 am

This is great, Petr. This morning I went searching for dmvpn info. Your paper is by far thebest that I found, in a class by itself.

14.

11 August - CCIE Quickies « CCIE Pursuit Blog says:August 11, 2008 at 2:39 pm

15.


37 of 47 1/5/2011 12:40 AM

[...] appetite and you have a few free hours, you should surf over to Internetwork ExpertBlog and read Petr’s magnum opus on the same topic. Possibly related posts:(automatically generated)Study StrategyIf I Knew You Were Comin’ [...]

Review Notes: Cisco Lifecycle Services ARSFE (Design Phase) « CarlProject says:August 17, 2008 at 12:29 am

[...] DMVPN design development (Here is a tutorial of DMVPN by Petr Lapukhov ofInternetwork Expert) [...]

16.

Baby, You Can Route My World! » Blog Archive » Back In the Swing says:August 19, 2008 at 11:42 am

[...] trying to learn everything I can about DMVPNs, IPSECs, GREs, etc. I’ve gone over anexcellent DMVPN article by Petr Lapukhov of InternetworkExpert as well as JeremyStretch of PacketLife.net’s clear explanations. But [...]

17.

Cisco Guy says:September 12, 2008 at 12:34 am

DMVPN is great for transferring multicast traffic securely. I have used this solution in astock exchange scenario where brokers get real-time ticker updates for stocks, which isactually multicast traffic. GRE here with the added security of IPSEC is an excellentcombination.

18.

Petr Lapukhov, CCIE #16379 says:September 12, 2008 at 12:37 am

To: Cisco Guy

Have you tried Cisco’s GET VPN technology for Secure Multicast in production?

19.

DMVPN Study « cciESolutions says:September 17, 2008 at 1:20 pm

[...] DMVPN Explained [...]

20.

l0k1 says:September 17, 2008 at 5:27 pm

hi my name is Daniel & I´m from Mexico, there´s an issue about my nhrpimplementation, I revised the RFC to figured out what´s wrong whit this..

there are some things that tell me that my nhs is not responding my resolutions requestsbut this occurs only whit some spokes,,, is there any limit on the server cache orsomething like that??

21.


38 of 47 1/5/2011 12:40 AM

this flags are saw on one of the spokes.12.12.13.254/32, Tunnel12 created 00:00:51, expire 00:02:13Type: incomplete, Flags: negativeCache hits: 2

Cisco Guy says:November 20, 2008 at 9:55 pm

To: Petr

Sorry for the late replyHno unfortunately I have not used GET VPN yet for securemulticast traffic.. It looks interesting technology though

22.

andri says:December 22, 2008 at 1:20 am

first of all, this is a real great deal of explaination, i salute u .

the problem is , i cant get my icmp packet be encrypted from spoke to spoke. I knowsomething must have gone wrong and i am still trying to search the problem.

I will be very please if you described the hardware configuration for this topology since ibelieve my problem is in the hardware.

waiting for your sincere reply

23.

DMVPN Phase 3 - CCIE Blog says:December 23, 2008 at 6:25 am

[...] DMVPN Phase 3 By Petr Lapukhov, CCIE #16379 In this post we are going to speakmainly of NHRP. The other important part of DMVPN – IPsec – is relatively the same,and did not change with introduction of NHRP Phase 3. To begin with, let’s quickly recallthe core features of NHRP Phase 1 & 2. For detailed overview, you may refer to DMVPNExplained [...]

24.

Dmvpn links « Reload In 5 says:December 29, 2008 at 7:00 am

[...] dmvpn, ios Collection of interesting dmvpn articles and video links: –Internetworkexpert.com phase 1 & 2 phase [...]

25.

Hamood says:December 31, 2008 at 2:43 am

Petr,

Can you give an example for NHRP with OSPF and why we have to use the database

26.


39 of 47 1/5/2011 12:40 AM

filter-all out command. Most of the sample configurations I saw are using the EIGRP.

James says:February 5, 2009 at 8:04 am

Great post.I get help from this post.I really appreciate to author.

27.

ttongfly says:April 5, 2009 at 8:58 pm

Thank you for post. I fixed problem on my LAB.

28.

Security « CCIE Study Notes says:May 12, 2009 at 1:03 am

[...] explained here but i’m out of steam for security today Comments [...]

29.

ob says:August 1, 2009 at 8:27 am

Great article!

I have two questions though:

1) How do I get spoke-to-spoke multicasting to work. Even the simple “ping 224.0.0.1″ onspokeA will not get a reply from spokeB, only the HUB.

2) How do I set up redundant NHS?

30.


1) You need to use PIM-NBMA mode on the hub tunnel interface to allow for neighbortracking and splitting the NBMA interface into a group of P2P links. This of course onlyworks with PIM sparse.2) With Phase 2, you need to configure every spoke with two NHS addresses andconfigure both NHSes to point to each other. This will ensure proper NHS cachesyncrhonization.2) With Phase 3, everythin should be working automatically, even if you have multiplespokes registering to different hubs, the chain of redirects is based on the routingtopology, not the NHS connections.

31.

Naru says:September 14, 2009 at 6:50 am

32.


40 of 47 1/5/2011 12:40 AM

Very nice explanation, keep the good work up!

Sinisa says:September 22, 2009 at 3:05 pm

This is a grat explanation of DMVPN, the best one I could find on the Internet.

Question: I have DMVPN topology with two redundant NHSs (pointing each other) andseveral NHCs, and that topology works fine. Problem is when I shut NHS#1 (or itcrashes) and bring it back, and then the same with NHS#2, my NHCs can’t see eachother any more. They even can’t see NHSs, and NHSs can see only each other. I canonly solve it by shut/no shut tunnel interfaces on NHCs.Is there a rule or a behaviour of NHRP that NHSs have to be configured first in network,then NHCs?

I run multi area OSPF over multiple GRE tunnels, and it’s all protected with IPSec. I haveremoved IPSec protection and OSPF in observed mGRE tunnel to have a clear situation.

Any help?

33.

Naru says:October 22, 2009 at 4:48 am

I am currently using Orion SolarWinds NPM to monitor my nodes, though I needed tomonitor the tunnels when they go up & down so I can send the report to my ISP, but Isaw that even if the tunnel is not connected, it still shows as UP – UP, is there a way tochange this behavior? For example, when you can’t reach that tunnel, the tunnel shouldgo like UP – DOWN , or if there’s any other method to monitor those tunnels would begreat.

Thanks

34.

Sinisa says:December 9, 2009 at 3:03 am

@Naru

You can use ip sla traps and snmp.

35.

Zen says:December 28, 2009 at 6:08 am

Hello,

I implemented DMVPN a while ago, and used to work great.Recently I noticed memory & cpu usages going very high, and when I checked with show

36.


41 of 47 1/5/2011 12:40 AM

processes cpu , I saw that NHRP was using 40-60% of the CPU constantly.

I ran show ip nhrp , and the results really frightened me, there are lots of unknown ipaddress in there , same is happening on my Hub and on Spokes.

Here’s one of the many entries:

97.100.x.x/32 via 192.168.x.1Tunnel0 created 00:00:03, expire 00:05:58Type: dynamic, Flags: router

There are many other different IP addresses that look like the one above, any idea whatsgoing on?

Ali says:January 20, 2010 at 2:44 am

Can I implement a Laptop installed with Cisco VPN client as a DMVPN Spoke Site, LIkefor eg. I have an employee working from a remote location and has internet connectivityand needs to access other spokes too , directly without the traffic being sent from thelaptop to the hub and then to the other remote site, instead a dynamic tunnel needs to becreated from the Laptop to the other spoke?

Is it possible to implement DMVPN/NHRP on a Cisco VPN client?

37.

Zen says:February 22, 2010 at 3:01 am

About my above post, i think it could help someone on the same foot.

After implementing the following command I havent noticed anymore weird nhrpregisters, and also we changed the address which crypto should allow to make a requestfor registers.

crypto isakmp aggressive-mode disablecrypto isakmp key 0 CISCO address [b]ActualIPAdress[/b]

38.

Periodic Failure » Single Spoke Dynamic Multipoint VPN (DMVPN) says:April 3, 2010 at 8:05 pm

[...] getting a fully working configuration, I stumbled across an excellent blog post by PetrLapukhov DMVPN Explained it goes through the configuration stage by stage with greatinsight on how its all put together. I [...]

39.

Sleep-1 , Cisco-32423 « TacAck – My security journey! says:May 11, 2010 at 7:10 pm

[...] , i want to do DMVPN today. There’s an awesome article on DMVPN written by Petr

40.


42 of 47 1/5/2011 12:40 AM

Lapukhov on blog.ine.com . I’m going to go through that as soon as i’m [...]

Jan says:June 5, 2010 at 2:36 pm

Good job Petr.Thanx for article.Can u add “sh ip route” (all routes) prints of NHRP Phase 2 devices?

41.

Roland says:June 27, 2010 at 9:06 am

Should I expect DMVPN on R&S CCIE lab? Can’t find it in the blueprint.

42.

Anthony Sequeira, #15626 says:June 28, 2010 at 5:41 am

@Ronald

DMVPN is tested heavily on the Security CCIE Track, but is no longer part of the Writtenor Lab materials for R&S

43.

EP says:July 16, 2010 at 9:17 pm

to: Petr Lapukhov

What a great article!!! Very well defineH Sir, hope any new technologies would comefrom youH. all hard to learn from cisco books are well explained from youH very wellinstructorH Thank you so muchH

Damn!!! I’m reading all your articlesH When I start reading it, I can’t free my sight notfinishing itH

44.

tjames says:September 8, 2010 at 3:30 pm

Guys

i’ve been bashing my head around this problem for a few days please help ip nhrp network-id this is used to turn on nhrp on an interface and to provide the logicalmembership to a certain DMVPN cloud, otherwise stated each DMVPN cloud shouldhave it’s unique network idWhat i did in my lab was to have each spoke and the hub in different DMVPN cloudsmeaning spoke one had network-id 1 , spoke 2 had network-id 2, hub had network-id 3

45.


43 of 47 1/5/2011 12:40 AM

etc and it seemed to work just fine so i dont understand the use of “ip nhrp network-id” ifwith different network-ids it works

Thanks

tjames says:September 10, 2010 at 10:51 am

Hi Petr

Can you provide some insights into the above “dilema” ?

Thanks a lot

46.

roger sambora says:October 21, 2010 at 6:50 am

Petr:

Excellent doc !! Better than any Cisco doc

Would you be putting out something more including IPSec VTI too?

BTW: not able to download the Phase 2 doc as Adobe? but Phase 3 docs downloadsjust fine.

Thx !!

Roger

47.

Mohinder Singh Rawat says:December 1, 2010 at 7:44 pm

Hi Petr,

Thanks a lot for your wonderful post. I was just checking your learning curve..4 CCIE in two years and pursuing Phd in Applied Mathematics..Awesome.

Mahi

48.

Leave a Reply

Name (required)

Mail (will not be published) (required)


44 of 47 1/5/2011 12:40 AM

Website

Search the Blog

Categories

Current Poll

How did you first hear about INE?

FriendGoogleForum/Mailing List

BlogFacebook/Twitter

YoutubeOther

Vote

View Results

Polls Archive

CCIE Bloggers

Brian Dennis CCIE #2210Routing & SwitchingISP DialSecurityService ProviderVoice

Brian McGahan CCIE #8593Routing & SwitchingSecurity


45 of 47 1/5/2011 12:40 AM

Service ProviderPetr Lapukhov CCIE #16379

Routing & SwitchingSecurityService ProviderVoice

Anthony Sequeira CCIE #15626Routing & Switching

Mark Snow CCIE #14073VoiceSecurity

Josh Finke CCIE #25707Voice

Bootcamps

November 15 - 19MockLab WorkshopLocation: London, UKEnroll NowDecember 5 - 10Mock Lab WorkshopLocation: London, UKEnroll NowDecember 6 - 1712-Day BootcampLocation: Reno, NVEnroll Now

Popular Posts

The INE Online Community Thanks - Deepak Arora"Self-Paced" Version of Mastering SNMP is CompleteUnderstanding Inter-Area Loop Prevention Caveats in OSPF ProtocolThe Basics of EIGRPFollow Anthony of INE on Twitter

twitter.com/inetraining

642-642 QoS Bootcamp now priced at just $395. Find out how you can get it for just$100 for a limited time: http://bit.ly/giwtAy #qos

Purchase select exam or technology bootcamps and get one free, such as the newnetwork services bootcamp! http://bit.ly/giwtAy

The Basics of EIGRP http://bit.ly/hngVGB


46 of 47 1/5/2011 12:40 AM

Follow

© 2010 Internetwork Expert, Inc., All Rights Reserved


47 of 47 1/5/2011 12:40 AM

Documents

DMVPN Explained _ CCIE Blog