1 of 6

DISRUPTION TOLERANT NETWORKING FOR MARINE CORPS CONDOR

Salil Parikh, Robert C. Durst The MITRE Corporation

McLean, VA

ABSTRACT

We present an overview of our work in applying Disrup-tion Tolerant Networking technology to the US Marine Corps CONDOR (C2 On-the-move Network Digital Over-the-horizon Relay) project, a near-term framework for ex-tending and bridging tactical data networks that are parti-tioned, for instance, by distance or terrain features. A portion of the CONDOR system involves deploying mobile platforms that can use satellite links when terrestrial radio networks are out of range, in order to maintain situational awareness and connectivity to tactical data networks and applications.

Disruption Tolerant Networking technology leverages the work of the Internet Research Task Force (IRTF) Delay Tolerant Networking Research Group (DTNRG). We dis-cuss our plan to enhance CONDOR with products of the DTNRG, present a proof-of-concept demonstration that incorporates DTNRG software into CONDOR field-prototype payload hardware, and discuss application de-sign considerations for operation in a networking envi-ronment prone to disruption.

INTRODUCTION

The recently proposed Delay Tolerant Networking [1,2,3] architecture aims at supporting efficient message-based communication over networks challenged by long delays, frequent partitioning, and bandwidth limitations. Delay Tolerant Networking achieves this by introducing an over-lay network (Figure 1) that interoperates across disparate underlying networking technologies, providing a common message-switching application interface, routing through volatile network connectivity, and reliable delivery ser-vices. In the context of tactical networks, we use the term Disruption Tolerant Networking, as this is the proper name of a closely related DARPA program (launched in 2005), and because it is more descriptive of the motivating char-acteristic of tactical environments. As the two efforts are intertwined, we use the acronym DTN here to refer to both Delay and Disruption Tolerant Networking.

A key strategy of DTN is to incrementally move messages closer to the destination in a store-and-forward manner using local storage at intermediate nodes, instead of rely-

ing upon an end-to-end stable path, to ultimately deliver messages with reduced latency and higher throughput. To support this, a primary research focus of DTN is the de-velopment of routing methods to handle node mobility and disconnection, and links that are on-demand, opportunistic (like a UAV flyover), scheduled (e.g. satellite), or pre-dicted.

Following brief overviews of DTN and CONDOR, we de-scribe the insertion of DTNRG software into the CONDOR system. We then discuss the issues of interfac-ing existing applications to DTN by introducing DTN ap-plication proxies.

Application

DTN

Transport

Network

Link

Phy

DTN

Transport

Network

Link

Phy

Network

Link

Phy

Region1: Tactical Data Network Region3:

Terrestrial Wired Network

Transport

Network

Link

Phy

Application

DTN

Transport

Network

Link

Phy

DTNGateway

DTNGateway

Region2: Satellite

DTN

Transport

Network

Link

Phy

Transport

Network

Link

Phy

Figure 1: DTN Overlay Network

OVERVIEW OF DTN IN TACTICAL NETWORKS

DTN addresses the gap between the vision for IP network-ing in the battlefield and the reality of frequent disruptions in tactical communication environments. DTN is an ap-proach toward networking that does not presume the avail-ability of a contemporaneous end-to-end path. It has at its basis a message-based store-and-forward approach, aug-mented with a technique called custody transfer, in which in-network nodes assume retransmission responsibility for data in transit. This relieves the source of retransmission responsibility, but more importantly in a tactical environ-ment, reduces the load that retransmission places on the network, because retransmissions don’t re-cover links that have already been successfully traversed.

With DTN, data advances toward the destination whenever an adjacent link is available, irrespective of whether the path is available end-to-end. Data latency is lower, links are more effectively used throughout the network, and more data gets to the destination sooner. Any retransmis-

2 of 6

sions that may be required are not requested of the source, but rather of the DTN node at which the data currently re-sides. By advancing the point of retransmission toward the destination, we reduce the number of hops over which a retransmission must propagate, avoiding the delay and diminution of network capacity that results.

OVERVIEW OF CONDOR

In response to the need of providing and maintaining a common operational picture, situational awareness, and application access to Marine units moving quickly out of range of their tactical radios, the Marine Corps has devel-oped CONDOR — Command and Control On-the-Move Network Digital Over-the-Horizon Relay — a near-term bridging strategy to link existing tactical radio and data networks, and to provide an over-the-horizon communica-tions capability to link line-of-sight radio systems that have moved beyond line-of-sight or that are precluded by terrain features or other obstacles.

CONDOR is intended to be a migratory approach in an-ticipation of the arrival of the Joint Tactical Radio System (JTRS), providing near-term secure mobile networking on the battlefield to connect mobile forces such as maneuver-ing commanders, armor units, and logistical convoys to the Joint C2 Intranet and SIPR/NIPR networks. The CONDOR Architecture defines three realizations: a Gate-way, a Point of Presence Vehicle, and a Jump C2 Vehicle [8].

The CONDOR Gateway is a vehicle equipped to connect EPLRS communities, extending communications beyond line-of-sight by bridging EPLRS networks through a commercial or military satellite service. It contains an EPLRS radio, a C2PC (Command & Control PC) gateway, and an over-the-horizon connection via a satellite service such as INMARSAT, Iridium, or TACSAT.

Figure 2 shows a CONDOR Gateway prototype unit, cur-rently being deployed for use and for field validation. The payload consists of a Cisco 3725 router, an EPLRS radio, a link encryption module, a Cisco access router, and a satel-lite communications terminal, like an INMARSAT termi-nal. While current CONDOR units contain use INMARSAT, future revisions may upgrade to other tech-nologies. Technicians mount the components in a transit case for installation in a HMMWV, similar to that shown in the photo.

Figure 2: CONDOR prototype payload

Cat-5 ethernet

Cat-5 ethernet

Cat-5 ISDN

Cat-5 ethernet

Cisco 3725

Cisco 2621 INMARSATterminal

KG-235

EPLRS

Figure 3: CONDOR Gateway wiring diagram

The CONDOR Point of Presence Vehicle (PoP-V) con-nects legacy radios to the data network, either by routing this traffic to a CONDOR Gateway via EPLRS, or possi-bly by using a low-bandwidth satellite like TACSAT di-rectly for over-the-horizon connectivity.

A CONDOR Jump C2 Vehicle (JC2-V) functions as a mo-bile command post, maintaining synchronization with servers and databases during maneuvers. It will have a continuous satellite link. Command vehicles use wireless networking technology to access the JC2-V.

In each of these variants, the over-the-horizon connection includes commercial and military satellite services, such as INMARSAT, MILSATCOM, Ku and Ka band, or UAV overhead relays.

Our work examines the scenario where CONDOR Gate-way nodes are used to restore connectivity to a Tactical Operations Center (TOC) when line-of-sight limits are ex-ceeded, by creating a point-to-point link via satellite be-tween the mobile unit and a TOC which has stable connec-tions to tactical networks and applications. Each link thus

3 of 6

requires a pair of CONDOR units: The mobile CONDOR unit establishes a satellite connection to a peer unit located at the TOC or command post; mobile units do not connect to each other in a peer-to-peer manner. Traffic between CONDOR-connected islands passes through the TOC; thus, the routing decision within a mobile CONDOR unit is tantamount to switching between the EPLRS radio and the satellite connection.

DTN FOR CONDOR

DTN for CONDOR focuses on making DTN technology usable by Marines in the field to address the challenges present in tactical networks: host and router mobility; ter-rain and environmental interference; intentional interfer-ence like jamming; intermittent connectivity; lower data rates; and long latency.

DTN provides a message store-and-forward infrastructure, pushing the retransmission point closer to the ultimate des-tination, instead of end-to-end retransmission. Since it is an overlay protocol which uses existing transports, DTN aids in building a heterogeneous system and facilitates joint application interoperability.

DTN is useful for data transfers in which the time-value of the information exceeds the duration of the disruption. Therefore, DTN will likely improve the delivery of im-agery, logistics information, and communications like email. However, it is uncertain if DTN can improve posi-tion/location information or chat applications because of the potential for high delays. A feature of the store-and-forward nature of the DTN overlay is that packet loss is prevented even while EPLRS connectivity is lost and the INMARSAT path is not yet available. Messages are not dropped or discarded during the transition phase.

DTN is defined as an overlay network that can exist at any logical node in a network; however, because the CONDOR nodes are on the edge of the disrupted network, and because they form bridges over the disrupted network, it is natural to place DTN functions in the CONDOR units themselves, thus adding a DTN gateway capability to them. This gives the DTN agents visibility into the state of the networks, and serves as a logical place to install prox-ies, since the CONDOR node is the ingress point into the disrupted network. Conceptually, we want to implant DTN agent software into the unit’s Cisco 3725 router, al-lowing a tight coupling between the DTN agent and the router’s configuration. We considered several approaches to develop this, all of them requiring introducing general processing capability into the CONDOR unit.

Ideally, the DTN agent software would run natively as a component of the router’s IOS, thereby adding DTN func-

tionality without additional hardware or power require-ments. Cisco is developing an experimental version of the IOS that permits executing third-party Java code – such as a DTN agent – as a process of the IOS. Similarly, DTN application proxies could be written in Java and executed directly on the router. However, this approach raises many concerns. First, it would likely run into performance is-sues, as the processing and memory in a router is insuffi-cient for the applications that are envisioned to run in a DTN. There is not enough available non-volatile memory accessible to the router's operating system to provide store-and-forward for substantial outages or for caching web content (see [7]). Second, porting the DTN Reference Im-plementation code to this Java environment is a nontrivial project. Third, this capability is not yet a released product, and the availability of a Java Virtual Machine in a router might well be seen by network administrators as a security vulnerability. We have set this approach aside for the time being.

A second approach is to add a general computer to the CONDOR payload to run the DTN agent and associated applications. This could be a thin 1U server, or compara-ble unit like a laptop computer or embedded system board, connected to an Ethernet port on the 3725 router. Client applications local to this gateway would connect to a DTN agent running on this server for access to DTN services. The server would also run proxies for web-browsing, e-mail, IRC, or tactical applications. This approach has the benefit of not needing any specialized porting of the DTN agent or proxies to the Java IOS environment. If desired, the IOS can be configured to transparently redirect appli-cation traffic by using port-based access control lists and policy routing, and the DTN agent can manipulate the router’s IOS via the secure shell (ssh) or secure HTTP (https) interfaces of the IOS, thus creating the illusion that DTN services are running internal to the router. With this approach, memory and storage pose less of a constraint, but it requires additional payload hardware and power. In the current CONDOR prototype configuration, there is not a slot or space available in the chassis to add an external server; so while this alternative remains viable in future configurations, we are not pursuing it at the moment.

In our recent work, we have developed a prototype solu-tion using a third approach that is conceptually similar to the server approach. The Cisco 3725 router belongs to a family of routers that has slots for expansion modules. One of these slots on the CONDOR prototype payload’s 3725 router is vacant. Cisco has an intrusion detection system (IDS) module (NM-CIDS-K9) product that is de-signed for one of the network module slots. This module is, essentially, a single board computer for the router’s backplane, containing a Pentium III processor, 512 MB of memory, a 20 GB disk, and an internal Ethernet connec-

4 of 6

tion to the router on the backplane. The module is loaded with a customized Linux operating system, on which the module’s IDS software runs as an application.

Installing our DTN software (agent and application prox-ies) on the IDS module is a straightforward exercise. We compiled the DTNRG’s Reference Implementation [12] on a workstation loaded with a similar version of Linux, and transferred the DTN executables and supporting libraries to the IDS module. For our experiments with CONDOR, this module provides a convenient, non-obtrusive, and po-tentially fieldable integration vector with an actual CONDOR payload. As the module is a general-purpose computer, we can also include a Space Communication Protocol Standard Transport Protocol (SCPS-TP) perform-ance enhancing proxy (PEP) to improve performance on low-capacity high-delay satellite connections. While the processor will likely be sufficient for most proxy applica-tions, disk space may be insufficient for applications that exchange large quantities of image data, especially if it is cached, as by a web proxy.

Disrupted LAN

Internet

Cisco 3745

INMARSATterminal

INMARSAT

Web Client

Cisco 3725

INMARSATterminal

Figure 4: Demonstration Configuration

Figure 4 shows our point-to-point mockup of the CONDOR prototype to illustrate operations on a parti-tioned network. We have a Cisco 3725 router outfitted with an IDS module loaded with the DTN agent software and a web browser proxy [7]. In place of an EPLRS radio, we use an Ethernet LAN connection to another similarly configured router. Each router has a serial card to inter-face to an ISDN Service Unit, which is then connected to an INMARSAT terminal.

The right side of the diagram represents a “forward ad-vancing” unit that outruns its line-of-sight EPLRS connec-tion to the left side, a Tactical Operations Center (TOC) that has application servers and stable connections to net-works (e.g., SIPR/NIPR). Upon losing the EPLRS link (simulated by interrupting the LAN), an operator then ac-tivates the satellite connection. We do this manually, as it is our understanding that this is in fact the field procedure because dial-on-demand is hampered by the security de-

vices that are in-line before the satellite terminal. (De-pending on the variant, the satellite connection might be “on-demand” or “always-on” as in a J2C-V.) During the transition, outgoing traffic is queued on the IDS module, awaiting the availability of the satellite link.

Users attached to the gateway configure their web brows-ers to use the browser proxy located on the IDS module. Page requests are repackaged by the proxy and delivered across the Ethernet connection to the peer proxy on the “remote side” of the EPLRS link. When the EPLRS con-nection is not available, the DTN agent uses the INMARSAT link as the over-the-horizon link to sustain connectivity.

Even the INMARSAT link is prone to disruption as a re-sult of maneuvering across uneven terrain [9]. In the event of a network partition, the DTN node in the IDS module (or, for that matter, other DTN nodes deployed in the field) will store the request or response and search for an avail-able path to forward it toward its destination. In the cur-rent configuration, there are few DTN nodes that exist in the path of a data transfer (two for mobile node to/through TOC, and three for mobile node to mobile node via the TOC). Even so, use of DTN's store and forward capability prevents data transfers from failing when the network be-comes partitioned.

The preceding paragraph implies that the satellite link is the most prone to loss in the network. This is not always the case, particularly in the case when a Point of Presence vehicle is extending over-the-horizon connectivity to a community of SINCGARS users. Consider data transfers destined for a user that is connected via SINCGARS. If data is to be sent to this user reliably, in a "traditional" en-vironment, retransmissions originate from the source. This has the unfortunate consequence of increasing loading throughout the network (including over the satellite link) in order to compensate for loss on the last link. DTN's store-and-forward capability can isolate these retransmis-sions, protecting the rest of the network from increased loading due to losses in the last-hop links. Further, DTN provides the ability to change transport protocols or trans-port protocol settings over this last hop in order to optimize data transfers for the specific environment. DTN's ability to change transport protocol might be extremely beneficial here, allowing use of alternate retransmission schemes or erasure coding techniques to improve the probability of receipt across these highly error-prone links.

DTN APPLICATION DEVELOPMENT

Ideally, applications would be written, or retrofitted, to be DTN-aware, using the DTN Bundle API [12] instead of directly interfacing to traditional network transport API’s

5 of 6

such as TCP or UDP sockets. Programming using Bundles for networked applications is a similar experience to using UDP, as the Bundle API does not provide a bytestream or virtual circuit abstraction. The essence of DTN applica-tion programming is to express inter-process communica-tions as messages, similar in style to the structure of SOAP-based applications, thereby minimizing round-trip exchanges.

When using Bundles, the application is responsible for recovery from misordered or lost messages, as the Bundle Agent does not buffer and reorder the arriving bundles be-fore delivery to the application, and out-of-order message delivery could conceivably be a common occurrence in disrupted networks. While the Bundle API offers a reli-able delivery service (via a mechanism called custody transfer), applications should still be prepared for the case of bundles discarded in-flight due to expiration.

In many cases, retrofitting an application with Bundles is not practical, thus calling for an application-aware proxy. The proxy terminates the native protocol on the client side, and then expresses the application protocol in bundles for transit through the challenged network. At the receiving end – presumably a location that has a stable connection to the target network – the proxy transforms the bundle back into the application’s native protocol and continues the conversation normally with the application server. For applications that use TCP for transport, the task of design-ing a Bundling proxy is similar to that of designing a proxy that uses UDP, except that Bundle lengths have no practical limitations, and Bundles offer a reliable delivery option. “Chatty” protocols, like SMTP, which have many small round-trip exchanges, are aggregated into a bulk message. In some cases, such as web-browsing, the client-side experience is altered to an off-line style of operation [7].

We are currently investigating adapting deployed tactical applications such as the USMC’s Command and Control on PC (C2PC) and Internet Relay Chat (IRC) to the DTN environment. We are also examining a web-browser based whiteboard and chat application called Lightweight Col-laborative Whiteboard (LCW) currently under develop-ment at MITRE. These applications illustrate proxy de-sign issues because it is impractical to modify the client software to use the DTN API, and because the applications rely upon TCP’s services.

LCW is a browser-based application that uses SOAP mes-sages to query and update a group server. Since the appli-cation runs in a web browser, it is easily directed to use a proxy. The proxy is responsible for encapsulating the messages into bundles and sending responses back to the requesting socket. An issue that arises here is how to han-

dle (or prevent) client-side timeouts when responses are delayed.

S STCP

TCP TCP

S SBundles

TCP TCP

C

C

C

DTN DTN

S CSDTNChallenged region

IRC client

IRC server

IRC server with Bundle interface

C

C

C

C

C

C

C

C

C

(a)

(b)

Figure 5: IRC proxy

Although TCP applications are often message-oriented, most rely upon in-order delivery of those messages. Con-sider the IRC Protocol [13]. IRC clients and servers ex-change delimited text messages over a TCP socket. IRC servers connect to other servers to create a relay network (Figure 5). One way to “proxy” IRC is to build a modified server that has a normal client interface and a server-to-server module that uses Bundles. This permits the users of a server to remain connected to each other during times when the server-to-server connection is disrupted. It also simplifies the handling of certain protocol details, like IRC’s Ping/Pong messages, because only the servers need modification, not the clients. Messages encapsulated in bundles could arrive out of order, causing temporal jum-bling of the chat session. One idea to mitigate this is for the proxies to append a timestamp to the message for users to reference in case of confusion. Depending on the usage environment, the proxy may need to buffer messages dur-ing disconnections, and deliver those messages later when the connection is restored, similar to the function of an IRC bouncer.

CONCLUSIONS AND FURTHER WORK

Disruption Tolerant Networking seeks to deal with the re-alities of military tactical communication. Current DTN research is working to support autoconfiguration, multi-cast, and on-the-fly connectivity planning and optimiza-tion, all of which will significantly enhance the capabilities of DTN in CONDOR and other tactical environments.

Currently, we have ported DTN into the Cisco IDS mod-ule, have supported it with a SCPS-TP based Performance Enhancing Proxy, and have developed a DTN-enabled web proxy to improve web access across disrupted environ-

6 of 6

ments. Our future plans include completing proxies for other key applications, including Internet Relay Chat and a Common Operational Picture application (C2PC, LCW, or similar). In addition, we are developing an embedded ver-sion of the DTN capability that we are integrating with the Cisco 3251 mobile access router in a rugged chassis. This capability will use extended temperature range PC/104 components and non-rotating mass storage to improve re-liability in deployed environments.

There remain other issues which we hope to address in the coming months, as well. The DTN protocol suite defines a multi-layer security approach that involves authenticated forwarding along with end-to-end payload data encryption (or authentication). We need to consider how and whether these optional DTN protocols can be beneficial in the CONDOR environment. Further, in the context of the DTNRG, we are beginning to develop extensions to the main Bundle protocol to support reliable multicast deliv-ery. (Currently, the Bundle protocol supports only unicast transfers.) Determining how to integrate these into the CONDOR environment will be important, as much traffic involved with maintaining a Common Operational Picture is multicast in nature. Finally, we wish to consider how to deal with streaming applications in DTN. Conceptually, DTN and UDP are very similar in this respect, and it is conceivable to adapt UDP streaming such as the Real Time Protocol (RTP) [10] to provide sequenced bundle transfers, with a fail-over to store-and-forward (e.g. "voicemail") operation when the network becomes parti-tioned or has insufficient capacity to support the streamed traffic. This, however, requires a substantial amount of additional effort.

DTN for CONDOR paves the way for introducing DTN into tactical environments and emerging applications, pro-viding an immediate benefit to USMC, and remaining relevant with the arrival of next-generation technologies like JTRS.

The DARPA Disruption Tolerant Networking program and the IRTF DTNRG will offer drop-in results, like the Ref-erence Implementation, which will likely form the basis of commercial DTN products, and lessons learned from CONDOR will be fed back into the DARPA program and IRTF-sponsored research.

ACKNOWLEDGEMENTS

The authors thank Peter Firey, Percy Schmidt, and James Van Guilder, all of the MITRE Corporation, for their dis-cussions on Lightweight Collaborative Whiteboard. Jason Andresen, also of MITRE, is developing an IRC proxy.

REFERENCES

[1] Delay Tolerant Networking Research Group. http://www.DTNRG.org.

[2] V. Cerf, et. al., "Delay Tolerant Network Architecture" (Internet Research Task Force draft document, draft-irtf-dtnrg-arch-03.txt), July 2005.

[3] K. Scott, S. Burleigh, "Bundle Protocol Specification" (Internet Research Task Force draft document, draft-irtf-dtnrg-bundle-spec-03.txt), July 2005.

[4] S. Burleigh, et al., “Delay Tolerant Networking: An Approach to Interplanetary Internet,” IEEE Commu-nications Magazine, June 2003.

[5] K. Fall, “A Delay-Tolerant Network Architecture for Challenged Internets,” Intel Research Berkeley, IRB-TR-03-003, February 2003.

[6] M. Demmer, et al., "Implementing Delay Tolerant Networking," Intel Research Berkeley, IRB-TR-04-020, December 28, 2004

[7] K. Scott, “Disruption tolerant networking Proxies for On-The-Move Tactical Network.” To be presented at MILCOM 2005.

[8] D. Mohney, "Soaring Condor Relieves Headaches," Mobile Radio Technology, 1 June 2004. (http://mrtmag.com/)

[9] STOM BRIDGE Littoral Combat & Future Naval Ca-pability (LC-FNC), August 2003 Report.

[10] Schulzrinne et al, "RTP: A Transport Protocol for Real-Time Applications," RFC 3550, June 2003. http://www.ietf.org/rfc/rfc3550.txt.

[11] K. Fall, “Disruption Tolerant Networking for Hetero-geneous Ad-Hoc Networks.” To be presented at MILCOM 2005.

[12] M. Demmer and DTNRG. DTN Reference Imple-mentation – Version 2. December 2004. Available at http://www.dtnrg.org.

[13] J. Oikarinen, D. Reed, “Internet Relay Chat Protocol”, RFC 1459, May 1993.