Performance Characteristics of Convergence Layers in Delay Tolerant Networks

A thesis presented to

the faculty of

the Russ College of Engineering and Technology of Ohio University

In partial fulfillment

of the requirements for the degree

Master of Science

Mithun Roy Rajan

August 2011

© 2011 Mithun Roy Rajan. All Rights Reserved.

2

This thesis titled

Performance Characteristics of Convergence Layers in Delay Tolerant Networks

by

MITHUN ROY RAJAN

has been approved for

the Electrical Engineering and Computer Science

and the Russ College of Engineering and Technology by

Shawn D. Ostermann

Associate Professor of Engineering and Technology

Dennis Irwin

Dean, Russ College of Engineering and Technology

3

Abstract

RAJAN, MITHUN ROY, M.S., August 2011, Computer Science Engineering

PerformanceCharacteristics of ConvergenceLayersin Delay TolerantNetworks (131pp.)

Director of Thesis: Shawn D. Ostermann

Delay Tolerant Networks (DTNs) are designed to operate in environments with high

delays, significant losses and intermittent connectivity.Internet protocols like TCP/IP and

UDP/IP are not designed to perform effectively in challenging environments. DTN uses

the Bundle Protocol which is an overlay protocol to store andforward data units. This

Bundle Protocol works in cooperation with convergence layers to function in extreme

environments. The convergence layers augment the underlying communication layer to

provide services like reliable delivery and message boundaries. This research focuses on

the kind of performance that can be expected from two such convergence layers - the TCP

Convergence Layer and the Licklider Transmission ProtocolConvergence Layer - under

various realistic conditions. Tests were conducted to calculate the throughput using these

convergence layers under different losses and delays. The throughput that was obtained

using different convergence layers was compared and the performance patterns were

analyzed to determine which of these convergence layers hasa higher performance under

the various scenarios.

Approved:

Shawn D. Ostermann

Associate Professor of Engineering and Technology

4

Acknowledgments

I would like to thank my advisor, Dr. Ostermann, for his endless supply of ideas and

suggestions in helping me complete my thesis. I am grateful to him for giving me the

opportunity to work in the IRG Lab. A very special thanks to Dr. Kruse for his guidance

and for always having answers to all my questions.

Thanks to Gilbert Clark, Josh Schendel, James Swaro, Kevin Janowiecki, Samuel

Jero and David Young for all their help in the lab. It was always great to have someone to

discuss research issues and world issues with.

I appreciate all the love and support from my family. Thanks for being patient and

having faith in me.

5

Table of Contents

Page

Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Acknowledgments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

List of Figures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2 DTN Architecture and Convergence Layers. . . . . . . . . . . . . . . . . . . . 142.1 DTN Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.2 Bundle Protocol. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.3 Convergence Layers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

2.3.1 TCP Convergence Layer. . . . . . . . . . . . . . . . . . . . . . . 222.3.2 LTP Convergence Layer. . . . . . . . . . . . . . . . . . . . . . . 25

3 Experiment Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.1 Hardware Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . 293.2 Software Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4 Experiments, Result and Analysis. . . . . . . . . . . . . . . . . . . . . . . . . 334.1 TCP Convergence Layer Tests. . . . . . . . . . . . . . . . . . . . . . . . 33

4.1.1 TCPCL Without Custody Transfer. . . . . . . . . . . . . . . . . . 334.1.2 TCPCL With Custody Transfer. . . . . . . . . . . . . . . . . . . 44

4.2 LTP Convergence Layer Tests. . . . . . . . . . . . . . . . . . . . . . . . 494.2.1 LTPCL Without custody transfer. . . . . . . . . . . . . . . . . . . 504.2.2 LTPCL With custody transfer. . . . . . . . . . . . . . . . . . . . 60

5 Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

Appendix A: ION Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

Appendix B: Supporting programs. . . . . . . . . . . . . . . . . . . . . . . . . . . 94

6

List of Tables

Page

3.1 Testbed Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.2 Experimental Settings and Parameters. . . . . . . . . . . . . . . . . . . . . . 31

4.1 Number of sessions used for various RTTs. . . . . . . . . . . . . . . . . . . . 53

7

List of Figures

Page

2.1 Bundle Protocol in Protocol Stack. . . . . . . . . . . . . . . . . . . . . . . . 152.2 Bundle Protocol Structure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172.3 Convergence Layer Protocol in Protocol Stack. . . . . . . . . . . . . . . . . . 212.4 TCPCL Connection Lifecycle. . . . . . . . . . . . . . . . . . . . . . . . . . 242.5 LTPCL Connection Lifecycle. . . . . . . . . . . . . . . . . . . . . . . . . . . 272.6 LTPCL Error Connection Lifecycle. . . . . . . . . . . . . . . . . . . . . . . . 28

3.1 Physical Testbed Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.2 Logical Testbed Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4.1 TCPCL Throughput vs RTT. . . . . . . . . . . . . . . . . . . . . . . . . . . 334.2 Outstanding Window Graph for TCP Reno with 2 sec RTT. . . . . . . . . . . 344.3 Outstanding Window Graph for TCP Cubic with 2 sec RTT. . . . . . . . . . . 354.4 Time Sequence Graph of TCP Link with 20sec RTT. . . . . . . . . . . . . . . 374.5 TCPCL Throughput vs Loss. . . . . . . . . . . . . . . . . . . . . . . . . . . 404.6 TCPCL(w/o custody) Throughput vs RTT & Loss for 200 Byte Bundles. . . . 414.7 TCPCL(w/o custody) Throughtput vs RTT & Loss for 5000 Byte Bundles. . . 424.8 TCPCL(w/o custody) Throughput vs RTT & Loss for 15000 Byte Bundles. . . 434.9 TCPCL(w/ custody) Throughtput vs RTT. . . . . . . . . . . . . . . . . . . . 464.10 Comparing TCPCL(w/ custody) and TCPCL(w/o custody) Throughtput vs RTT474.11 TCPCL(w/ custody) Throughput vs Loss. . . . . . . . . . . . . . . . . . . . . 484.12 Comparing TCPCL(w/ custody) and TCPCL(w/o custody) Throughtput vs Loss494.13 TCPCL(w/ custody) Throughput vs RTT & Loss for 200 Byte Bundles. . . . . 504.14 TCPCL(w/ custody) Throughput vs RTT & Loss for 5000 Byte Bundles. . . . 514.15 TCPCL(w/ custody) Throughput vs RTT & Loss for 15000 Byte Bundles. . . 524.16 TCPCL(w/ custody) Throughput vs RTT & Loss for 40000 Byte Bundles. . . 534.17 LTPCL Throughtput vs RTT. . . . . . . . . . . . . . . . . . . . . . . . . . . 544.18 Comparing LTPCL and TCPCL Throughput vs RTT. . . . . . . . . . . . . . 554.19 LTPCL Throughput vs Loss. . . . . . . . . . . . . . . . . . . . . . . . . . . 574.20 Comparing LTPCL and TCPCL Throughput vs Loss. . . . . . . . . . . . . . 594.21 LTPCL Throughput vs RTT & Loss for 5000 Byte Bundles. . . . . . . . . . . 604.22 LTPCL Throughput vs RTT & Loss for 15000 Byte Bundles. . . . . . . . . . 614.23 Comparing LTPCL(w/o custody) and LTPCL(w/ cusotdy) Throughput vs Delay624.24 Comparing LTPCL(w/o custody) and LTPCL(w/ custody) Throughput vs Loss. 634.25 LTPCL(w/ custody) Throughput vs RTT & Loss for 5000 Byte Bundles. . . . 644.26 LTPCL(w/custody) Throughput vs RTT & Loss for 15000 Byte Bundles. . . . 65

8

1 Introduction

Environments with limited network connectivity, high losses and delays require

special networks for communication. Terrestrial networkswhich generally use TCP/IP do

not perform well under such conditions. Deep space exploration, wireless and sensor

networks, etc. operate in such challenging environments and it was important to develop a

solution for such challenging environments. Terrestrial networks can be described as

having the following characteristics:

• delay in the order of milliseconds

• low error rates

• continuous connectivity

• high bandwidth

The protocols that have been developed for terrestrial networks exploit the above

mentioned benefits. For example, TCP performs a three-way handshake between two

communicating entities to establish connection between them. In a high bandwidth

environment with negligible delay, the additional time required to establish a connection

will be insignificant. Furthermore, during a TCP transmission with packet loss, TCP will

block the delivery of subsequent packets until the lost packet is retransmitted and this

would lead to under-utilization of the bandwidth. Since terrestrial networks have

continuous connectivity, under-utilization of the bandwidth will not affect data delivery.

The sender will store the data until it receives an acknowledgement from the receiver

because if any packets are lost during transmission, the lost packets can be retransmitted.

Since the round trip time is in the order of milliseconds, thesender’s retransmission buffer

will only need to store the sent data for a few milliseconds orseconds.

9

On the other end of the spectrum, a challenged network can be defined by the following

characteristics:

• delay in the order of minutes/hours

• high error rates depending on the environmental conditions

• disrupted connectivity

• limited bandwidth

• asymmetric bandwidth

Burleigh et al. [2003] explains why using an Internet protocol on a delayed/disrupted

link is not advisable. When TCP/IP is used in disrupted networks, the connection

establishment by the 3-way handshake will be an overhead. The high round-trip time for

the 3-way handshake will delay the flow of application data. This is unfavorable in an

environment where connection can be frequently disrupted.Moreover, if there is limited

opportunity to send data, using that limited connectivity to establish a connection will be a

misuse of the available bandwidth. Similarly, waiting on lost packets will lead to

under-utilization of the bandwidth. It is important to maximize the usage of bandwidth

when there is connectivity instead of waiting for lost packets. End-to-end TCP

retransmissions will cause the retransmission buffer at the sender’s end to retain the data it

sends for long periods of time. These periods of time can range in length from a few

minutes to hours [Burleigh et al. 2003].

There are a few modified versions of TCP with extensions specifically for deep space

communication, like Space Communications Protocol Standards-Transport Protocol

(SCPS-TP) [Durst et al. 1996], and TCP peach [Akyildiz et al. 2001]. In addition, there

are also some protocols/architectures which were developed for interplanetary

communication, like Satellite Transport Protocol (STP) [Henderson and Katz 1999], and

10

Performance enhancing transport architecture (PETRA) [Marchese et al. 2004]. Most of

these protocols and architecture solve the high delays, losses, and asymmetric bandwidth

problems. Delay/Disruption Tolerant Networks [Fall 2003; Burleigh et al. 2003] is one

such architecture that has been designed to solve the problems posed by such

environments. Delay tolerant networks are deployed for deep space communication

[Wood et al. 2008, p.1], networks for mobile devices and military networks [Fall 2003].

Delay and Disruption Tolerant networks are generally deployed in environments

where the source and destination nodes do not have end-to-end connectivity. In areas with

limited networking infrastructure or areas where communication networks are difficult to

build, it is easier and cheaper to set up Delay Tolerant Networks. Farrell and Cahill [2006]

explains a number of interesting applications of DTNs. Someof these applications are

detailed in the following paragraphs.

Deep-space communication is an excellent example of an environment that deploys

Delay/Disruption Tolerant networks. The round-trip times for space links are on the order

of minutes (Mars is anywhere between 9 to 50 minutes depending on the location of Earth

and Mars on its orbit [Akan et al. 2002]). The communication link between nodes/entities

in space is intermittent because there may be points in time when the nodes are not in line

of sight of each other. Losses are also common in deep space communication because of

low signal strength over large distances.

A good example for DTN is the communication between Mars rovers and ground

stations. A Mars rover could transmit data directly to earthstations but doing so will be

energy-intensive and could drain the power of the rover. Thus, it would be reasonable to

use a relay to transmit the data, so the rover can conserve power. Rovers on Mars collect

data and pictures which need to be transmitted to a ground station on earth. After the rover

collects the data, it will wait until it can make contact withthe Mars orbiter. Therefore, the

rover stores the data until it can forward it on to the orbiter. The rover transfers the data to

11

the orbiter when it can make contact. The Mars orbiter will wait until it is in line of sight

of the Deep Space Network antenna on earth, so it can relay that data from the rover.

Juang et al. [2002] explains another interesting application of Delay Tolerant

Networks called Zebranet. Zebranet tracks the migration ofzebras in central Kenya.

There are a number of options available to track zebras. One option is to have collars for

the zebras that use sophisticated GPS to track their positions and upload the data directly

to a satellite. Since these collars will be powered by solar cells and a satellite upload will

drain the cell quickly and make the collar useless, this option is not viable. Another

alternative is to build a network infrastructure, but building a network infrastructure in the

wilderness would be an expensive alternative.

The cost effective way of doing this is to use Delay Tolerant Networks. Inthis case,

the zebras will be fitted with collars that have a GPS, a short range radio and solar cells to

power the collar. The GPS wakes up periodically to record theposition of the zebra.

When a collar is in close range with another collar, the radiowill transmit all the data of

one zebra to the other zebra and vice versa. When a node comes within close range of

multiple nodes, it will aggregate data from multiple zebras. Finally, when a researcher

drives by and comes within range of any of these collars, it will transfer the aggregated

data to its final destination, which is the base station. The collars will delete the

accumulated data when it has transferred the data to the basestation [Juang et al. 2002].

DTN Architecture (explained in Chapter 2) is an overlay architecture which can

function on top of multiple transport layers; thus making DTN deployable in many

different environments that support various transport layer protocols. DTN uses a standard

message format called Bundle Protocol (BP)[Scott and Burleigh 2007] for transferring

data. An extra layer of reliability can be added to BP by usingcustody transfer, which is

explained in Chapter 2. Multiple transport layers can be used under BP by using the

corresponding convergence layers above the transport layer. These convergence layers

12

enhance the underlying transport layer protocol and use thetransport protocol to send and

receive bundles. Some of the Convergence layer protocols that DTN supports are TCP

[Demmer and Ott 2008], UDP [Kruse and Ostermann 2008], LTP [Burleigh et al. 2008]

and Saratoga [Wood et al. 2008]. The TCP convergence layer (TCPCL) uses TCP, but the

other three convergence layers use UDP to send and receive bundles. LTP is also designed

to use Consultive Committee for Space Data Systems (CCSDS) link layer

protocol[CCSDS 2006] and is currently being developed to use DCCP [Kohler et al.

2006].

Interplanetary Overlay Network (ION) implements DTN architecture and it was

developed by the NASA Jet Propulsion Lab. ION was intended tobe used on embedded

system such as space flight mission systems but it can also be used to develop DTN

protocols and test the performance of DTN protocols. DTN2[DTNRG 2010], DTN1

[DTNRG 2010], IBR-DTN [Doering et al. 2008] are some of the other implementations of

DTN architecture.

There has been earlier work that evaluates the performance of TCPCL implemented

by DTN2, LTP implemented by ION, and TCPCL-LTPCL hybrid for long-delay cislunar

communication inWang et al. [2010]. This thesis, unlike previous works, evaluates the

maximum performance that can be expected from these convergence layers under various

conditions of delay and losses. The tests are conducted witha 4 node setup, with the

capability of modifying the link characteristics of the middle link. The throughput of

TCPCL and LTPCL is compared for different delays, losses, combination of delays and

losses, and bundle sizes. Performance is evaluated for BP with custody transfer and

without custody transfer.

Some of the main contributions of this thesis are LTP dissector for wireshark and

SBP Iperf. The LTP dissector decodes LTP data in wireshark. Thishas proved to be very

useful tool to debug and analyze LTP behaviour. SBPIperf is a network performance tool

13

that is used with DTN protocols. All the performance tests inthis thesis were performed

using SBPIperf.

Chapter 2 gives an insight into DTN architecture, BP and the convergence layers that

were evaluated. Chapter 3 discusses the experimental setup, test parameters, and testing

tools. Chapter 4 presents the results of the performance tests and analyzes the result of the

tests. Chapter 5 concludes with some recommendations for future work.

14

2 DTN Architecture and Convergence Layers

2.1 DTN Architecture

Fall [2003] proposes an overlay architecture that will function abovetransport layer

protocols like TCP, UDP, and sensor network transport protocols. The architecture

suggests that networks be divided into regions, which can beconnected by DTN gateways.

The nodes within a region can send and receive data without using the DTN gateway. The

DTN gateway will act as a bridge between the 2 regions; therefore, the DTN gateway

should understand the transport protocols of both regions.The naming convention

consists of 2 parts: region name and entity name. The region name should be unique, but

the entity name needs to be unique only within the region. Hence, while routing packets in

delay tolerant networks, the DTN gateways use the region names for routing. Only the

DTN gateway at the end of the destination region needs to resolve the entity name.

Selecting a path and scheduling a packet transfer in DTN is performed by the use of a

list of contact information of the nodes. The contact information consists of the relative or

absolute time when contact can be made with other nodes, the delay between nodes, and

the data rate that the link will support. This method of routing is contact graph routing.

Other than contact graph routing, there are other routing techniques that can be used

depending on the environment. The zebranet explained in theprevious chapter uses flood

routing, where each node transfers all data to every node it comes in contact with.

The routing nodes can either have persistent or non-persistent storage. The persistent

routing nodes will support custody transfer, which impliesthat the routing node will store

a packet/bundle that it receives in persistent storage and send an acknowledgement to the

sending node. The routing node will store the bundle until ithas successfully transferred it

to the next node that accepts custody. Custody transfer of the overlay architecture is an

important concept of DTN because it gives reliability to thearchitecture amidst losses and

15

Figure 2.1: Bundle Protocol in Protocol Stack

delays. Some of the transport layer protocols like TCP, LTP,etc. also provide reliability.

Therefore, custody transfer will add a second layer of reliability. Since this overlay

architecture functions over transport layer protocols, this architecture would need to

enhance the underlying transport protocols that will be used by using a convergence layer

above the transport protocol. If the transport layer protocol is a connection oriented

protocol, then it is the responsibility of the convergence layer to restart connectivity when

the connection is lost. Convergence layers are explained more in depth later in this chapter

[Fall 2003, p.30 - 33].

16

2.2 Bundle Protocol

The BP is the overlay protocol that is used in the DTN architecture. This protocol

gives the DTN architecture the capability to store a bundle and then forward it to the next

node. A Bundle is a collection of data blocks that is precededby a common header. As

shown in Figure2.1, BP operates on top of the transport layer protocol, and it isdesigned

to function over a wide range of networks including deep space networks, sensor

networks, and ad-hoc networks.Wood et al. [2009] closely studies the BP and examines

some of the BPs design decisions. Every delay tolerant network differs vastly in

characteristics, and it is difficult to find a general networking solution to the problem.

Therefore, DTN architecture decided to implement a generalized message format known

as BP, which could be used over a variety of networks. Hence the protocol by itself is not

the solution to delay tolerant/disrupted networks; however, it needs to work in cooperation

with convergence transport layer protocols or transport layer protocols designed specially

for delay tolerant networks. This protocol only provides a standard format for data but

most of the support for a disrupted network is provided by theconvergence layers

implemented in the architecture [Wood et al. 2009].

Figure2.2shows the bundle structure as specified inScott and Burleigh [2007]. A

bundle consists of a primary block followed by one or more payload blocks. Most of the

fields in the BP structure are Self-Delimiting Numeric Values (SDNVs) [Eddy and Davies

2011], which helps to reduce size of the bundle. Using SDNV also helps to make the

design scalable for the various underlying network layers that the bundle protocol might

use.

The following are the fields in the bundle structure :

• The bundle processing flags are sub-divided into general flags, class of service and

status report.

17

Figure 2.2: Bundle Protocol Structure

18

• The block length is an SDNV of the total remaining length of the block.

• The dictionary offsets represents the offset of specified fields in the dictionary. It

contains the scheme offset and scheme specific part (SSP) offset of the destination,

source, report-to, and custodian. Hence, the destination scheme offset is the position

within the dictionary which contains the scheme name of the endpoint. Similarly,

the destination SSP offset holds the offset within the dictionary which contains

scheme-specific part of the destination endpoint.

• The next field in the primary bundle block is creation timestamp and creation

timestamp sequence number both of which are SDNVs. Creationtimestamp is the

UTC when the bundle was created. If more than one bundle is created in the same

second, then the bundles are differentiated by the incrementing bundle timestamp

sequence number.

• Dictionary information contains the dictionary length which is the SDNV of the size

of the dictionary and the dictionary itself. The dictionaryaccommodates the scheme

and the scheme specific parts which are referenced by the offsets earlier in the block.

• The last two fields in the primary bundle block (fragment offset, payload size) are

included in the block only if the bundle processing flags signal that the bundle is a

fragment. The fragment offset is the offset from the beginning of the actual

application data, and payload size is an SDNV of the total application data of the

bundle.

The primary bundle block is usually followed by one or more canonical bundle

blocks. The general format of the bundle block is also represented in Figure2.2. The

bundle block has a 1 byte type field, and if the bundle block is apayload block then type

field will be 1. The type field is followed by bundle processingflags which can set certain

19

special bundle features, like custody transfer, etc. The length field which is an SDNV,

represents the size of payload field, and the payload field contains the application data

[Scott and Burleigh 2007].

In the DTN Architecture, the application hands over data to alocal bundle agent, and

the bundle agent is responsible to get the data over to the bundle agent at the destination.

Bundling of the data is done by these bundle agents. The bundle agent applies routing

algorithms to calculate the next hop to forward the bundle. The bundle agent then

advances the bundle to the corresponding convergence layeradapter to forward the bundle

to its next hop. In environments where DTNs are applied, there is a high probability that

the forwarding will fail because of loss or lack of connectivity. If the convergence layer

adapters return with failure due to lack of connectivity, the bundle agent will wait until

connection is restored and then reforward the bundle. However, if a bundle is lost due to

network losses, then it will depend on the reliability of thetransport layer protocol and/or

the reliability of bundle protocol, which ensure the bundlegets from one hop to the next.

On reception of a bundle, the bundle agent needs to compute the next hop and continue

the same steps mentioned above. However, if a bundle is destined for a local endpoint, the

application data will be sent to the application after reassembly.

BP can add a layer of reliability by using an optional servicecalled custody transfer.

Custody transfer requests that a node with persistent storage store the bundle if it has

storage capacity. The node that accepts custody of a bundle will transmit a custody signal

to the node that previously had custody of the bundle. On reception of the custody signal,

the previous custodian will delete the bundle from the custodian’s storage and/or

retransmission buffer. The previous custodian is not required to wait until the bundle

reaches the destination to delete the bundle from its storage. The responsibility of reliably

transferring the bundle to the destination lies with the node that accepted custody of the

bundle. Therefore, if the bundle is lost or corrupted at somepoint of time, it is not the

20

responsibility of the sender to resend the bundle, but the responsibility of the node that

accepted custody or the custodian of that bundle to resend the bundle.

Custody transfer not only ensures that the bundle reaches the destination from the

source, but it also shifts the accountability of the bundle in the direction of the destination.

Every time a node accepts custody of a bundle, it will free theprevious custodian’s

resources. On the other hand, bundle transmission without custody transfer relies on the

reliability of the transport protocol.

BP is a relatively newer protocol, so there are still some undecided issues.Wood

et al. [2008] andWood et al. [2009] list the problems with the current design of the BP.

The following are some of the significant concerns :

• Synchronizing time on all the nodes is important for BP because a bundle can be

rejected if the time stamp check claims that the bundle has expired due to

unsynchronized clocks on the sender and receiver.

• More than one naming scheme is currently used in different BP implementations

and each of the naming schemes have separate rules for creating end point identifier

(EID).

• There is no accepted routing protocol, essentially becausethere is more than one

naming scheme. Dynamic routing protocols will help to improve the scalability of

the network.

2.3 Convergence Layers

The convergence layer acts as an interface between the BP andthe transport layer

protocol. Figure2.3shows the position of the convergence layer protocol in the protocol

stack. Since BP is an overlay protocol that can be used above different transport layer

protocols, the corresponding convergence layer protocol needs to be used to allow data

21

flow from BP to the transport protocol and vice versa. The mainfunction of this layer is to

aid the underlying transport layer. As mentioned before, TCPCL, UDPCL, LTP, and

Saratoga are some of the DTN convergence layer protocols.

Figure 2.3: Convergence Layer Protocol in Protocol Stack

A DTN that uses TCP for communication will require a TCP Convergence layer

between the BP and TCP. The convergence layer enhances the underlying transport

protocol by adding some additional functionality to the transport layer protocol which

might make it suitable in extreme environments. For example, the convergence layer for a

connection oriented protocol will have to maintain the connection state. Thus, if

connection is lost for any reason, the convergence layer will try to re-establish connection.

22

Additionally, if there are transport protocols that do not provide congestion control, then

the convergence layer adds this functionality to the stack.

2.3.1 TCP Convergence Layer

DTN uses TCPCL when the transport layer protocol being used is TCP. TCP is a

reliable protocol, and it ensures that a packet reaches fromone node to another. Every

packet that a node sends has to be acknowledged by the sender.If a packet is lost in

transmission and the receiver does not receive any acknowledgement for a packet, then it

will retransmit the packet. Timers on the sender side will keep track of the time when the

acknowledgment is expected. When the timer runs out, the packet is retransmitted.

Similarly when an acknowledgment is lost, the sender will have to retransmit the packet.

TCP implements flow control using congestion window. The congestion window size

dictates the amount of data that is in the network. TCP also uses certain congestion

control algorithms, when there is loss of packets. TCP is designed to conclude that packet

losses are because of congestion. Slow start, congestion avoidance, fast retransmit, and

fast recovery [Allman et al. 2009] are used for congestion control. A TCP connection

starts with slow start and it is again applied when there is a retransmission timeout. The

congestion window size is 1 segment size and during slow start this congestion window

size is increased by 1 segment size for every acknowledgmentreceived; thus, increasing

the congestion window size exponentially. Furthermore, TCP performs congestion

avoidance when the congestion window size either reaches the slow start threshold or

when there is data loss. The initial size of the slow start threshold is fixed high, which is

usually the receiver window size.

In the congestion avoidance phase, every round trip time, the congestion window size

increases by 1 segment size. However, the congestion windowcannot exceed the receiver

window size. When a packet is lost, the receiver will send duplicate ACKs for the last

23

packet in sequence that it received. On receiving 3 duplicate ACKs, the sender will

retransmit the missing segments. This algorithm is called fast retransmit because the

sender does not wait for the retransmission timer. During fast retransmit, the congestion

window size is reduced by half its original value and so is theslow start threshold . Fast

recovery phase follows fast retransmit. During fast recovery the lost packet is

retransmitted and the congestion window size is advanced by1 segment size each time a

duplicate acknowledgement is received. When the receiver acknowledges all the missing

data, the congestion window size drops to the value of the slow start threshold and returns

to the congestion avoidance phase. On the other hand, if it does not receive an

acknowledgment there will be retransmission timeout. Thistimeout causes the congestion

window to drop to 1 segment size and the connection returns toslow start.

The transport layer implements both congestion control andflow control. Hence

TCPCL does not have to perform any congestion or flow control.TCP also ensures

reliable data delivery; therefore, TCPCL can assign the responsibility of reliable data

transfer to TCP. On the other hand, the convergence layer’s responsibilities include

re-initiating connection and managing message boundariesfor the TCP stream.

Demmer and Ott [2008] proposes a TCP-based convergence layer protocol. Similar

to TCP’s three-way handshake, TCPCL establishes connection by exchanging contact

headers. This exchange sets up connection parameters, likekeep alive period,

acknowledgements, etc. Figure2.4shows an example TCPCL message exchange. When

NodeA needs to send a bundle to NodeB using TCPCL, NodeA first needs to establish a

TCPCL connection with NodeB. To establish a TCPCL connection, NodeA sends a

contact header to NodeB. NodeB responds with a contact header and the connection

parameters are negotiated. Following connection establishment, both the nodes can

exchange data. TCPCL has an option to add an extra layer of reliability by using bundle

acknowledgements. If the 2 communicating nodes decide during the contact header

24

exchange that they will support acknowledgements, then each bundle that is transmitted

has to be acknowledged by the receiver. The nodes also send keep-alives back and forth,

the keep-alives is a method to ensure that the nodes are stillconnected.

Figure 2.4: TCPCL Connection Lifecycle

TCP/IP is already considered to be a chatty protocol due to the three-way handshake

and acknowledgements. Exchange of contact headers and TCPCL acknowledgements

adds an extra layer of chattiness. In environments with limited connectivity and

bandwidth the additional packets and bundles that are exchanged for connection setup can

affect the performance of the convergence layer negatively. Intheory, there is a possibility

25

of 3 layers of reliability using TCPCL over TCP - TCP reliability, TCPCL reliability using

TCPCL acknowledgements, which is optional, and BP reliability using custody transfer.

Akan et al. [2002] presents the performance of different versions of TCP on deep

space links. The performance of TCPCL can be expected to be very similar to TCP.

According to the test results inAkan et al. [2002], TCP recorded a throughput high of

120KB/sec on a 1 Mbps link with no delay, and the throughput drops down to 10 bytes/sec

when the round trip time (RTT) is 40 minutes.

2.3.2 LTP Convergence Layer

LTP was specially designed for links with high delays and intermittent connectivity.

LTP is used over space links or links with long round-trip time as a convergence layer.

Similar to TCP, LTP acknowledges bundles that it receives and retransmits any lost

bundles. However, unlike TCP, LTP does not perform a 3-way handshake for connection

setup. LTP is generally used above the data link layer on space links, but for testing

purposes on terrestrial networks it is also used over UDP.

Burleigh et al. [2008] explains the objective behind the design of this protocol and

the reasoning behind the design decisions. Some of the important design decisions are

listed below:

• To utilize the intermittent link’s bandwidth efficiently, LTP uses the concept of

sessions. Each LTP data block is sent using a session and LTP will open as many

sessions as the link permits. This will allow multiple data blocks to be sent over the

link in parallel.

• LTP retransmits unacknowledged data, which is what makes LTP a reliable

protocol. LTP receiver uses report segments to let the sender know all the data

segments the receiver received successfully.

26

• Connection setup has been omitted in LTP because in an environment with long

delay and limited communication opportunities, a protocolthat needs connection

setup will under utilize the bandwidth.

• Unlike TCP, LTP connections are unidirectional; therefore, communication between

nodes talking to each other on LTP will have two unidirectional connections open.

Ramadas [2007] andBurleigh et al. [2008] describe the operation of the LTP

protocol. A basic LTP operation without link errors or losses is depicted in Figure2.5.

The sender opens up a session to send a LTP data block. As mentioned earlier, a node can

open as many sessions as the link will permit. This is also a flow control mechanism

because bandwidth is limited by the number of sessions that can be opened

simultaneously. Each time a node sends a LTP data block, it will use a different session

number. The last data segment of a data block is a checkpoint/End of Block (EOB)

segment. When the receiver receives a checkpoint, it will respond with a report segment.

The report segment is an acknowledgement of all the data segments of the data block that

it received. On receiving the report acknowledging all the segments of the bundle block,

the sender closes the session and sends a report acknowledgement. The receiver will close

the import session on reception of the report acknowledgement.

The following figure, Figure2.6, shows an LTP session on an error prone network.

The sender will send all the data segments of the block. If some of the data segments are

lost during transmission, the receiver will report all the segments it received in its report

segment. This will let the sender know which data segments itneeds to retransmit. The

sender responds with a report acknowledgement, and then it retransmits the lost segments.

After the sender retransmits all the lost segments, it will send a checkpoint segment to the

receiver. On reception of the checkpoint, the receiver willrespond with another report

segment with a report of all the segments it received. If the report claims to have received

27

Figure 2.5: LTPCL Connection Lifecycle

all the segments of the block, the sender will close the session and send a report

acknowledgement. When the receiver receives the report acknowledgement, it will close

the LTP session [Ramadas 2007; Burleigh et al. 2008].

Sessions is the flow control mechanism in LTP. LTP can be configured to open a

certain number of sessions, so when all the sessions are being used, data blocks that need

to be sent have to wait until a previous session closes. LTP can technically transmit at

link’s bandwidth only limited by the number of sessions thatLTP can open

simultaneously.

28

Figure 2.6: LTPCL Error Connection Lifecycle

29

3 Experiment Setup

This chapter explains the hardware and software configuration of the test setup.

3.1 Hardware Configuration

Figure 3.1: Physical Testbed Setup

Figure3.1shows the physical setup of the testbeds. The 4 testbeds are connected to

each other using a 100Mbps Fast Ethernet Switch. The logicaltestbed setup for the tests is

shown in Figure3.2. Node1 is assigned as the source node and Node4 is assigned asthe

destination node. All the nodes have host-specific routes. Node1 has to make 2 hops to get

to Node4 through Node2 and Node3. Similarly Node4 also needsto make 2 hops to reach

Node1 through Node3 and Node2. The routing information is present in the ION

configuration files presented inA.

30

Figure 3.2: Logical Testbed Setup

Table3.1provides details about the individual testbeds. The tests are done on Linux

machines, running Linux kernel version 2.6.24. All the testbeds are 32 bit machines

except for Node2. Node2 is a 64-bit machine, so that ION will allocate more small pool

memory. This will ensure that it doesn’t run out of small poolmemory during the tests.1

Table 3.1: Testbed Configuration

Configuration Node1 Node2 Node3 Node4

Operating System Ubuntu

8.04

Ubuntu

8.04

Ubuntu

8.04

Ubuntu

8.04

Linux Kernel Version 2.6.24 2.6.24 2.6.24 2.6.24

CPU Intel Pen-

tium 4 CPU

2.80GHz

AMD Tu-

rion 64

X2

Intel Pen-

tium 4 CPU

3.00GHz

Intel Pen-

tium 4 CPU

2.80GHz

Memory 1GB 2GB 1GB 1GB

Architecture 32-bit 64-bit 32-bit 32-bit

1 ION can allocate a maximum of 16MB small pool memory on a 32-bit machine and a maximum of 256

bytes. The small pool memory is used to store bundle related information within ION. Since Node2 mighthave to retain bundles in the system when there are losses anddelays, it is important for Node2 to have moresmall pool memory. Otherwise Node2 will run out of small poolmemory and crash ION.

31

3.2 Software Configuration

Table3.2lists some of the important experimental factors used in theperformance

tests. All the testbeds run an opensource version of Interplanetary Overlay Network which

is an implementation of DTN architecture from NASA’s Jet Propulsion Lab. The number

of syn/synacks that TCP will send is increased from the default value 5 to 255 by setting

the kernel parameternet.ipv4.tcp syn retries = 255 and emphnet.ipv4.tcpsynackretries=

255. This ensures that TCP does not give up connection establishment for high delay

links. The bundle lifetime within ION is set to 10 hours. Thishelps to ensure that the

bundles do not expire during the duration of the test. The tests are performed over 2 sets

of DTN protocol stacks - BP over TCPCL over TCP/IP and BP over LTP over UDP.

Table 3.2: Experimental Settings and Parameters

Parameters Values

DTN implementation ION Opensource

DTN Convergence LayersLTP(ION v2.4.0), TCP(ION v2.3.0)

Bundle Size(bytes) 200 - 40000

Channel Bandwidth 100 Mbits/sec

RTT(sec) 0 - 20

Loss(%) 0 - 10

The throughput of the 2 protocols (TCP, LTP) is measured for bundle sizes varying

from 200 bytes to 40000 bytes. The behavior of these protocols in extreme environments

is modeled by measuring the throughput for various values ofround-trip time and packet

loss. We also measure the throughput in environments with some delay and loss. The

32

delay or loss on this multi hop link between Node1 and Node4 issimulated on Link2

between Node2 and Node3. Netem [Hemminger 2005] is used on Node2 and Node3 to

emulate delay and loss. The netem rules have been set up such that the delay or loss are

applied to incoming traffic on Node2 and Node3. The network emulation rules are applied

to incoming traffic rather than outgoing traffic because if loss is applied to outgoing traffic,

the loss is reported to the higher layers, and protocols likeTCP will retransmit the lost

packets, which is not the desired effect.

All the tests use a tool called SBPIperf, which is a variation of the commonly used

Iperf. SBPIperf is similar to Iperf except that it runs over DTN protocols. The SBPIperf

client runs on Node1, and the SBPIperf server runs on Node4. The SBPIperf client

transmits data and SBPIperf server waits until it receives all the bundles and responds

back to the client with a server report. The SBPIperf client and server code is presented

in B.3.

The client will use an incrementing sequence number in each of the data segment it

sends. The last data segment will have a negative sequence number, which helps the

server to determine that the client has stopped sending data. The SBPIperf server will

wait for a few seconds before it sends the server report. Thiswill ensure that none of the

out of order bundles are reported as lost bundles. This wait period can be adjusted by

changing the timeout parameter on the server side. The server report contains the

following information : throughput, loss, jitter, bundlestransmitted and transmission time.

33

4 Experiments, Result and Analysis

4.1 TCP Convergence Layer Tests

This section discusses the results of the performance testswith the TCPCL.

4.1.1 TCPCL Without Custody Transfer

The first set of tests uses Bundle Protocol over TCP Convergence Layer over TCP/IP.

We measure the throughput for varying bundle sizes and round-trip times. In this test the

SBP Iperf client transmits data for 30 seconds, and when the server receives all the data

the server sends a report back to the client. As explained in the previous chapter, the

round-trip time between Node2 and Node3 is varied using netem. This test does not use

the custody transfer option in BP and the loss between the links is set to 0. Figure4.1

shows the TCPCL throughput in Kbits/sec for various round-trip times(seconds).

101

102

103

104

105

0 2 4 6 8 10 12 14 16 18 20

Thro

ughp

ut (K

bits

/sec

)

RTT (sec)

200 bytes500 bytes

1000 bytes

1500 bytes2000 bytes5000 bytes

7500 bytes10000 bytes15000 bytes

Figure 4.1: TCPCL Throughput vs RTT

34

As we can see in Figure4.1, throughput decreases with increase in round-trip time.

This is because the TCP congestion window size will increasemore gradually when the

RTT is high, during slow start. The congestion window is doubled every round trip during

slow start. However, this increase will be more gradual whenRTT is higher.

Figure 4.2: Outstanding Window Graph for TCP Reno with 2 sec RTT

The tests where RTT is greater than 0.4 seconds and less than 10 seconds use TCP

Cubic instead of TCP Reno which is used for the other tests. TCP Cubic yields better

throughput for these range of RTT values since TCP Cubic is able to increase the

35

Figure 4.3: Outstanding Window Graph for TCP Cubic with 2 secRTT

congestion window higher than TCP Reno. Figure4.2and Figure4.3shows the difference

in outstanding window when using TCP Reno and TCP Cubic on a link with a 2 second

RTT. The outstanding window in the figures is depicted by the red line.

Additionally, when the RTT is greater than 9 seconds (when the SYN is retransmitted

for the third time), the sender will assume that the SYN packet is lost since it has not

received the SYN/ACK from the receiver within 9 seconds. This behavior will trigger a

retransmission timeout. Under normal circumstances, the slow start threshold (ssthresh) is

36

set to the size of the advertised window. Contrary to this, when TCP detects loss due to

congestion, TCP calculates the ssthresh as [Allman et al. 2009]

ssthresh=max(Flight Size/ 2, 2 * Maximum Segment Size)

Since, data has not been sent as a part of this connection, theflight size will be zero.

Thus setting ssthresh to twice the maximum segment size(MSS) . The ssthresh and the

size of the congestion window (cwnd) determines whether a TCP connection is in slow

start or in congestion avoidance. If ssthresh is greater than cwnd, TCP will be in slow start

phase and in slow start the congestion window will double every RTT. The TCP

connection will go into congestion avoidance when the congestion window increases to a

value greater than the ssthresh. The congestion window willincrease by 1 segment size

every RTT during congestion avoidance phase as mentioned earlier in chapter 2.Allman

et al. [2009] also specifies that the initial window size is set to 1 maximum segment size

when either a SYN or SYN/ACK is lost.

Hence in the above mentioned case where the RTT is greater than 9 seconds, the

initial window size is set to 1 MSS and the ssthresh will be setto twice the MSS. This low

value of ssthresh will cause TCP to go into congestion avoidance after one roundtrip time.

This behavior of TCP is shown in the TCP time sequence graph inFigure4.4. Therefore, a

connection with high RTT will take longer to reach its maximum congestion window size.

This drop in the ssthresh value due to retransmission timeout event during the TCP

handshake period can be prevented if we can configure the initial timeout depending on

the RTT of the link. But the TCP version in Linux 2.6.24 does not allow us to set this

value without recompiling the kernel hence causing any connection with an RTT greater

than 3 seconds to start with the congestion avoidance phase rather than slow start.

37

Figure 4.4: Time Sequence Graph of TCP Link with 20sec RTT

We also notice that that the throughput varies with bundle size. When bundle size is

200 bytes, the throughput is approximately 2500Kbits/sec, and the throughput is 40 times

higher when bundle size is 15000 bytes. This is so because IONmakes approximately 64

semop2 calls for every bundle that is sent. We use a simple program that performs a

million semop operations and calculate the time it takes to run the program. By running

this program on one of the testbeds, we calculated that the time taken to perform one

semop function call is 1.07 * 10-6 seconds(sec).

2 semop is a system call that performs semaphore operations. Semaphores are used for inter-processcommunication to regulate access to a common resource by multiple processes.

38

1 Bundle=> 64 semop

1 Semop=> 1.07∗ 10−6sec

Semop overhead/ bundle=> 64∗ 1.07∗ 10−6sec

Semop overhead/ bundle=> 68.87∗ 10−6sec

Hence there is a semop overhead per bundle that is sent. Otherthan the semop

overhead, there is also a semaphore contention overhead within ION. Semaphores are used

within ION for inter-process communication. The ION application will add the bundle to

the forward queue and depending on the addressing scheme, itwill release the scheme

specific semaphore. The scheme specific forwarder will wait on the scheme specific

semaphore. The forwarder, on grabbing this semaphore, willenqueue the bundle into the

outduct queue and release the outduct semaphore. The outduct will wait on the outduct

semaphore, and on receiving the semaphore, the outduct process will dequeue the bundle

and send it out. Additionally, ION also uses semaphores whenit makes changes to shared

memory. The combination of all these semaphores will cause acontention overhead.

We use a program called SystemTap, which can be used to get information about the

linux system. We tap ipclock() to record the amount of time ION waits on this functionto

get an approximate contention overhead time. Refer toB.2 for the SystemTap script.

Semaphore contention overhead/ bundle=> 149.38∗ 10−6sec

Adding the semop overhead and the contention overhead for each bundle we get

Total overhead/ bundle=> 218.25∗ 10−6sec

No of bundles transmitted per second=> 1/218.25∗ 10−6sec

39

No of bundles transmitted per second=> 4582 bundles

Hence for a bundle of size 200 bytes, the expected maximum throughput is

Throughput of 200 byte bundle=> 4582∗ 200 bytes/sec

Throughput of 200 byte bundle=> 6.99 Mbps

The client reports the transmission rate at which the data left the client. The server,

on receiving all the data sends a server report back to the client reporting the rate at which

it received the data. The client node reports a transmissionrate of approximately 6.3Mbps

for 200 byte bundles, which reduces to 3Mbps on the server report. Some factor of semop

and contention overhead needs to be factored into throughput over 3 hops from client to

server. That explains the difference in throughput between the client report and the server

report. Since the number of bundles that can be transmitted is limited by semop and

contention overhead, throughput will increase with increase in size of bundles.

The next set of tests measures TCPCL throughput for different percentages of packet

loss without custody transfer. In the tests X% packet loss isdefined as, X packets out of

100 packets are dropped from Node2 to Node3, and X packets outof 100 packets are

dropped from Node3 to Node2. Like the previous set of tests, the SBPIperf client in this

case also sends data for 30 seconds and waits for the SBPIperf server to respond back

with the report. Figure4.5shows the throughput in Kbits/sec of TCPCL for different

losses and the maximum theoretical throughput. We use the Mathis equation [Mathis et al.

1997] to model the the maximum theoretical throughput for a TCP connection with packet

loss. The Mathis equation is given by

40

102

103

104

105

0 1 2 3 4 5 6 7 8 9 10

Thro

ughput (K

bits

/sec)

Loss (%)

Max200 bytes500 bytes

1000 bytes1500 bytes2000 bytes

5000 bytes7500 bytes

10000 bytes

15000 bytes

Figure 4.5: TCPCL Throughput vs Loss

Bandwidth = (Maximum Segment Size× C)/(RTT ×√

p) (4.1)

Where, C is Mathis constant and

p is loss probability

As you can see in Figure4.5, the theoretical throughput and the actual throughput are

almost the same for 2% loss and the lines start to diverge fromeach other after 3% loss.

As the loss increases, chances of retransmission timeouts also increase. Unfortunately, the

41

100

101

102

103

104

105

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Thro

ughp

ut (K

bits

/sec

)

RTT (sec)

0% Losstheoretical max(.05 % loss)

.05% Losstheoretical max(2 % loss)

2% Losstheoretical max(5 % loss)

5% Loss

Figure 4.6: TCPCL(w/o custody) Throughput vs RTT & Loss for 200 Byte Bundles

Mathis equation does not consider retransmission timeoutsin the model. That is why the

maximum theoretical throughput diverges from the actual throughput.

TCP will go into congestion avoidance phase when there is packet loss. When the

TCP sender gets 3 duplicate acknowledgements from the receiver, the TCP sender reduces

the congestion window to half the previous size and retransmit the next unacknowledged

segment. However, in cases when either one of the 3 duplicateacknowledgements is lost,

or the retransmission gets lost, a timeout event occurs. When a timeout event occurs, TCP

reduces the congestion window to 1 maximum segment size and goes back into slow start.

For smaller loss percentage, losing one of the triple duplicate acknowledgements or losing

a retransmission is rare. But the occasional packet loss will halve the congestion window

42

100

101

102

103

104

105

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Thro

ughp

ut (K

bits

/sec

)

RTT (sec)

0% Losstheoretical max (.05% loss)

.05% Losstheoretical max (2% loss)

2% Losstheoretical max (5% loss)

5% Loss

Figure 4.7: TCPCL(w/o custody) Throughtput vs RTT & Loss for 5000 Byte Bundles

from time to time, and since the chances of timeout events areless, the congestion

window size will gradually increase with time. On the other hand, higher loss percentage

increases the chance of timeout events due to loss of one of the triple duplicate

acknowledgements or due to loss of the retransmitted packet. Thus causing the connection

to reduce its congestion window to 1 MSS from time to time. This explains the drop in

throughput after 3% packet loss.

The last set of tests for TCPCL without custody transfer measures the throughput for

combinations of loss and delay. The throughput is measured for 0, .05, 2 and 5% loss for

delay ranging from 0 to 5 seconds. Unlike the previous 2 sets of tests, in this set we only

transfer 25MB of data from the SBPIperf client to the SBPIperf server. This helps to

43

100

101

102

103

104

105

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Thro

ughp

ut (K

bits

/sec

)

RTT (sec)

0% Losstheoretical max(.05% loss)

.05% Losstheoretical max(2% loss)

2% Losstheoretical max(5% loss)

5% Loss

Figure 4.8: TCPCL(w/o custody) Throughput vs RTT & Loss for 15000 Byte Bundles

reduce the run time of some of the tests. Since the throughputof some of the tests are

really low, the run time of some of the extreme test cases can be very long.

Figure4.6, Figure4.7and Figure4.8presents the throughput of TCPCL without

custody transfer for a combination of RTTs and losses for 200byte, 5000 byte and 15000

byte bundles respectively. The combination of loss and delay has a very significant impact

on throughput when compared to the impact on throughput solely due to delay or loss.

The above figures also depict the maximum theoretical throughput which is obtained

using Equation4.1. The observed throughput values are very close to the maximum

theoretical throughput according to the Mathis equation [Mathis et al. 1997].

44

4.1.2 TCPCL With Custody Transfer

This sections presents the results from using TCP Convergence Layer with custody

transfer enabled in the BP layer. The same set of tests performed in Section4.1.1are

repeated with custody transfer. We also compare the throughput of TCPCL with and

without custody transfer. In all the TCPCL with custody transfer test cases, SBPIperf

client sends data for 10 seconds. The reason behind reducingthe amount of time the

SBP Iperf client spends sending data when custody transfer is enabled is that the ION

nodes will have to store the bundles for a longer time since the nodes have to wait for a

custody signal from its neighboring node before destroyingthe bundle. Storing huge

amounts of bundles can cause the ION node to run out of memory;hence, to ensure the

tests do not fail we have to reduce the amount of data the client sends. In this test scenario,

SBP Iperf client sends approximately 52000 bundles in 10 seconds when the bundle size

is 200 bytes.

The behavior of ION when custody transfer is enabled is explained here. When a

bundle requesting custody transfer is received by a node, a custody signal will be sent to

the previous custodian of the bundle. If the receiving node decides to accept custody for

the bundle, the custodian is notified by sending a custody accepted signal. The custodian

will now delete the bundle from its persistent storage. But if the receiver discards the

bundle for reasons like redundant bundle, no memory, error block, etc., the receiver will

send a custody error signal with a reason code to the custodian. The custodian will take

appropriate action depending on the error code. If the receiver discards the bundle because

it does not have memory, the custodian will reforward the bundle to another node which

can transmit the bundle to the desired destination.

We explore the behavior of ION when custody signal is lost, since these nodes operate on

links with losses. Custody signals will not be lost when using TCP as the transport layer,

but there is a possiblity of losing the custody signal when running LTP over UDP.

45

• Custody acceptance signal - When the custody acceptance signal is lost, the receiver

(custodian2) will accept custody for the bundle, but the previous custodian

(custodian1) will not release custody for the bundle. Technically, there will be 2

custodians for the bundle in the network. Custodian2 will continue to forward the

bundle to its destination. Custodian1 will keep the bundle in persistent storage until

its TTL expires. On expiration of TTL, custodian1 will delete the bundle and send a

status report to the source.

• Custody error signal - If the custody error signal from the receiver is lost, the

custodian retains the bundle in persistent storage until the TTL expires. When the

TTL expires, the custodian will delete the bundle and send a status report to the

source.

The TCPCL performance for different RTT is shown in Figure4.9. Figure4.10

compares the performance of TCPCL with and without custody transfer for varying RTT.

The throughput of TCPCL with custody is lower than the throughput of TCPCL without

custody. When custody transfer is enabled, the bundle is notdestroyed until a node

receives custody signal from a node that has accepted custody for the bundle. On receiving

a custody signal, the node will find the bundle correspondingto the custody signal and

delete it from ION memory. This additional processing time reduces the number of

bundles ION can transmit, hence affecting the performance of TCPCL with custody.

The difference in throughput is more profound for smaller bundles because more

bundles have to be transmitted when the bundle size is smaller. As mentioned earlier when

bundle size is 200 bytes, SBPIperf client will send approximately 52000 bundles in 10

seconds. On the other hand, SBPIperf client will send approximately 8000 bundles in 10

seconds for a bundle of size 15000 bytes. Hence the number of custody signals will be

lesser when bundle size is 15000 bytes as compared to bundle size of 200 bytes. This

46

101

102

103

104

105

0 5 10 15 20

Thro

ughput (K

bits

/sec)

RTT (sec)

200 bytes 5000 bytes 15000 bytes 40000 bytes

Figure 4.9: TCPCL(w/ custody) Throughtput vs RTT

would imply that ION will spend less time processing custodysignals when bundle size is

15000 bytes.

The performance of TCPCL with custody for different loss rates is presented in

Figure4.11. The maximum theoretical throughput using Mathis equation[Mathis et al.

1997] is included in this figure. The performance comparison between TCPCL with

custody and without custody for different loss rates is illustrated in Figure4.12.

47

101

102

103

104

105

0 5 10 15 20

Thro

ughput (K

bits

/sec)

RTT (sec)

200 bytes without custody200 bytes with custody

15000 bytes without custody15000 bytes with custody

Figure 4.10: Comparing TCPCL(w/ custody) and TCPCL(w/o custody) Throughtput vs

RTT

The performance of TCPCL with custody for higher loss rates is similar to the

performance of TCPCL without custody for higher loss rates.A combination of low

congestion window size and retransmission time outs throttle the number of TCP

segments that can be transmitted. Hence the additional timeION nodes spend processing

custody signals, does not have an impact on the throughput. The number of bundles ION

can release for transmission with the custody signal processing overhead is still more than

what TCP can actually send at higher loss rates.

48

102

103

104

105

0 1 2 3 4 5 6 7 8 9 10

Thro

ughput (K

bits

/sec)

Loss (%)

theoretical max200 bytes

5000 bytes15000 bytes

40000 bytes

Figure 4.11: TCPCL(w/ custody) Throughput vs Loss

The last set of tests measure the performance of TCPCL with custody for a

combination of different RTT and loss rates. The throughput measure for 200, 5000,

15000 and 40000 byte bundles are shown in Figure4.13, Figure4.14, Figure4.15and

Figure4.16respectively.

The performance of TCPCL with custody transfer for a combination of delay and

loss is similar to the performance of TCPCL without custody transfer. Similar to TCPCL

49

102

103

104

105

0 1 2 3 4 5 6 7 8 9 10

Thro

ughput (K

bits

/sec)

Loss (%)

theoretical max200 bytes without custody

200 bytes with custody

15000 bytes without custody15000 bytes with custody

Figure 4.12: Comparing TCPCL(w/ custody) and TCPCL(w/o custody) Throughtput vs

Loss

without custody, the throughput curve for TCPCL with custody is identical to the

maximum theoretical throughput curve as given by the Mathisequation.

4.2 LTP Convergence Layer Tests

We discuss the results of performance tests using LTP convergence layer over

UDP/IP in this section. The first subsection presents the resultsof LTPCL without custody

transfer followed by the results of performance tests of LTPCL with custody.

50

100

101

102

103

104

105

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Thro

ughp

ut (K

bits

/sec

)

RTT (sec)

0% Losstheoretical max(.05 % loss)

.05% Losstheoretical max(2 % loss)

2% Losstheoretical max(5 % loss)

5% Loss

Figure 4.13: TCPCL(w/ custody) Throughput vs RTT & Loss for 200 Byte Bundles

4.2.1 LTPCL Without custody transfer

The first set of tests measure the throughput of LTPCL withoutcustody transfer for

different delay between Node2 and Node3. The SBPIperf client sends 25MB of data to

the SBPIperf server. Refer to SectionA.2 for the LTPCL configuration files used in ION

for the tests.

The performance of LTPCL without custody transfer for various RTT is shown in

Figure4.17. The LTP configuration file uses a command called span to control the

transmission rate. The main parameters in span that regulate transmission are aggregation

size, LTP segment size, and number of export sessions. The span parameters between

Node1-Node2 and Node3-Node4 always remain the same becausedelay and loss is

51

100

101

102

103

104

105

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Thro

ughp

ut (K

bits

/sec

)

RTT (sec)

0% Losstheoretical max (.05% loss)

.05% Losstheoretical max (2% loss)

2% Losstheoretical max (5% loss)

5% Loss

Figure 4.14: TCPCL(w/ custody) Throughput vs RTT & Loss for 5000 Byte Bundles

applied between Node2 and Node3. This would mean only Node2-Node3 span has to be

tweaked for different delays and losses.

Aggregation size is the amount of data that ION will aggregate into an LTP block

before using an LTP session to transmit the data. As soon as ION accumulates data equal

to or more than the aggregation size, ION will break the LTP block into LTP segments and

hand it down to the UDP layer for transmission. However, if the internal clock within ION

triggers before reaching the aggregation size, ION will segment the LTP block and send it

anyway. LTP divides the LTP block into LTP segments of size specified by the LTP

segment size parameter within the span. Export session configuration, as explained earlier,

is a method to do flow control in LTP.

52

100

101

102

103

104

105

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Thro

ughp

ut (K

bits

/sec

)

RTT (sec)

0% Losstheoretical max(.05% loss)

.05% Losstheoretical max(2% loss)

2% Losstheoretical max(5% loss)

5% Loss

Figure 4.15: TCPCL(w/ custody) Throughput vs RTT & Loss for 15000 Byte Bundles

A link with no artificial delay uses smaller number of export session (2 in this case)

and a higher LTP segment size (5600 bytes). Increasing the number of export session

increases the amount of data transmitted, but it also risks the chances of packets getting

dropped because of buffer overflow. The intermediate nodes do twice as much work as the

end nodes. Thus increasing the number of the export sessionswill send more bundles to

the intermediate node than it can process. Packet loss will cause a drop in throughput

because the lost packets will have to be retransmitted. Additionally, using a higher LTP

segment size will reduce the time ION spends segmenting an LTP block. In turn, the IP

layer will segment the bigger LTP segment. The advantage of this approach is, IP

segmentation takes less time than LTP segmentation, and hence throughput will be higher.

On the other hand, losing one IP segment of the bigger LTP segment will mean LTP will

53

100

101

102

103

104

105

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

Thro

ughp

ut (K

bits

/sec

)

RTT (sec)

0% Losstheoretical max(.05% loss)

.05% Losstheoretical max(2% loss)

2% Losstheoretical max(5% loss)

5% Loss

Figure 4.16: TCPCL(w/ custody) Throughput vs RTT & Loss for 40000 Byte Bundles

have to resend the whole LTP segment again. A link with no delay uses lesser number of

export sessions: hence, we can use the IP segmentation of theLTP segment to our

advantage, since losses will be minimal.

Table 4.1: Number of sessions used for various RTTs

No. of Sessions RTT(sec)

2 0

50 .2

200 1

400 >1

54

103

104

105

0 5 10 15 20

Thro

ughp

ut (K

bits

/sec

)

RTT (sec)

5000 bytes 15000 bytes 40000 bytes

Figure 4.17: LTPCL Throughtput vs RTT

For a link with delay we use a higher number of export sessionsand a lower LTP

segment size (1400 bytes) on the nodes connected to the delayed link. Table4.1shows the

number of export sessions that we configure LTP to use for various RTTs. The extra

export sessions will guarantee that there is constant dialogue between the sender and the

receiver. Consider a case where the number of export sessions is 10 and the delay between

the nodes is 5 seconds. If ION can fill 10 sessions worth of dataat a time, once we send 10

sessions of data, ION needs to wait for 10 seconds to receive the corresponding reports

from the receiver so it can clear out the session. And only after the session is cleared can it

be re-used to send data. However, if we have 400 export sessions, then we can use the

other 390 unused sessions to transmit data while waiting forthe report segments from the

first 10 sessions.

55

102

103

104

105

0 5 10 15 20

Thro

ughp

ut (K

bits

/sec

)

RTT (sec)

5000 bytes with LTPCL5000 bytes with TCPCL

15000 bytes with LTPCL15000 bytes with TCPCL

Figure 4.18: Comparing LTPCL and TCPCL Throughput vs RTT

Figure4.18shows the comparison in performance of LTPCL and TCPCL on link

with delay. TCP is optimized to perform well on links with minimal delay. But on links

with higher delays, LTP can be fine tuned to obtain higher performance than TCP. Unlike

TCP, LTP is designed to use the available bandwidth at all time. TCP spends a

considerable amount of time in slow start/congestion avoidance on high delay links, but

LTP is always transmitting at the constant configured transmission rate. As mentioned

earlier, the transmission rate is controlled by the number of export sessions, aggregation

size, and LTP segment size. The heap memory reserved for LTP depends on the number of

sessions.

56

LTP space reservation= (Max export bundle size× No. of export sessions)+

(Max import bundle size× No. of import sessions)

If ION is configured such that heap memory allocated for LTP isless than the actual

LTP space reservation, then ION will store the extra LTP blocks in files instead of storing

it on the heap. However, writing and reading from a file will bemore expensive than

writing and reading from the heap. Hence reserving the rightamount of space for LTP

blocks is important for the performance of LTP.

The amount of heap reserved for LTP should also not exceed thetotal amount of heap

memory allocated to ION. Hence the number of session that canbe configured for LTP

indirectly depends on the memory. Therefore, the LTP test results are constrained by the

amount of memory allocated to ION and also by the OS buffer size for UDP connections.

The maximum number of sessions that we use in our tests is 400.Theoretically we can

open more sessions on links with high delay to get better throughput, but practically we

are constrained by the memory.

The LTPCL throughput for various amounts of packet loss is shown in Figure4.19.

When an LTP segment is lost, the LTP sender and receiver will keep the session for the

block open until the LTP segment is retransmitted. The drop in throughput is mainly

because of the retransmission of lost segments and under utilization of the session. The

session which has a lost segment will have to complete retransmission of the LTP segment

before the session can be re-used to send other LTP blocks. Not only does this prevent

other LTP blocks from using this session, but it also under utilizes the session during

retransmission. Take for example a LTP block of size 40000 bytes and LTP segment size

of 1500 bytes. This would require a session to send approximately 25 segments of LTP

blocks. If 2 segments in this session are lost, the retransmission will send the 2 segments.

57

103

104

105

0 2 4 6 8 10

Thro

ughp

ut (K

bits

/sec

)

Loss(%)

5000 bytes 15000 bytes 40000 bytes

Figure 4.19: LTPCL Throughput vs Loss

Thus a session which would generally send 40000 bytes duringthe retransmission phase

sends only 3000 bytes. The situation worsens with loss because more sessions will have to

retransmit lost segments. Hence the performance reduces with loss.

The behavior of ION on losing different segments is explained below

• Data segment - When a data segment is lost, the loss is reported by the receiver

when it sends a report segment. The sender on receiving the report, acknowledges

the report segment and retransmits the lost segment.

• Checkpoint segment- After sending a checkpoint segment, the sender will start a

checkpoint retransmission timer. If the report segment doesn’t arrive before the

timer expires, it will resend the checkpoint segment. Consequently, on losing a

58

checkpoint segment, the retransmission timer will expire,forcing the sender to

retransmit the checkpoint segment.

• Report Segment - Loss of a report segment will cause the checkpoint retransmission

timer on the sender side and the report segment retransmission timer on the receiver

side to expire. This will result in both checkpoint retransmission and a report

retransmission.

• Report Acknowledgement - The receiver will send a report segment, when it

receives a checkpoint segment from the sender. The sender responds to the report

segment with a report acknowledgment. If the receiver reports that all the segments

were received successfully, the sender will send a report acknowledgement and

close the export session. However, if the report acknowledgement is lost, the report

retransmission timer on the receiver side will expire, and it will retransmit a report

segment. Since the sender closed the session, it will have norecord of the session

the report segment is reporting about, so it will discard thereport segment. The

receiver will retransmit the report segment until the number of retransmissions reach

a preset retransmission max. On reaching the retransmission maximum, the receiver

will cancel the session. However, the sender will still haveno record of the session

that the receiver is trying to cancel, so it will discard the cancel segment.

Furthermore the receiver will retransmit the cancel segment until it reaches the

retransmission maximum.

Figure4.20gives the performance comparison of LTPCL and TCPCL versus loss. As

mentioned earlier, LTP under utilizes sessions during retransmission of lost LTP segments.

The session transmits less data than it would normally send during regular transmission.

TCP, on losing a packet will lower its congestion window by half, hence transmitting half

as much as it was previously transmitting. The congestion window size of the TCP sender

59

102

103

104

105

0 2 4 6 8 10

Thro

ughp

ut (K

bits

/sec

)

Loss (%)

5000 bytes with LTPCL5000 bytes with TCPCL

15000 bytes with LTPCL15000 bytes with TCPCL

Figure 4.20: Comparing LTPCL and TCPCL Throughput vs Loss

is increased by one segment each round trip time until it reaches advertised receive

window size, or there is another packet loss. However, when the loss rate is high, the

congestion window constantly remains low, and hence the throughput degrades constantly

with loss. On the other hand, LTP’s behavior remains approximately the same for low loss

and high loss. Therefore, the performance of LTPCL is betterthan TCPCL for greater

losses.

The performance of LTPCL without custody for delay and loss combination is shown

in Figure4.21and Figure4.22. The performance of LTPCL on a link with delay and loss

is similar to the performance of LTPCL with only loss. The number of export session that

can be used to transmit LTP blocks will help keep a constant dialogue between the sender

60

103

104

105

0 1 2 3 4 5

Thro

ughp

ut (K

bits

/sec

)

RTT (sec)

0% Loss .05% Loss 2% Loss 5% Loss

Figure 4.21: LTPCL Throughput vs RTT & Loss for 5000 Byte Bundles

and the receiver even on links with some delay. Hence the performance of LTP on links

with RTT less than 5 seconds is determined by the loss on the link.

4.2.2 LTPCL With custody transfer

This section presents the results of LTPCL with custody transfer on a link with

different delays and losses. Figure4.23displays the throughput of LTPCL with custody

for various RTTs and compares the throughput of LTP with custody and without custody.

As explained in the Section4.1.2, when custody transfer is enabled, the nodes needs to

spend extra time handling custody signals. The intermediate nodes in the setup have to

handle incoming bundles, routing bundles and transmittingbundles if custody transfer is

not enabled. When custody transfer is enabled, these nodes additionally need to send

61

103

104

105

0 1 2 3 4 5

Thro

ughp

ut (K

bits

/sec

)

RTT (sec)

0% Loss .05% Loss 2% Loss 5% Loss

Figure 4.22: LTPCL Throughput vs RTT & Loss for 15000 Byte Bundles

custody signals to the previous custodian and handle incoming custody signals from the

next custodian. The additional processing reduces the transmission rate for LTPCL with

custody transfer.

As Figure4.24illustrates, the performance of LTPCL with custody is better than LTP

without custody when losses are high. A link with lower loss acts similar TCPCL, where

the throughput of LTPCL without custody is higher than LTPCLwith custody. The

difference in throughput for low loss links can be attributed to custody signal processing

overhead. Contrary to that, links with high loss have a high probability of losing custody

signals. Custody signals are transferred between administrative endpoints on the nodes

and do not use the sessions that are used for transmitting data. Losing a data segment or a

report segment would entail the session to be open until the segment is retransmitted.

62

103

104

105

0 5 10 15 20

Thro

ughput (K

bits

/sec)

RTT (sec)

5000 bytes without custody5000 bytes with custody

15000 bytes without custody15000 bytes with custody

Figure 4.23: Comparing LTPCL(w/o custody) and LTPCL(w/ cusotdy) Throughput vs

Delay

Therefore, the session needs to be open for a longer time, andit prevents the session from

being reused for other bundles. On the contrary, losing a custody signal will not affect the

session behavior and consequently will not affect the transmission rate. Losing custody

signals has bigger consequences in the long run, as mentioned earlier, but in this test

scenario results in better performance.

63

103

104

105

0 2 4 6 8 10

Thro

ughput (K

bits

/sec)

Loss (%)

5000 bytes without custody5000 bytes with custody

15000 bytes without custody15000 bytes with custody

Figure 4.24: Comparing LTPCL(w/o custody) and LTPCL(w/ custody) Throughput vs

Loss

Figure4.25and Figure4.26illustrates the throughput of LTPCL with custody over

links with both delay and loss.

64

103

104

0 1 2 3 4 5

Thro

ughp

ut (K

bits

/sec

)

RTT (sec)

0% Loss .05% Loss 2% Loss 5% Loss

Figure 4.25: LTPCL(w/ custody) Throughput vs RTT & Loss for 5000 Byte Bundles

65

103

104

105

0 1 2 3 4 5

Thro

ughp

ut (K

bits

/sec

)

RTT (sec)

0% Loss .05% Loss 2% Loss 5% Loss

Figure 4.26: LTPCL(w/custody) Throughput vs RTT & Loss for 15000 Byte Bundles

66

5 Conclusions

This thesis evaluates the maximum performance that can be expected from TCPCL

and LTPCL in extreme network conditions. The throughput data establishes a reference

point for the performance of the LTPCL and TCPCL in those testenvironments. The

constraints on throughput imposed by some external factorsare also explained in the

document. Our study also focused on setting up the TCPCL and LTPCL optimally, to

elicit maximum performance while at the same time avoiding configuration and code bugs.

As we would expect, TCP performs well under conditions with minimal delay and

loss. TCP has been optimized to perform extremely well in such conditions. TCP

automatically controls its transmission rate under extreme conditions, which will help to

prevent further losses and delays. Although this works wellin environments with

end-to-end connectivity, this is not the best approach whenbandwidth and connectivity

are a deficient resource.

LTPCL on the other hand, performs better than TCPCL in conditions with high delay

and high loss. LTP, by design takes advantage of the available bandwidth to transmit as

much data as possible when the connection is open. In the caseof the ION version of LTP,

its performance is highly dependent on the shared memory that can be allocated to ION in

the system. This is because the number of sessions depends onthe amount of memory

available, which in turn determines LTP’s transmission rate. Hence on a system that can

allocate more memory to ION, we can expect the performance ofLTP to be higher than

what is shown in these tests.

TCP dynamically adjusts its transmission rate depending onthe operating

environment. On the contrary, LTP has to be pre-configured depending on the

environment. Since there are so many parameters that have tobe configured to fine tune

LTP’s performance, it would be useful to abstract away some of the parameters in the

67

code, so it will be easier to set up LTP in ION. Some more resources and tools to aid in

setting up LTP will also be valuable.

The performance of these convergence layers on a link with disconnection was not

studied as a part of this thesis. Other than delay and loss, disconnection is another

important factor in Delay Tolerant Networks, so it is important to map the performance of

these convergence layers on a link with intermittent connectivity. Also profiling the other

commonly used convergence layers will give us a better idea of what convergence layers

best suit a specific setting.

Finally, a formal study of how the various LTP parameters - maximum bundle size,

number of sessions, aggregation size, aggregation time, and LTP segment size - affect the

performance of LTP will also be an interesting experiment for future work.

68

References

Akan, O. B., Fang, J.,and Akyildiz, I. F. 2002. Performance of TCP Protocols in DeepSpace Communication Networks.IEEE Communications Letters 6, 11, 478–480.

Akyildiz, I. F., Morabito, G.,and Palazzo, S. 2001. TCP-Peach: A New CongestionControl Scheme for Satellite.IP Networks, IEEE/ACM Trans. Networking 9,307–321.

Allman, M., Paxson, V., and Blanton, E. 2009. TCP Congestion Control. RFC 5681(Draft Standard).

Burleigh, S., Hooke, A., Torgerson, L., Fall, K., Cerf, V., Durst, B., Scott, K., andWeiss, H. 2003. Delay-Tolerant Networking: An Approach to InterplanetaryInternet.IEEE Communications Magazine 41, 6, 128–136.

Burleigh, S., Ramadas, M., and Farrell, S. 2008. Licklider Transmission Protocol -Motivation. RFC 5325 (Informational).

CCSDS. 2006. Proximity-1 Space Link Protocol - Data Link Layer. Technical ReportBlue Book, CCSDS 211.0-B-4, The Consultative Committee forSpace DataSystems, Washington, DC. July.

Demmer, M. and Ott, J. 2008. Delay Tolerant Networking TCP Convergence LayerProtocol. draft-irtf-dtnrg-tcp-clayer-02.txt.

Doering, M., Lahde, S., Morgenroth, J.,andWolf, L. 2008. IBR-DTN: an efficientimplementation for embedded systems. InProceedings of the third ACM workshopon Challenged networks. CHANTS ’08. ACM, New York, NY, USA, 117–120.

DTNRG. 2010. Dtn reference implementation.

Durst, R. C., Miller, G. J.,and Travis, E. J. 1996. TCP Extensions for SpaceCommunications. InProceedings of MOBICOM ’96.

Eddy, W. and Davies, E. 2011. Using Self-Delimiting Numeric Values in Protocols.draft-irtf-dtnrg-sdnv-09.

Fall, K. 2003. A Delay-Tolerant Network Architecture for Challenged Internets. InSIGCOMM 03: Proceedings from 2003 conference on Applications, technologies,architectures, and protocols for computer communications. ACM, 27–34.

Farrell, S.and Cahill, V. 2006.Delay- and Disruption-Tolerant Networking, First ed.Artech House.

69

Hemminger, S. 2005. Network Emulation with NetEm. InLCA 2005, Australia’s 6thnational Linux conference (linux.conf.au), M. Pool, Ed. Linux Australia, SydneyNSW, Australia.

Henderson, T. R.and Katz, R. H. 1999. Transport Protocols for Internet-CompatibleSatellite Networks.IEEE J. Select. Areas Commun 17, 326–344.

Juang, P., Oki, H., Wang, Y., Martonosi, M., Peh, L. S.,and Rubenstein, D. 2002.Energy-Efficient Computing for Wildlife Tracking: Design Tradeoffs and EarlyExperiences with ZebraNet. InProceedings of the 10th international conferenceon Architectural support for programming languages and operating systems.ASPLOS-X. ACM, New York, NY, USA, 96–107.

Kohler, E., Handley, M., and Floyd, S. 2006. Datagram Congestion Control Protocol(DCCP). RFC 4340 (Proposed Standard). Updated by RFCs 5595,5596.

Kruse, H. and Ostermann, S. 2008. UDP Convergence Layers for the DTN Bundle andLTP Protocols. draft-irtf-dtnrg-udp-clayer-00 (work in progress).

Marchese, M., Rossi, M., and Morabito, G. 2004. PETRA: Performance EnhancingTransport Architecture for Satellite Communications.Selected Areas inCommunications, IEEE Journal on 22, 2, 320 – 332.

Mathis, M., Semske, J., Mahdavi, J.,and Ott, T. 1997. The Macroscopic Behavior ofthe TCP Congestion Avoidance Algorithm.Computer Communication Review 27.

Ramadas, M. 2007. Phd dissertation. Ph.D. thesis, Ohio University,Department ofElectrical Engineering and Computer Science.

Scott, K. and Burleigh, S. 2007. Bundle Protocol Specification. RFC5050(Experimental).

Wang, R., Wu, X., Wang, T., Liu, X., and Zhou, L. 2010. TCP ConvergenceLayer-Based Operation of DTN for Long-Delay Cislunar Communications.Systems Journal, IEEE 4, 3, 385 –395.

Wood, L., Eddy, W., and Holliday, P. 2009. A Bundle of Problems. InProceedings fromIEEE Aerospace Conference. IEEE, Big Sky, Montana.

Wood, L., Ivancic, W., Eddy, W. M., Stewart, D., Northam, J., Jackson, C.,and Curiel,A. D. S. 2008. Use of the Delay-Tolerant Networking Bundle Protocol fromSpace. InProceedings from 59th International Astronautical Congress.

70

Appendix A: ION Configuration

A.1 TCPCL Configuration

A.1.1 Node1 configuration file

## begin ionadmin

# ionrc configuration file for 4node-tcp-test for Node1

# This uses tcp as the primary convergence layer.

# This command should be run FIRST.

#

# Initialization command (command 1).

# Set this node to be node 1 (as in ipn:1).

# Use sdr configuration in

# "/home/testsuite/scripts/host1/ion.conf"

1 1 /home/testsuite/scripts/host1/ion.conf

# start ion node

s

## end ionadmin

## begin ionsecadmin

1

e 1

## end ionsecadmin

## begin bpadmin

71

# bprc configuration file for the 4node-tcp-test

# This command should be run AFTER ionadmin and ltpadmin and

# BEFORE ipnadmin or dtnadmin.

#

# Initialization command (command 1).

1

# Add an EID scheme.

# The scheme’s name is ipn.

# This scheme’s forwarding engine is handled by the program

# ’ipnfw.’

# This scheme’s administration program (acting as the

# custodian daemon) is ’ipnadminep.’

a scheme ipn ’ipnfw’ ’ipnadminep’

# Add endpoints.

# Establish endpoints ipn:1.1 and ipn:1.2 on the local node.

# The behavior for receiving a bundle when there is no

# application currently accepting bundles, is to queue them

# ’q’, as opposed to immediately and silently discarding them

# (use ’x’ instead of ’q’ to discard).

# Note that the custodian endpoint "ipn:1.0" is automatically

# generated.

a endpoint ipn:1.1 q

a endpoint ipn:1.2 q

72

# Add a protocol.

# Add the protocol named tcp.

# Estimate transmission capacity assuming 1400 bytes of each

# frame (in this case, tcp on ethernet) for payload, and 100

# bytes for overhead.

a protocol tcp 1400 100

# Add an induct. (listen)

# Add an induct to accept bundles using the tcp protocol.

# The induct will listen at the loopback IP address.

# The induct will listen on port 4556, the IANA assigned

# default DTN TCP convergence layer port.

# The induct itself is implemented by the ’tcpcli’ command.

a induct tcp testbed1.cs.ohio.edu:4556 tcpcli

# Add an outduct.

# Add an outduct to send bundles using the tcp protocol.

# The outduct itself is implemented by the ’tcpclo’ command.

a outduct tcp testbed1.cs.ohio.edu:4556 tcpclo

a outduct tcp testbed6.cs.ohio.edu:4556 tcpclo

s

## end bpadmin

## begin ipnadmin

# ipnrc configuration file for the 4node-tcp-test

73

# Essentially, this is the IPN scheme’s routing table.

# This command should be run AFTER bpadmin (likely to be run

# last).

# Add an egress plan.

# Bundles to be transmitted to element number 1 (that is,

# yourself). Transmission should use the tcp convergence

# layer tcp/testbed1.cs.ohio.edu:4556. All other transmission

# uses the tcp convergence layer

# tcp/testbed6.cs.ohio.edu:4556

# See your bprc file or bpadmin for outducts/protocols you

# can use.

a plan 1 tcp/testbed1.cs.ohio.edu:4556

a plan 2 tcp/testbed6.cs.ohio.edu:4556

a plan 3 tcp/testbed6.cs.ohio.edu:4556

a plan 4 tcp/testbed6.cs.ohio.edu:4556

## end ipnadmin

A.1.2 Node2 configuration file

## begin ionadmin

1 2 /home/testsuite/scripts/host2/ion.conf

s

## end ionadmin

74

## begin ionsecadmin

1

e 1

## end ionsecadmin

## begin bpadmin

1

a scheme ipn ’ipnfw’ ’ipnadminep’

a endpoint ipn:2.1 q

a endpoint ipn:2.2 q

a protocol tcp 1400 100

a induct tcp testbed6.cs.ohio.edu:4556 tcpcli

a outduct tcp testbed1.cs.ohio.edu:4556 tcpclo

a outduct tcp testbed6.cs.ohio.edu:4556 tcpclo

a outduct tcp testbed7.cs.ohio.edu:4556 tcpclo

s

## end bpadmin

## begin ipnadmin

a plan 1 tcp/testbed1.cs.ohio.edu:4556

a plan 2 tcp/testbed6.cs.ohio.edu:4556

75

a plan 3 tcp/testbed7.cs.ohio.edu:4556

a plan 4 tcp/testbed7.cs.ohio.edu:4556

## end ipnadmin

A.1.3 Node3 configuration file

## begin ionadmin

1 3 /home/testsuite/scripts/host3/ion.conf

s

## end ionadmin

## begin ionsecadmin

1

e 1

## end ionsecadmin

## begin bpadmin

1

a scheme ipn ’ipnfw’ ’ipnadminep’

a endpoint ipn:3.1 q

a endpoint ipn:3.2 q

a protocol tcp 1400 100

76

a induct tcp testbed7.cs.ohio.edu:4556 tcpcli

a outduct tcp testbed6.cs.ohio.edu:4556 tcpclo

a outduct tcp testbed7.cs.ohio.edu:4556 tcpclo

a outduct tcp testbed8.cs.ohio.edu:4556 tcpclo

s

## end bpadmin

## begin ipnadmin

a plan 1 tcp/testbed6.cs.ohio.edu:4556

a plan 2 tcp/testbed6.cs.ohio.edu:4556

a plan 3 tcp/testbed7.cs.ohio.edu:4556

a plan 4 tcp/testbed8.cs.ohio.edu:4556

## end ipnadmin

A.1.4 Node4 configuration file

## begin ionadmin

1 4 /home/testsuite/scripts/host4/ion.conf

s

## end ionadmin

## begin ionsecadmin

1

e 1

77

## end ionsecadmin

## begin bpadmin

1

a scheme ipn ’ipnfw’ ’ipnadminep’

a endpoint ipn:4.1 q

a endpoint ipn:4.2 q

a protocol tcp 1400 100

a induct tcp testbed8.cs.ohio.edu:4556 tcpcli

a outduct tcp testbed7.cs.ohio.edu:4556 tcpclo

a outduct tcp testbed8.cs.ohio.edu:4556 tcpclo

s

## end bpadmin

## begin ipnadmin

a plan 1 tcp/testbed7.cs.ohio.edu:4556

a plan 2 tcp/testbed7.cs.ohio.edu:4556

a plan 3 tcp/testbed7.cs.ohio.edu:4556

a plan 4 tcp/testbed8.cs.ohio.edu:4556

## end ipnadmin

78

A.2 LTPCL Configuration

A.2.1 Node1 configuration file

# This is the config file for the 4node LTP test

## begin ionadmin

# ionrc configuration file for the 4node LTP test

# This uses ltp as the primary convergence layer.

# Initialization command (command 1).

# Set this node to be node 1 (as in ipn:1).

# Use sdr configuration in

# /home/testsuite/scripts/host1/ion.conf

1 1 /home/testsuite/scripts/host1/ion.conf

# start ion node

s

# Add a contact.

# It will start at +1 seconds from now, ending +36000 seconds

# from now.

# It will connect node 1 to itself

# It will transmit 100000000 bytes/second.

a contact +1 +36000 1 1 100000000

a contact +1 +36000 1 2 100000000

79

a contact +1 +36000 2 1 100000000

a contact +1 +36000 2 2 100000000

a contact +1 +36000 2 3 100000000

a contact +1 +36000 3 2 100000000

a contact +1 +36000 3 3 100000000

a contact +1 +36000 3 4 100000000

a contact +1 +36000 4 3 100000000

a contact +1 +36000 4 4 100000000

# Add a range. This is the physical distance between nodes.

# It will start at +1 seconds from now, ending +36000 seconds

# from now.

# It will connect node 1 to itself.

# Data on the link is expected to take 1 second to

# reach the other end (One Way Light Time).

a range +1 +36000 1 1 1

a range +1 +36000 1 2 1

a range +1 +36000 1 3 1

a range +1 +36000 1 4 1

a range +1 +36000 2 3 1

a range +1 +36000 2 4 1

a range +1 +36000 3 4 1

# set this node to consume and produce a mean of

# 1000000 bytes/second.

80

m production 100000000

m consumption 100000000

## end ionadmin

## begin ionsecadmin

1

e 1

## end ionsecadmin

## begin ltpadmin

# ltprc configuration file for the 4 node ltp test

# Initialization command (command 1)

# Set total number estimated export session to 8

# LTP space reservation is 30000000

# which should be >= (max export size * max export bundle size) +

# (max import size * max import bundle size)

# for all spans

1 8 30000000

# a b c d e f g h i

a span 1 2 100000 2 100000 5600 10000 1

’udplso testbed1.cs.ohio.edu:1113’

a span 2 2 100000 2 100000 5600 10000 1

’udplso testbed6.cs.ohio.edu:1113’

#

# a = remote LTP engine number

81

# b = max export sessions

# c = max exort bundle size

# d = max import sessions

# e = max import bundle size

# f = LTP segment size

# g = aggregation size

# h = aggregation time

s ’udplsi testbed1.cs.ohio.edu:1113’

## end ltpadmin

## begin bpadmin

# bprc configuration file for the 4 node ltp test

# This command should be run AFTER ionadmin and ltpadmin and

# BEFORE ipnadmin or dtnadmin.

#

# Initialization command (command 1).

1

# Add an EID scheme.

# The scheme’s name is ipn.

# This scheme’s forwarding engine is handled by the

# program ’ipnfw.’

# This scheme’s administration program (acting as the

# custodian daemon) is ’ipnadminep.’

82

a scheme ipn ’ipnfw’ ’ipnadminep’

# Add endpoints.

# Establish endpoints ipn:1.1 and ipn:1.2 on the local node.

# The behavior for receiving a bundle when there is no

# application currently accepting bundles, is to queue

# them ’q’, as opposed to immediately and silently

# discarding them (use ’x’ instead of ’q’ to discard).

# Note that the custodian endpoint "ipn:1.0" is

# automatically generated.

a endpoint ipn:1.1 q

a endpoint ipn:1.2 q

# Add a protocol.

# Add the protocol named ltp.

# Estimate transmission capacity assuming 1400 bytes of

# each frame (in this case, tcp on ethernet) for payload,

# and 100 bytes for overhead.

a protocol ltp 1400 100

# Add an induct. (listen)

# Add an induct to accept bundles using the ltp protocol.

# The induct itself is implemented by the ’ltpcli’ command.

a induct ltp 1 ltpcli

# Add an outduct. (send to yourself)

# Add an outduct to send bundles using the ltp protocol.

83

# The outduct itself is implemented by the ’ltpclo’ command.

a outduct ltp 1 ltpclo

a outduct ltp 2 ltpclo

s

## end bpadmin

## begin ipnadmin

# ipnrc configuration file for the 4node ltp test

# Essentially, this is the IPN scheme’s routing table.

# This command should be run AFTER bpadmin

# (likely to be run last).

#

# Add an egress plan.

# Bundles to be transmitted to element number 1

# (that is, yourself).

# Transmission should use the ltp span represented by

# engine number 1

# Bundles to be transmitted to element number 2.

# Transmission should use the ltp span

# represented by engine number 2

a plan 1 ltp/1

a plan 2 ltp/2

a plan 3 ltp/2

a plan 4 ltp/2

84

## end ipnadmin

A.2.2 Node2 configuration file

## begin ionadmin

1 2 /home/testsuite/scripts/host2/ion.conf

s

a contact +1 +36000 1 1 100000000

a contact +1 +36000 1 2 100000000

a contact +1 +36000 2 1 100000000

a contact +1 +36000 2 2 100000000

a contact +1 +36000 2 3 100000000

a contact +1 +36000 3 2 100000000

a contact +1 +36000 3 3 100000000

a contact +1 +36000 3 4 100000000

a contact +1 +36000 4 3 100000000

a contact +1 +36000 4 4 100000000

a range +1 +36000 1 1 1

a range +1 +36000 1 2 1

a range +1 +36000 1 3 1

a range +1 +36000 1 4 1

a range +1 +36000 2 3 1

a range +1 +36000 2 4 1

85

a range +1 +36000 3 4 1

m production 100000000

m consumption 100000000

## end ionadmin

## begin ionsecadmin

1

e 1

## end ionsecadmin

## begin ltpadmin

1 808 70000000

#configuration used for no delay no loss links

#1 12 30000000

a span 1 2 100000 2 100000 5600 10000 1

’udplso testbed1.cs.ohio.edu:1113’

a span 2 2 100000 2 100000 5600 10000 1

’udplso testbed6.cs.ohio.edu:1113’

#configuration used for no delay no loss links

#a span 3 2 100000 2 100000 5600 10000 1

’udplso testbed7.cs.ohio.edu:1113’

a span 3 400 100000 400 100000 1400 10000 1

86

’udplso testbed7.cs.ohio.edu:1113’

s ’udplsi testbed6.cs.ohio.edu:1113’

## end ltpadmin

## begin bpadmin

1

a scheme ipn ’ipnfw’ ’ipnadminep’

a endpoint ipn:2.1 q

a endpoint ipn:2.2 q

a protocol ltp 1400 100

a induct ltp 2 ltpcli

a outduct ltp 1 ltpclo

a outduct ltp 2 ltpclo

a outduct ltp 3 ltpclo

s

## end bpadmin

## begin ipnadmin

a plan 1 ltp/1

87

a plan 2 ltp/2

a plan 3 ltp/3

a plan 4 ltp/3

## end ipnadmin

A.2.3 Node3 configuration file

## begin ionadmin

1 3 /home/testsuite/scripts/host3/ion.conf

s

a contact +1 +36000 1 1 100000000

a contact +1 +36000 1 2 100000000

a contact +1 +36000 2 1 100000000

a contact +1 +36000 2 2 100000000

a contact +1 +36000 2 3 100000000

a contact +1 +36000 3 2 100000000

a contact +1 +36000 3 3 100000000

a contact +1 +36000 3 4 100000000

a contact +1 +36000 4 3 100000000

a contact +1 +36000 4 4 100000000

a range +1 +36000 1 1 1

a range +1 +36000 1 2 1

88

a range +1 +36000 1 3 1

a range +1 +36000 1 4 1

a range +1 +36000 2 3 1

a range +1 +36000 2 4 1

a range +1 +36000 3 4 1

m production 100000000

m consumption 100000000

## end ionadmin

## begin ionsecadmin

1

e 1

## end ionsecadmin

## begin ltpadmin

# configuration used for no delay no loss links

#1 12 30000000

1 808 90000000

# configuration used for no delay no loss links

#a span 2 2 100000 2 100000 5600 10000 1

’udplso testbed6.cs.ohio.edu:1113’

a span 2 400 100000 400 100000 1400 10000 1

89

’udplso testbed6.cs.ohio.edu:1113’

a span 3 2 100000 2 100000 5600 10000 1

’udplso testbed7.cs.ohio.edu:1113’

a span 4 2 100000 2 100000 5600 10000 1

’udplso testbed8.cs.ohio.edu:1113’

s ’udplsi testbed7.cs.ohio.edu:1113’

## end ltpadmin

## begin bpadmin

1

a scheme ipn ’ipnfw’ ’ipnadminep’

a endpoint ipn:3.1 q

a endpoint ipn:3.2 q

a protocol ltp 1400 100

a induct ltp 3 ltpcli

a outduct ltp 2 ltpclo

a outduct ltp 3 ltpclo

a outduct ltp 4 ltpclo

s

90

## end bpadmin

## begin ipnadmin

a plan 1 ltp/2

a plan 2 ltp/2

a plan 3 ltp/3

a plan 4 ltp/4

## end ipnadmin

A.2.4 Node4 configuration file

## begin ionadmin

1 4 /home/testsuite/scripts/host4/ion.conf

s

a contact +1 +36000 1 1 100000000

a contact +1 +36000 1 2 100000000

a contact +1 +36000 2 1 100000000

a contact +1 +36000 2 2 100000000

a contact +1 +36000 2 3 100000000

a contact +1 +36000 3 2 100000000

a contact +1 +36000 3 3 100000000

a contact +1 +36000 3 4 100000000

91

a contact +1 +36000 4 3 100000000

a contact +1 +36000 4 4 100000000

a range +1 +36000 1 1 1

a range +1 +36000 1 2 1

a range +1 +36000 1 3 1

a range +1 +36000 1 4 1

a range +1 +36000 2 3 1

a range +1 +36000 2 4 1

a range +1 +36000 3 4 1

m production 100000000

m consumption 100000000

## end ionadmin

## begin ionsecadmin

1

e 1

## end ionsecadmin

## begin ltpadmin

1 8 30000000

a span 3 2 100000 2 100000 5600 10000 1

’udplso testbed7.cs.ohio.edu:1113’

a span 4 2 100000 2 100000 5600 10000 1

92

’udplso testbed8.cs.ohio.edu:1113’

s ’udplsi testbed8.cs.ohio.edu:1113’

## end ltpadmin

## begin bpadmin

1

a scheme ipn ’ipnfw’ ’ipnadminep’

a endpoint ipn:4.1 q

a endpoint ipn:4.2 q

a protocol ltp 1400 100

a induct ltp 4 ltpcli

a outduct ltp 3 ltpclo

a outduct ltp 4 ltpclo

s

## end bpadmin

## begin ipnadmin

a plan 1 ltp/3

93

a plan 2 ltp/3

a plan 3 ltp/3

a plan 4 ltp/4

## end ipnadmin

94

Appendix B: Supporting programs

B.1 Netem Script

!/bin/bash

# This is the delay / loss emulator script on Node2

# There is a similar script in Node3

# This script takes 3 arguments - interface, delay, loss

#

#

echo "Delay Script.."

if [ $# -ne 3 ];

then

echo "Usage $0 <interface> <delay>ms/sec/usec <loss>" 1>&2

exit 1

fi

interface=$1

delay=$2

loss=$3

loss_s="$3%"

echo "Interface : " $interface

echo "Delay : " $delay

echo "Loss : " $loss

sudo tc qdisc add dev $interface root handle 1: prio

95

if [ $loss -gt 0 ];then

sudo tc qdisc add dev $interface parent 1:3 handle 30: \

netem delay $delay loss $loss_s limit 10000 1>&2

else

sudo tc qdisc add dev $interface parent 1:3 handle 30: \

netem delay $delay limit 10000 1>&2

fi

sudo tc filter add dev $interface protocol ip parent 1:0 prio 1 \

u32 match ip dst 132.235.3.39 match ip dport 1113 0xffff flowid 1:3

sudo tc filter add dev $interface protocol ip parent 1:0 prio 1 \

u32 match ip dst 132.235.3.39 match ip sport 1113 0xffff flowid 1:3

sudo tc filter add dev $interface protocol ip parent 1:0 prio 1 \

u32 match ip dst 132.235.3.39 match ip dport 4556 0xffff flowid 1:3

sudo tc filter add dev $interface protocol ip parent 1:0 prio 1 \

u32 match ip dst 132.235.3.39 match ip sport 4556 0xffff flowid 1:3

sudo tc filter add dev $interface protocol ip parent 1:0 prio 1 \

u32 match ip src 132.235.3.39 match ip dport 1113 0xffff flowid 1:3

sudo tc filter add dev $interface protocol ip parent 1:0 prio 1 \

u32 match ip src 132.235.3.39 match ip sport 1113 0xffff flowid 1:3

sudo tc filter add dev $interface protocol ip parent 1:0 prio 1 \

u32 match ip src 132.235.3.39 match ip dport 4556 0xffff flowid 1:3

sudo tc filter add dev $interface protocol ip parent 1:0 prio 1 \

u32 match ip src 132.235.3.39 match ip sport 4556 0xffff flowid 1:3

96

exit 0

B.2 SystemTap Script

#! /usr/bin/env stap

# This script number of times ION calls ipc_lock

global total_count

global wait_time

probe begin

{

printf("Log of ipc_lock wait time\n")

}

probe kernel.function("ipc_lock@ipc/util.c").return

{

total_count <<< 1

time_elapsed = gettimeofday_us() -

@entry(gettimeofday_us())

wait_time <<< time_elapsed

}

probe end

{

printf("Total sem_op calls : %d", @count(total_count))

97

printf("Total wait time in ipc_lock(us) : %d \n",

@sum(wait_time) )

}

B.3 SBP Iperf

B.3.1 SBP Server

/*---------------------------------------------------------------

* Copyright (c) 1999,2000,2001,2002,2003

* The Board of Trustees of the University of Illinois

* All Rights Reserved.

*---------------------------------------------------------------

* Permission is hereby granted, free of charge, to any person

* obtaining a copy of this software (Iperf) and associated

* documentation files (the "Software"), to deal in the Software

* without restriction, including without limitation the

* rights to use, copy, modify, merge, publish, distribute,

* sublicense, and/or sell copies of the Software, and to permit

* persons to whom the Software is furnished to do

* so, subject to the following conditions:

*

*

* Redistributions of source code must retain the above

* copyright notice, this list of conditions and

* the following disclaimers.

*

*

98

* Redistributions in binary form must reproduce the above

* copyright notice, this list of conditions and the following

* disclaimers in the documentation and/or other materials

* provided with the distribution.

*

*

* Neither the names of the University of Illinois, NCSA,

* nor the names of its contributors may be used to endorse

* or promote products derived from this Software without

* specific prior written permission.

*

* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,

* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES

* OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND

* NONINFRINGEMENT. IN NO EVENT SHALL THE CONTIBUTORS OR COPYRIGHT

* HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,

* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,

* ARISING FROM, OUT OF OR IN CONNECTION WITH THE

* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

* ________________________________________________________________

* National Laboratory for Applied Network Research

* National Center for Supercomputing Applications

* University of Illinois at Urbana-Champaign

* http://www.ncsa.uiuc.edu

* ________________________________________________________________

*

99

* Server.cpp

* by Mark Gates <mgates@nlanr.net>

* Ajay Tirumala (tirumala@ncsa.uiuc.edu>.

*

* This has been modified to run over DTN protocols using SBP-API

* by Mithun Roy <mr774007@ohio.edu>

* -----------------------------------------------------------------

* A server thread is initiated for each connection accept() returns.

* Handles sending and receiving data, and then closes socket.

* Changes to this version : The server can be run as a daemon

* ---------------------------------------------------------------*/

#define HEADERS()

#include "headers.h"

#include "Server.hpp"

#include "List.h"

#include "Extractor.h"

#include "Reporter.h"

#include "Locale.h"

/* -----------------------------------------------------------------

* Stores connected socket and socket info.

* ---------------------------------------------------------------*/

Server::Server( thread_Settings *inSettings ) {

100

mSettings = inSettings;

mBuf = NULL;

// initialize buffer

mBuf = new char[ mSettings->mBufLen ];

FAIL_errno( mBuf == NULL, "No memory for buffer\n", mSettings );

}

/* -----------------------------------------------------------------

* Destructor close socket.

* ---------------------------------------------------------------*/

Server::˜Server() {

if ( mSettings->mSock != INVALID_SOCKET ) {

#ifdef ION

sbp_close( mSettings->mSock );

#else

int rc = close( mSettings->mSock );

WARN_errno( rc == SOCKET_ERROR, "close" );

#endif

mSettings->mSock = INVALID_SOCKET;

}

DELETE_ARRAY( mBuf );

}

void Server::Sig_Int( int inSigno ) {

101

}

/* -----------------------------------------------------------------

* Receieve data from the (connected) TCP/UDP socket.

* Sends termination flag several times at the end.

* Does not close the socket.

* ---------------------------------------------------------------*/

void Server::Run( void ) {

long currLen;

max_size_t totLen = 0;

struct UDP_datagram* mBuf_UDP = (struct UDP_datagram*) mBuf;

ReportStruct *reportstruct = NULL;

reportstruct = new ReportStruct;

if ( reportstruct != NULL ) {

reportstruct->packetID = 0;

mSettings->reporthdr = InitReport( mSettings );

do {

// perform read

#ifdef ION

currLen = sbp_recv( mSettings->mSock, mBuf,

mSettings->mBufLen, 0 );

#else

currLen = recv( mSettings->mSock, mBuf, /

mSettings->mBufLen, 0 );

102

#endif

if ( isUDP( mSettings ) ) {

// read the datagram ID and sentTime

// out of the buffer

reportstruct->packetID = ntohl( mBuf_UDP->id );

reportstruct->sentTime.tv_sec =

ntohl( mBuf_UDP->tv_sec );

reportstruct->sentTime.tv_usec =

ntohl( mBuf_UDP->tv_usec);

reportstruct->packetLen = currLen;

gettimeofday( &(reportstruct->packetTime), NULL );

} else {

totLen += currLen;

}

// terminate when datagram begins with negative index

// the datagram ID should be correct, just negated

if ( reportstruct->packetID < 0 ) {

reportstruct->packetID = -reportstruct->packetID;

currLen = -1;

}

if ( isUDP (mSettings))

ReportPacket( mSettings->reporthdr, reportstruct );

} while ( currLen > 0 );

// stop timing

103

gettimeofday( &(reportstruct->packetTime), NULL );

if ( !isUDP (mSettings)) {

reportstruct->packetLen = totLen;

ReportPacket( mSettings->reporthdr, reportstruct );

}

CloseReport( mSettings->reporthdr, reportstruct );

// send a acknowledgement back only

// if we’re NOT receiving multicast

if ( isUDP( mSettings ) && !isMulticast( mSettings ) ) {

// send back an acknowledgement of the

// terminating datagram

write_UDP_AckFIN( );

}

} else {

FAIL(1, "Out of memory! Closing server thread\n",

mSettings);

}

Mutex_Lock( &clients_mutex );

Iperf_delete( &(mSettings->peer), &clients );

Mutex_Unlock( &clients_mutex );

DELETE_PTR( reportstruct );

EndReport( mSettings->reporthdr );

}

104

// end Recv

/* -----------------------------------------------------------------

* Send an AckFIN (a datagram acknowledging a FIN) on the socket,

* then select on the socket for some time. If additional datagrams

* come in, probably our AckFIN was lost and they are re-transmitted

* termination datagrams, so re-transmit our AckFIN.

* ---------------------------------------------------------------*/

void Server::write_UDP_AckFIN( ) {

int rc;

fd_set readSet;

FD_ZERO( &readSet );

struct timeval timeout;

int count = 0;

while ( count < 10 ) {

count++;

UDP_datagram *UDP_Hdr;

server_hdr *hdr;

UDP_Hdr = (UDP_datagram*) mBuf;

105

if ( mSettings->mBufLen > (int) ( sizeof( UDP_datagram )

+ sizeof( server_hdr ))) {

Transfer_Info *stats = GetReport(mSettings->reporthdr);

hdr = (server_hdr*) (UDP_Hdr+1);

hdr->flags = htonl( HEADER_VERSION1 );

hdr->total_len1 =

htonl( (long) (stats->TotalLen >> 32));

hdr->total_len2 =

htonl( (long) (stats->TotalLen & 0xFFFFFFFF));

hdr->stop_sec = htonl( (long) stats->endTime );

hdr->stop_usec =

htonl( (long)((stats->endTime - (long)stats->endTime)

* rMillion));

hdr->error_cnt = htonl( stats->cntError );

hdr->outorder_cnt = htonl( stats->cntOutofOrder );

hdr->datagrams = htonl( stats->cntDatagrams );

hdr->jitter1 = htonl( (long) stats->jitter );

hdr->jitter2 =

htonl( (long) ((stats->jitter - (long)stats->jitter)

* rMillion) );

}

// write data

106

#ifdef ION

sbp_write( mSettings->mSock, mBuf, mSettings->mBufLen );

#else

write( mSettings->mSock, mBuf, mSettings->mBufLen );

#endif

// wait until the socket is readable, or our timeout expires

#ifndef ION

// since sbp doesnt supoort

// select I’m going to wait for timeout to expire

FD_SET( mSettings->mSock, &readSet );

#endif

timeout.tv_sec = 1;

timeout.tv_usec = 0;

#ifdef ION

rc = 0;

#else

rc = select( mSettings->mSock+1, &readSet,

NULL, NULL, &timeout );

#endif

FAIL_errno( rc == SOCKET_ERROR, "select", mSettings );

if ( rc == 0 ) {

// select timed out

return;

} else {

107

// socket ready to read

#ifdef ION

rc = sbp_read( mSettings->mSock, mBuf,

mSettings->mBufLen );

#else

rc = read( mSettings->mSock, mBuf,

mSettings->mBufLen );

#endif

WARN_errno( rc < 0, "read" );

if ( rc <= 0 ) {

// Connection closed or errored

// Stop using it.

return;

}

}

}

fprintf( stderr, warn_ack_failed, mSettings->mSock, count );

}

// end write_UDP_AckFIN

B.3.2 SBP Client

/*---------------------------------------------------------------

* Copyright (c) 1999,2000,2001,2002,2003

* The Board of Trustees of the University of Illinois

* All Rights Reserved.

108

*---------------------------------------------------------------

* Permission is hereby granted, free of charge, to any person

* obtaining a copy of this software (Iperf) and associated

* documentation files (the "Software"), to deal in the Software

* without restriction, including without limitation the

* rights to use, copy, modify, merge, publish, distribute,

* sublicense, and/or sell copies of the Software, and to permit

* persons to whom the Software is furnished to do

* so, subject to the following conditions:

*

*

* Redistributions of source code must retain the above

* copyright notice, this list of conditions and

* the following disclaimers.

*

*

* Redistributions in binary form must reproduce the above

* copyright notice, this list of conditions and the following

* disclaimers in the documentation and/or other materials

* provided with the distribution.

*

*

* Neither the names of the University of Illinois, NCSA,

* nor the names of its contributors may be used to endorse

* or promote products derived from this Software without

* specific prior written permission.

109

*

* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,

* EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES

* OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND

* NONINFRINGEMENT. IN NO EVENT SHALL THE CONTIBUTORS OR COPYRIGHT

* HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,

* WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,

* ARISING FROM, OUT OF OR IN CONNECTION WITH THE

* SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

* ________________________________________________________________

* National Laboratory for Applied Network Research

* National Center for Supercomputing Applications

* University of Illinois at Urbana-Champaign

* http://www.ncsa.uiuc.edu

* ________________________________________________________________

*

* Client.cpp

* by Mark Gates <mgates@nlanr.net>

* ----------------------------------------------------------------

* A client thread initiates a connect to the server and handles

* sending and receiving data, then closes the socket.

* --------------------------------------------------------------*/

#include "headers.h"

#include "Client.hpp"

#include "Thread.h"

110

#include "SocketAddr.h"

#include "PerfSocket.hpp"

#include "Extractor.h"

#include "delay.hpp"

#include "util.h"

#include "Locale.h"

/* ----------------------------------------------------------------

* Store server hostname, optionally local hostname, and socket

* info.

* ------------------------------------------------------------- */

Client::Client( thread_Settings *inSettings ) {

mSettings = inSettings;

mBuf = NULL;

// initialize buffer

mBuf = new char[ mSettings->mBufLen ];

pattern( mBuf, mSettings->mBufLen );

if ( isFileInput( mSettings ) ) {

if ( !isSTDIN( mSettings ) )

Extractor_Initialize( mSettings->mFileName,

mSettings->mBufLen, mSettings );

else

Extractor_InitializeFile( stdin,

mSettings->mBufLen, mSettings );

111

if ( !Extractor_canRead( mSettings ) ) {

unsetFileInput( mSettings );

}

}

// connect

Connect( );

if ( isReport( inSettings ) ) {

ReportSettings( inSettings );

if ( mSettings->multihdr && isMultipleReport(inSettings)){

#ifndef ION

mSettings->multihdr->report->connection.peer =

mSettings->peer;

mSettings->multihdr->report->connection.size_peer =

mSettings->size_peer;

mSettings->multihdr->report->connection.local =

mSettings->local;

SockAddr_setPortAny

( &mSettings->multihdr->report->connection.local );

mSettings->multihdr->report->connection.size_local =

mSettings->size_local;

#endif

}

}

112

} // end Client

/* ----------------------------------------------------------------

* Delete memory (hostname strings).

* ------------------------------------------------------------- */

Client::˜Client() {

if ( mSettings->mSock != INVALID_SOCKET ) {

#ifdef ION

sbp_close( mSettings->mSock );

#else

int rc = close( mSettings->mSock );

WARN_errno( rc == SOCKET_ERROR, "close" );

#endif

mSettings->mSock = INVALID_SOCKET;

}

DELETE_ARRAY( mBuf );

} // end ˜Client

const double kSecs_to_usecs = 1e6;

const int kBytes_to_Bits = 8;

void Client::RunTCP( void ) {

long currLen = 0;

struct itimerval it;

113

max_size_t totLen = 0;

int err;

char* readAt = mBuf;

// Indicates if the stream is readable

bool canRead = true, mMode_Time = isModeTime( mSettings );

ReportStruct *reportstruct = NULL;

// InitReport handles Barrier for multiple Streams

mSettings->reporthdr = InitReport( mSettings );

reportstruct = new ReportStruct;

reportstruct->packetID = 0;

lastPacketTime.setnow();

if ( mMode_Time ) {

memset (&it, 0, sizeof (it));

it.it_value.tv_sec = (int) (mSettings->mAmount / 100.0);

it.it_value.tv_usec = (int) 10000 * (mSettings->mAmount -

it.it_value.tv_sec * 100.0);

err = setitimer( ITIMER_REAL, &it, NULL );

if ( err != 0 ) {

perror("setitimer");

exit(1);

114

}

}

do {

// Read the next data block from

// the file if it’s file input

if ( isFileInput( mSettings ) ) {

Extractor_getNextDataBlock( readAt, mSettings );

canRead = Extractor_canRead( mSettings ) != 0;

} else

canRead = true;

// perform write

currLen = write( mSettings->mSock,

mBuf, mSettings->mBufLen );

if ( currLen < 0 ) {

WARN_errno( currLen < 0, "write2" );

break;

}

totLen += currLen;

if(mSettings->mInterval > 0) {

gettimeofday( &(reportstruct->packetTime), NULL );

reportstruct->packetLen = currLen;

ReportPacket( mSettings->reporthdr, reportstruct );

}

115

if ( !mMode_Time ) {

mSettings->mAmount -= currLen;

}

} while ( ! (sInterupted ||

(!mMode_Time && 0 >= mSettings->mAmount))

&& canRead );

// stop timing

gettimeofday( &(reportstruct->packetTime), NULL );

// if we’re not doing interval reporting, report the

// entire transfer as one big packet

if(0.0 == mSettings->mInterval) {

reportstruct->packetLen = totLen;

ReportPacket( mSettings->reporthdr, reportstruct );

}

CloseReport( mSettings->reporthdr, reportstruct );

DELETE_PTR( reportstruct );

EndReport( mSettings->reporthdr );

}

/* ----------------------------------------------------------------

* Send data using the connected UDP/TCP socket,

* until a termination flag is reached.

116

* Does not close the socket.

* ------------------------------------------------------------- */

void Client::Run( void ) {

struct UDP_datagram* mBuf_UDP = (struct UDP_datagram*) mBuf;

long currLen = 0;

int delay_target = 0;

int delay = 0;

int adjust = 0;

char* readAt = mBuf;

#if HAVE_THREAD

if ( !isUDP( mSettings ) ) {

RunTCP();

return;

}

#endif

// Indicates if the stream is readable

bool canRead = true, mMode_Time = isModeTime( mSettings );

// setup termination variables

if ( mMode_Time ) {

mEndTime.setnow();

117

mEndTime.add( mSettings->mAmount / 100.0 );

}

if ( isUDP( mSettings ) ) {

// Due to the UDP timestamps etc, included

// reduce the read size by an amount

// equal to the header size

// compute delay for bandwidth restriction,

// constrained to [0,1] seconds

delay_target = (int) ( mSettings->mBufLen *

((kSecs_to_usecs * kBytes_to_Bits)/ mSettings->mUDPRate));

if ( delay_target < 0 ||

delay_target > (int) 1 * kSecs_to_usecs ) {

fprintf( stderr, warn_delay_large, delay_target /

kSecs_to_usecs );

delay_target = (int) kSecs_to_usecs * 1;

}

if ( isFileInput( mSettings ) ) {

if ( isCompat( mSettings ) ) {

Extractor_reduceReadSize(

sizeof(struct UDP_datagram), mSettings );

readAt += sizeof(struct UDP_datagram);

} else {

Extractor_reduceReadSize(

118

sizeof(struct UDP_datagram) +

sizeof(struct client_hdr), mSettings );

readAt += sizeof(struct UDP_datagram) +

sizeof(struct client_hdr);

}

}

}

ReportStruct *reportstruct = NULL;

// InitReport handles Barrier for multiple Streams

mSettings->reporthdr = InitReport( mSettings );

reportstruct = new ReportStruct;

reportstruct->packetID = 0;

lastPacketTime.setnow();

// Flush out the read queue, there might be some server reports

// from previous runs

#ifdef ION

while(sbp_recv(mSettings->mSock, mBuf, mSettings->mBufLen,

MSG_DONTWAIT) > 0);

#endif

do {

// Test case: drop 17 packets and send 2 out-of-order:

119

// sequence 51, 52, 70, 53, 54, 71, 72

//switch( datagramID ) {

// case 53: datagramID = 70; break;

// case 71: datagramID = 53; break;

// case 55: datagramID = 71; break;

// default: break;

//}

gettimeofday( &(reportstruct->packetTime), NULL );

memset(mBuf_UDP, sizeof(UDP_datagram), 0);

if ( isUDP( mSettings ) ) {

// store datagram ID into buffer

mBuf_UDP->id =

htonl( (reportstruct->packetID)++ );

mBuf_UDP->tv_sec =

htonl( reportstruct->packetTime.tv_sec );

mBuf_UDP->tv_usec =

htonl( reportstruct->packetTime.tv_usec );

// delay between writes

// make an adjustment for how long the last loop

// iteration took

// TODO this doesn’t work well in certain cases,

// like 2 parallel streams

adjust = delay_target +

lastPacketTime.subUsec( reportstruct->packetTime );

120

lastPacketTime.set( reportstruct->packetTime.tv_sec,

reportstruct->packetTime.tv_usec );

if ( adjust > 0 || delay > 0 ) {

delay += adjust;

}

}

// Read the next data block from

// the file if it’s file input

if ( isFileInput( mSettings ) ) {

Extractor_getNextDataBlock( readAt, mSettings );

canRead = Extractor_canRead( mSettings ) != 0;

} else

canRead = true;

// perform write

#ifdef ION

currLen = sbp_write( mSettings->mSock,

mBuf,mSettings->mBufLen);

#else

currLen = write( mSettings->mSock,

mBuf, mSettings->mBufLen );

#endif

if ( currLen < 0 && errno != ENOBUFS ) {

WARN_errno( currLen < 0, "write2" );

121

break;

}

// report packets

reportstruct->packetLen = currLen;

ReportPacket( mSettings->reporthdr, reportstruct );

if ( delay > 0 ) {

delay_loop( delay );

}

if ( !mMode_Time ) {

mSettings->mAmount -= currLen;

}

} while ( ! (sInterupted ||

(mMode_Time &&

mEndTime.before( reportstruct->packetTime )) ||

(!mMode_Time && 0 >= (int)mSettings->mAmount))

&& canRead );

if ( isUDP( mSettings ) ) {

// send a final terminating datagram

// Don’t count in the mTotalLen. The server counts

122

// this one, but didn’t count our first datagram, so we’re

// even now. The negative datagram ID signifies

// termination to the server.

// store datagram ID into buffer

//memset(mBuf, 0, mSettings->mBufLen);

mBuf_UDP->id = htonl( -(reportstruct->packetID) );

mBuf_UDP->tv_sec =

htonl( reportstruct->packetTime.tv_sec );

mBuf_UDP->tv_usec =

htonl( reportstruct->packetTime.tv_usec );

}

// stop timing

gettimeofday( &(reportstruct->packetTime), NULL );

//Changes reportstruct->packetID!!

CloseReport( mSettings->reporthdr, reportstruct );

if ( isUDP( mSettings ) ) {

if ( isMulticast( mSettings ) ) {

#ifdef ION

sbp_write( mSettings->mSock,

mBuf, mSettings->mBufLen );

#else

write( mSettings->mSock, mBuf, mSettings->mBufLen );

#endif

} else {

123

write_UDP_FIN( );

}

}

DELETE_PTR( reportstruct );

EndReport( mSettings->reporthdr );

}

// end Run

void Client::InitiateServer() {

if ( !isCompat( mSettings ) ) {

int currLen;

client_hdr* temp_hdr;

if ( isUDP( mSettings ) ) {

UDP_datagram *UDPhdr = (UDP_datagram *)mBuf;

temp_hdr = (client_hdr*)(UDPhdr + 1);

} else {

temp_hdr = (client_hdr*)mBuf;

}

Settings_GenerateClientHdr( mSettings, temp_hdr );

if ( !isUDP( mSettings ) ) {

currLen = send( mSettings->mSock, mBuf,

sizeof(client_hdr), 0 );

if ( currLen < 0 ) {

WARN_errno( currLen < 0, "write1" );

}

124

}

}

}

/* ----------------------------------------------------------------

* Setup a socket connected to a server.

* If inLocalhost is not null, bind to that address, specifying

* which outgoing interface to use.

* ------------------------------------------------------------- */

void Client::Connect( ) {

int rc;

SockAddr_remoteAddr( mSettings );

SBP_Init();

assert( mSettings->inHostname != NULL );

// create an internet socket

#ifdef ION

int type = SOCK_BUNDLE;

#else

int type = ( isUDP( mSettings ) ? SOCK_DGRAM : SOCK_STREAM);

#endif

#ifdef ION

int domain = AF_DTN;

125

#else

int domain = (SockAddr_isIPv6( &mSettings->peer ) ?

#ifdef HAVE_IPV6

AF_INET6

#else

AF_INET

#endif

: AF_INET);

#endif

#ifdef ION

mSettings->mSock = sbp_socket( domain, type, 0 );

#else

mSettings->mSock = socket( domain, type, 0 );

#endif

WARN_errno( mSettings->mSock == INVALID_SOCKET, "socket" );

#ifdef ION

int val = 1; //wait 1 seconds before giving up

int cust = 1;

sbp_setsockopt(mSettings->mSock, SOL_SOCKET, DTNOPT_BLOCKING,

&val, sizeof(val));

if( isCustody( mSettings ))

{

sbp_setsockopt(mSettings->mSock, SOL_SOCKET,

126

DTNOPT_CUSTODY, &cust, sizeof(cust));

}

#else

SetSocketOptions( mSettings );

#endif

SockAddr_localAddr( mSettings );

#ifdef ION

if (mSettings->mLocalhost != NULL) {

rc = sbp_bind(mSettings->mSock, &mSettings->local,

mSettings->size_local);

WARN_errno( rc == SOCKET_ERROR, "bind" );

}

rc = sbp_connect( mSettings->mSock, &mSettings->peer,

mSettings->size_peer);

WARN_errno( rc == SOCKET_ERROR, "connect" );

mSettings->size_peer = sizeof(sbp_sockaddr);

strcpy(mSettings->peer.uri, mSettings->mHost);

#else

if ( mSettings->mLocalhost != NULL ) {

// bind socket to local address

rc = bind( mSettings->mSock, (sockaddr*) &mSettings->local,

SockAddr_get_sizeof_sockaddr

127

( &mSettings->local ) );

WARN_errno( rc == SOCKET_ERROR, "bind" );

}

// connect socket

rc = connect( mSettings->mSock, (sockaddr*) &mSettings->peer,

SockAddr_get_sizeof_sockaddr( &mSettings->peer));

WARN_errno( rc == SOCKET_ERROR, "connect" );

getsockname( mSettings->mSock, (sockaddr*) &mSettings->local,

&mSettings->size_local );

getpeername( mSettings->mSock, (sockaddr*) &mSettings->peer,

&mSettings->size_peer );

#endif

} // end Connect

/* ----------------------------------------------------------------

* Send a datagram on the socket. The datagram’s contents should

* signify a FIN to the application. Keep re-transmitting until an

* acknowledgement datagram is received.

* --------------------------------------------------------------*/

void Client::write_UDP_FIN( ) {

int rc;

fd_set readSet;

struct timeval timeout;

server_hdr *hdr;

128

char *udp_fin;

int count = 0;

// copy udp_fin from mBuf, before it gets overwritten

udp_fin = (char*) malloc (mSettings->mBufLen);

memcpy(udp_fin, mBuf, mSettings->mBufLen);

while ( count < 10 ) {

count++;

// write data

#ifdef ION

// The first 5 FIN’s are sent 1 second apart

// but the last 5 are sent ater waiting for

// mTTL seconds(set by timeout)

if( count == 5)

{

// if mTTL is set by the user

if(mSettings->mTTL > 0){

sbp_setsockopt(mSettings->mSock,

SOL_SOCKET, DTNOPT_BLOCKING,

&mSettings->mTTL, sizeof(mSettings->mTTL));

}

else

{

129

int default_wait = 2;

sbp_setsockopt(mSettings->mSock,

SOL_SOCKET, DTNOPT_BLOCKING, &default_wait,

sizeof(default_wait));

}

}

sbp_write( mSettings->mSock, udp_fin, mSettings->mBufLen);

#else

write( mSettings->mSock, mBuf, mSettings->mBufLen );

#endif

// wait until the socket is readable,

// or our timeout expires

FD_ZERO( &readSet );

#ifndef ION

// Since select is not implemented

// in sbp will wait for timeout

FD_SET( mSettings->mSock, &readSet );

#endif

timeout.tv_sec = 0;

timeout.tv_usec = 250000; // quarter second, 250 ms

#ifdef ION

rc = 1;

#else

130

rc = select( mSettings->mSock+1, &readSet, NULL, NULL,

&timeout );

#endif

FAIL_errno( rc == SOCKET_ERROR, "select", mSettings );

if ( rc == 0 ) {

// select timed out

continue;

} else {

// socket ready to read

#ifdef ION

rc = sbp_read( mSettings->mSock,

mBuf, mSettings->mBufLen);

#else

rc = read( mSettings->mSock,

mBuf, mSettings->mBufLen );

WARN_errno( rc < 0, "read" );

#endif

if ( rc < 0 ) {

#ifdef ION

continue;

#else

break;

#endif

} else if ( rc >= (int) (sizeof(UDP_datagram) +

sizeof(server_hdr)) ) {

131

#ifdef ION

hdr = (server_hdr*) ((UDP_datagram*)mBuf +1);

if( ntohl(hdr->packetID) ==

-(ntohl( ((UDP_datagram*)udp_fin)->id))){

#endif

ReportServerUDP( mSettings,

(server_hdr*) ((UDP_datagram*)mBuf + 1) );

#ifdef ION

}

else{

continue;

}

#endif

}

free(udp_fin);

return;

}

}

free(udp_fin);

fprintf( stderr, warn_no_ack, mSettings->mSock, count );

}

// end write_UDP_FIN