Embed Size (px)
Citation preview
A distributed delay tolerant data transfer protocol forISTSAT-1
Pedro Miguel Pereira Eugénio Andrea Gameiro
Thesis to obtain the Master of Science Degree in
Bologna Master Degree in Telecommunications andInformatics Engineering
Supervisors: Prof. Rui Manuel Rodrigues Rocha
Dr. Gonçalo Lopes
Examination Committee
Chairperson: Prof. Ricardo Jorge Fernandes Chaves
Supervisor: Prof. Rui Manuel Rodrigues Rocha
Member of the Committee: Prof. Fernando Henrique Côrte-Real Mira da Silva
December 2018
Resumo
Um cubesat é um pequeno satélite artificial, de baixo custo, criado para efeitos de investigação cientí-
fica espacial, que se tornou muito popular na comunidade científica. O ISTSAT-1 é o primeiro cubesat
a ser desenvolvido no Instituto Superior Técnico (IST), com um tamanho de 10x10x10cm, respeitante a
especificação cubesat 1U.
As comunicações espaciais oferecem desafios diferentes daqueles encontrados em Terra. As ligações
são tipicamente de baixo débito, com grandes atrasos e as disrupções podem ser frequentes. Tudo isto
motivou a criação de protocolos Delay Tolerant (DT), uma arquitectura desenhada para lidar com os
problemas característicos de ambientes disruptivos. No entanto, mesmo com protocolos Delay Tolerant
(DT), a transmissão de ficheiros de elevada dimensão pode ser difícil. Isto é especialmente verdade no
caso dos cubesats, que devido às baixas órbitas em que são colocados, sofrem de longos períodos de
disrupção nas suas ligações com as estações de rádio em Terra.
Com este projecto é proposta uma solução mais adequada. A criação de um protocolo Delay Toler-
ant (DT) distribuído, capaz de realizar as tarefas já presentes nos restantes protocolos Delay Tolerant
(DT), mas onde uma única transmissão pode ser distribuída por diferentes dispositivos (como satélites
e estações de rádio em terra), aumentando assim o número de ligações, reduzindo os períodos de
disrupção e melhorando assim o desempenho das transmissões espaciais.
Palavras chave: IST, nanosat, LEO, ISTSAT-1, distribuído, DT
ii
Abstract
A cubesat is a very small, low cost, artificial satellite designed for space research purposes, very popu-
lar in the academic community. The ISTSAT-1 is the first cubesat being developed in Instituto Superior
Técnico (IST), with a size of a 10x10x10cm cube respecting the 1U cubesat standard.
In space communications one finds different challenges from the ones on Earth. Links are typically un-
stable, of low debit, of high delay and disruptions can be frequent. This motivated the creation of Delay
Tolerant (DT) protocols, an architecture designed to deal with the characteristic problems of disruptive
environments. However, even with DT, the transmission of big data files can be difficult. This is specially
true in the case of cubesats. Due to their Low Earth Orbit (LEO) deployment, cubesats suffer from very
long disruptions periods with the ground-stations on Earth.
With this project we intend to create an enhanced solution. A distributed DT protocol capable of perform-
ing the normal DT tasks, but in a manner where a single transmission can be distributed over different
devices (like ground-stations or satellites), thus increasing the number of links, reducing the disruptions
time periods and increasing the much needed performance of space transmissions.
Keywords: IST, nanosat, LEO, ISTSAT-1, distributed, DT
iii
iv
Contents
Resumo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii
Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
Glossary xi
Acronyms xv
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Objectives and challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Research contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Document structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 State of the art 5
2.1 DTN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Saratoga . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3 SCPS-TP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.4 T-CSP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.5 Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3 Architecture 15
3.1 Application scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.1.1 One-to-one transmissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.1.2 Multi-to-one transmissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.1.3 One-to-multi transmissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.1.4 Distributed transmissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.2 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.3 Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.3.1 Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.3.2 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
v
3.4 Application Scenarios operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.4.1 One-to-one transmissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.4.2 Multi-to-one transmissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.4.3 One-to-multi transmissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.4.4 Distributed transmissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.4.5 POCKET++ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4 Implementation 29
4.1 Software Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.1.1 Multi platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.1.2 Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.1.3 OMNET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.1.4 Multi Network stack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.1.5 Data structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.2 Programming language selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.3 Build tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.4 Code Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.4.1 Static . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.4.2 Dynamic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.5 Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.5.1 Software reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.6 Source Code Version Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.7 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.8 Development Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5 Experimental evaluation 41
5.1 One to one . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.1.1 One to one - Delay tolerant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.2 Multi to one . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.3 One to multi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.4 Distributed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
5.5 Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6 Conclusion and future work 51
Bibliography 53
vi
List of Tables
2.1 Comparison of the existing solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
5.1 One to one transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
vii
viii
List of Figures
1.1 ADS-B System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 ISTSAT-1 Distributed transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1 Store and Forward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 DTN stack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3 DTN transmission comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.4 Mars network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.5 Saratoga Message Sequence Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.6 Saratoga stack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.7 Saratoga STATUS message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.8 T-CSP segment over CSP packet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.9 T-CSP transmission example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.1 One to one transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.2 Multi to one transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.3 Multicast transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.4 Distributed transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.5 DDTP Message types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.6 DDTP sender process state machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.7 DDTP receiver process state machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.8 MSC One-to-one transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.9 MSC Multi-to-one transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.10 One-to-multi transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.11 MCC Distributed Transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.12 Distributed transmission - cubesat cluster example . . . . . . . . . . . . . . . . . . . . . . 28
4.1 DDTP modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.2 DDTP software architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.3 DDTP software platform architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.4 DDTP - Network Abstraction Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.5 DDTP Network stack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.6 DDTP Multi stack transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
ix
5.1 Testing simulation topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.2 DDTP vs Saratoga - One to one . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.3 DDTP DT transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.4 Saratoga DT transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.5 Multi to one - DDTP vs Saratoga . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.6 One to multi topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.7 DDTP - One to multi transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
5.8 DDTP distributed transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
5.9 Pocket++ transmission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
x
Glossary
ADS-B Automatic dependent surveillance-broadcast is a surveillance technology in which an aircraft de-
termines its position via satellite navigation and periodically broadcasts it, enabling it to be tracked.
AX25 A data link layer protocol derived from the X.25 protocol suite and designed for use by amateur
radio operators.
C A general-purpose, imperative computer programming language.
C++ introduces object-oriented programming (OOP) features the C language.
Cubesat A type of miniaturized, low cost satellite for space research of popular use in the academic
community.
Doxygen Doxygen is a documentation generator, a tool for writing software reference documentation.
The documentation is written within code, and is thus relatively easy to keep up to date.
ESA The European Space Agency is an intergovernmental organization of 22 member states dedicated
to the exploration of space.
Ethernet A data link layer protocol, widely used in conjunction with the Transmission Control Protocol
(TCP)/Internet Protocol (IP) stack.
FreeRTOS FreeRTOS is a real-time operating system kernel for embedded devices that has been
ported to 35 microcontrollers.
FYS ESA’s Fly Your Satellite! (FYS) programme is a recurring, hands-on programme designed and
managed by the ESA Education Office in close collaboration with European Universities, with the
objective to complement academic education and inspire, engage, and better prepare university
students for a more effective introduction to their future professions in the space sector.
ISTSAT-1 ISTSAT-1 is the first cubesat being developed in Portugal, by IST-UL, with a 1U size (103 cm
cube) by students, professors and radio-amateurs. Its mission is first and foremost an educational
one, joining together multiple students from different expertise and engineering programmes to
foster their enthusiasm for space science and complementing their education with a challenging
hands-on project.
xi
Java Java is a general-purpose computer-programming language that is concurrent, class-based, object-
oriented, and specifically designed to have as few implementation dependencies as possible.
Libcsp CubeSat Space Protocol (CSP) is a small network-layer delivery protocol designed for Cube-
Sats. The idea was developed by a group of students from Aalborg University in 2008, and further
developed for the AAUSAT3 CubeSat mission that was launched in 2013.
MCC A mission control center is a facility that manages a spacecraft mission and operations, including
both personnel and equipment. It is part of the ground segment of spacecraft operations.
Netkit Netkit is an environment for setting up and performing networking experiments at low cost. It
allows to "create" several virtual network devices (full-fledged routers, switches, computers, etc.)
that can be easily interconnected in order to form a network. Netkit exploits open source software
(mostly licensed under GPL) and is heavily based on the User Mode Linux (UML) variant of the
Linux kernel.
OMNET OMNeT++ is a modular, component-based C++ simulation library and framework, primarily for
building network simulators.
Peer2peer Peer-to-peer (P2P) computing or networking is a distributed application architecture that
partitions tasks or workloads between peers. Peers are equally privileged, equipotent participants
in the application. They are said to form a peer-to-peer network of nodes.
Pocket++ A simple, deterministic and very efficient compression algorithm developed by ESA for space-
crafts with limited computacional resources. It can work on the real-time data stream and therefore
one benefits from stored data compression for free, as well as the multiplication factor during play-
back.
Python Python is an interpreted high-level programming language for general-purpose programming.
Created by Guido van Rossum and first released in 1991, Python has a design philosophy that
emphasizes code readability.
Ruby A dynamic, open source, reflective, object-oriented, general-purpose programming language with
a focus on simplicity and productivity.
Saratoga Saratoga is a very scalable, peer-to-peer reliable transport protocol, capable of transferring
very large files guaranteeing error-free data delivery.
Sattelite cluster Sattelite cluster is the concept that multiple satellites can work together in a group to
accomplish the objective of one larger, usually more expensive, satellite.
SNACK Selective Negative Acknowledge - The receiver explicitly lists the segments in a stream that
were not received (even if non-continuous).
xii
UDP-lite Lightweight User Datagram Protocol (UDP-Lite) is a connectionless protocol similar to User
Datagram Protocol (UDP) but that does not check the integrity of the payload.
Unicast Unicast is communication between a single sender and a single receiver over a network. The
term exists in contradistinction to multicast, communication between a single sender and multiple
receivers.
Valgrind Valgrind is a programming tool for memory debugging, memory leak detection, and profiling.
xiii
xiv
Acronyms
ACK Acknowledge.
CCSDS Consultative Committe for Space Data Systems.
CPU Center Processing Unit.
CSP Cubesat Space Protocol.
DDTP Distributed Delay Tolerant Protocol.
DICE Dynamic Ionosphere Cubesat Experiment.
DMC Disaster Monitoring Constellation.
DT Delay/Disruptive Tolerant.
DTN Delay/Disruptive Tolerant Networking.
EID Endpoint Identifier.
FTP File Transfer Protocol.
FYS Fly Your Satellite.
GDB GNU Debugger.
GS Ground Station.
IDE Integrated Development Environment.
IP Internet Protocol.
IPN Interplanetary Internet Project.
IPSec IP Security.
ISA Instruction Set Architecture.
ISL Inter Satellite Links.
xv
IST Instituto Superior Técnico - University of Lisbon.
ISWCS International Symposium on Wireless Communication Systems.
ITU WRC-15 International Telecommunication Union’s World Radio communication Conference.
KISS Keep It Simple and Stupid.
LEO Low Earth Orbit.
LOS Line Of Sight.
MCC Mission Control Center.
MSC Message Sequence Chart.
MTU Maximum Transmission Unit.
NACK Negative Acknowledge.
NASA National Aeronautics and Space Administration.
NPAL Network Protocol Abstration Layer.
OS Operating System.
OSAL Operating System Abstration Layer.
OSI Open Systems Interconnection.
RDP Reliable Datagram Protocol.
SCPS Space Communications Protocol Specifications.
SCPS-FP SCPS File transfer Protocol.
SCPS-NP SCPS Network Protocol.
SCPS-SP SCPS Security Protocol.
SCPS-TP SCPS Transfer Protocol.
SNACK Selective Negative Acknowledge.
SNR Signal to Noise Ratio.
SSTL Surrey Satellite Technology Ltd.
T-CSP Tolerant CSP.
TCP Transmission Control Protocol.
xvi
UDP User Datagram Protocol.
URI Uniform Resource Identifiers.
WSN Wireless Sensor Network.
xvii
Chapter 1
Introduction
Space research has traditionally been limited to organizations who can afford the large investments as-
sociated with it. This presents a major limitation in technological development.
A popular new trend, specially in universities and the academic community, is the development of minia-
turized satellites known as cubesats[1]. These satellites offer a major opportunity for organizations with
limited resources to perform space research[2]. This is not only due to their small sizes, but since they
can be built with common electronic components they are of low cost. Its design was proposed in 1999
by professors Jordi Puig-Suari of California Polytechnic State University and Bob Twiggs of Stanford
University.
Standard cubesats usually have a volume of 1 liter (103 cm cube), weight 1kg and are called 1U(one
unit) cubesats. There are also some bigger models, scaled in the vertical axis, 2U(20x10x10cm) and
3U(30x10x10cm) and even as big as 6U and 12U.
ISTSAT-1[3] is the first cubesat being developed in Portugal, by Instituto Superior Técnico - University of
Lisbon (IST), with a 1U size (103 cm cube) by students, professors and radio-amateurs. The ISTSAT-1
mission is first and foremost an educational one, joining together multiple students from different exper-
tise and engineering programmes to foster their enthusiasm for space science and complementing their
education with a challenging hands-on project.
In 2014, under unexplained circumstances, the Malaysia Airlines flight 370 plane disappeared in the mid-
dle of the ocean. This event triggered a new discussion about safety in the airline industry and a growing
interest in tracking airplanes in remote areas. In November 2015, the International Telecommunication
Union’s World Radio communication Conference (ITU WRC-15) allocated a new frequency band to be
used by the ADS-B[4] system in earth-space direction1. This enabled the possibility of tracking aircraft
positions from space, at a global scale. Today, this signal is only monitored through ground-based sta-
tions, with no coverage in oceanic routes and remote areas. The ADS-B system is a security feature that
most planes today use, and all of them will be obliged to use by 2020. It periodically (each 0.5 seconds)
emits a message with information about the plane, it’s location, identification and status. ISTSAT-1[5] will
be used for carrying a feasibility study of the use of cubesats for receiving ADS-B signals from traveling
1https://www.itu.int/net/pressoffice/press_releases/2015/51.aspx
1
airplanes, as illustrated in figure 1.1. The ISTSAT-1 spacecraft is also being developed in the context of
an ESA educational programme, the Fly Your Satellite (FYS), along with 5 other european teams devel-
oping their own cubesat2. In this programme the ISTSAT-1 development will be assisted by experienced
engineers and professionals from ESA that have developed, launched and operated several spacecrafts
in their carer. This is a major opportunity that as drastically impacted this project. The FYS programme
as also offered the ISTSAT-1 a launch opportunity, currently schedule for 2020.
To address the problems associated with space communications (characterized by long distances, high
delay environments, frequent disruptions or disconnections), a new communication paradigm emerged,
the Delay/Disruptive Tolerant Networking (DTN). The DTN specification defines an hop-to-hop architec-
ture where link disruption is acceptable and transmissions will pause and resume as necessary. This
paradigm is already being used in the National Aeronautics and Space Administration (NASA) mars
rover mission3.
Figure 1.1: ADS-B System
1.1 Motivation
Cubesats are normally deployed in Low Earth Orbit (LEO), which is defined as 300-1500km above the
earth surface, with an orbital period of approximately 90 minutes at a speed of 7.8 km/s. However, a
big consequence of this, is that a particular ground-station only has a very small time-window (15-20
minutes) of line-of-sight to communicate with the satellite. Offering even more obstacles, space-link
communications are very unstable, error prone, and of low debit. The challenges in space-link commu-
nications are not traffic congestion (like on earth), but long propagation delays and high bit-error rates.
The current approach to this problem is the use of DTN [6][7] protocols. These (in contrast to common,
terrestrial protocols) are designed to tolerate large periods of delay (or disrupted links) and transmit at
high bitrates when the link is available. This approach allows transmissions to be paused when the
communication space link is unavailable, and to resume when a line-of-sight is once again available.
The purpose of this project is to tackle this difficulties in a new, more powerful way. To create a distributed
2https://www.esa.int/Education/CubeSats_-_Fly_Your_Satellite/Six_new_CubeSat_missions_selected_for_next_
cycle_of_Fly_Your_Satellite3http://mars.nasa.gov/mer/home/
2
protocol capable of expanding a single transmission session over several links and hosts (Ground Sta-
tions (GSs) in this particular case).
This is illustrated in figure 1.2, where the satellite, after starting a transmission with a given GS, leaving
its line-of-sight and consequently disconnecting the link, does not need to perform a full orbit to resume
the transmission. It can just continue the transmission with the next GS available in its orbit. This can re-
sult in a very significant performance enhancement, both due to the very limited available time-windows
and the typical characteristics of space communications, like high delays and low throughputs.
Another interesting use case for such a solution is the transmission of data through, and within, cubesat
clusters[8]. In this types of spacecraft constellations, such as the humsat project 4, transmissions can
be tricky since every single node is moving on its own orbit, and also relative to earth’s GSs. Due to this,
a distributed approach like the one proposed by this project can be very useful. With enough cubesats
in a cluster, and taking advantage of a distributed solution, it could even be possible to create a real-time
service for tracking ADS-B signals.
Figure 1.2: ISTSAT-1 Distributed transmission
1.2 Objectives and challenges
With this project we intend to create a Delay/Disruptive Tolerant (DT) protocol, prepared to deal with
space communications related challenges that are already being dealt with in today existing DT proto-
cols, but one that is capable of transmitting in a fully distributed manner. This means that, not only this
protocol will be able to deal with disruptive prone environments, but it will also be capable of performing
a single transmission through different nodes.
4https://www.humsat.org
3
In a typical cubesat communication, a DT transmission occurs between a ground station and a satellite
(optional with other hops). This is not enough, specially when transmitting big data files. With the typi-
cal DT protocols, this is handled by resuming the transmission (where it was left of) when the cubesat
completes an orbit and the line-of-sight is restored. Note, however, the passes over the GS receiving (or
transmitting) the big data are not always azimuth-favorable in consecutive orbits due to its quasi-polar
characteristics[9]. Therefore, it might take a while to complete the entire transmission successfully for a
given GS.
With a distributed protocol the transmission could be performed over different ground-stations, located
in different geographical locations.
The proposed protocol should be bidirectional (i.e. capable of transmitting both in the uplink and down-
link), capable of prioritization when dealing with several simultaneous transmissions, guarantee the cor-
rect integrity of the received data and retransmit the broken segments. These are all common features
found in transport protocols.
This protocol is intended to be used over Open Systems Interconnection (OSI) layer 4 Reliable Datagram
Protocol (RDP) protocol that in its turn will use Cubesat Space Protocol (CSP). CSP is a very popular,
well tested, protocol for cubesat systems. It was designed for the purpose of space transmission and
it is able to address different, possibly independent, modules of a given satellite. It works in both OSI
layers 2 and 3 and its original implementation source code is freely available for download and changes.
1.3 Research contributions
During the development of this project, a scientific paper[10] was also written. It features the designed
solution and some preliminary results. It was presented and published at the 15th International Sympo-
sium on Wireless Communication Systems (ISWCS) conference in 2018.
1.4 Document structure
This report is organized in six chapters. The present one, Introduction, explains the proposed project,
its objectives, motivations and challenges. The second one, State of the art, presents a comparison of
the existing technologies, showing the need for this project and analyzing the good characteristics of the
existing solutions, offering an useful insight for the design of this project. The third one, Architecture,
focus one the logical design of developed solution. The fourth one, Implementation will explain in detail
the protocol, taking as reference the design mentioned in the previous chapter. The fifth one, Experi-
mental evaluation, will present practical results from experimental execution of the developed solution.
And finally, the sixth one, Conclusion and future work, sums up the work of the whole document, the
research involved, its conclusions and possible future work.
4
Chapter 2
State of the art
In this chapter we will compare some of the current solutions to the presented problem. These are used
in the space communication link between the radio stations on earth (known as GS), and the orbiting
satellites.
2.1 DTN
DTN [6] is a network architecture designed to address the problems of disruptive environments, like the
ones found in this project related with space communications, (keeping in mind the inefficiency of terres-
trial protocols to address this issues) like high delay environments, long distance communications and
frequent disconnection of links. In this architecture a transmission is not terminated when its physical
link is disconnected. Instead, link disruption is dealt with by pausing and resuming transmissions as
needed, and by the use of multiple hops, in a store-and-forward[11] fashion. This strategy is illustrated
in figure 2.1, where node A starts by transmitting to node B normally, but then it is forced to store the
data, since a link is not available or was disrupted mid-transmission (as illustrated by the dotted line).
When it does become available, the data is forwarded along the network using the same strategy at
every node until it finally reaches its destination.
To implement this architecture an end-to-end message-oriented overlay, called bundle layer [12], is de-
fined over the OSI transport layer (layer 4). This compatibility layer ensures the independence of the
DTN protocol from the rest of the stack, thus allowing a transmission to occur over different network
stacks (like TCP/IP or AX25/CSP)[13]. In figure 2.2 this concept is illustrated, showing how a transmis-
sion can occur over multiple regions, each with potentially different layers bellow the bundle layer.
Devices implementing the bundle layer are called DTN nodes. Each node uses a persistent memory to
store the transmissions being sent. This is done due to the disruption tolerant nature of the architecture.
Since links are expected to disconnect, each node receives transmissions even if no output link is avail-
able for the final receiver, stores the data, and then, when an output link becomes available, retransmits
it towards its final destination. This means that a successful transmission can occur without, at any given
moment, the existence of a complete path (with the use of hops) between the sender and the receiver.
5
D
B
A
Source
C
F
E
Destination
Store
Store
Forward
Forward
Carry
Carry
Disruptive link
Normal link
Figure 2.1: Store and Forward
Transmissions can be sent in a hop-by-hop manner, with each hop node storing persistently the data
when its link is disconnected. Each DTN node is identified by an Endpoint Identifier (EID) which is based
on Uniform Resource Identifiers (URI) [14].
In figure 2.3 a DTN transmission is illustrated and compared with a non DTN transmission over the in-
ternet using the TCP/IP stack. In the top diagram (the non DTN transmission) a link disruption between
any of hops means the termination of the transmission and the tear-down of the TCP connection. On
the bottom diagram (the DTN transmission) a link disruption between any of the hops does not mean
the termination of the transmission. Instead, all the nodes with valid links will continue to transmit. The
ones without an output link will store the received data in their persistent memory, waiting for a chance
to resume the transmission. This allows a transmission to be processed, never aborting or restarting
when there is no connection between the sender and receiver.
The concept of DTN began to be studied by the end of the 1990’s decade as part of the implementation
of the Interplanetary Internet Project (IPN)1.
A real life example of a DTN is the NASA Mars mission2. In this network the Mars Rover[15] (named
curiosity ), illustrated in figure 2.4, transmits data to one of the three orbiting satellites. They, in turn,
store and retransmit this data to Earth when their orbit is favorable (when they have line-of-sight with
Earth). While the Mars Rover could send the messages directly to Earth, this DTN approach enables it
to upload data more efficiently and even when it has no line-of-sight with the Earth ground-stations.
In summary, the DTN architecture addresses many of the problems of space networks that are charac-
terized by an environment of long delays and discontinuous connectivity.
1http://ipnsig.org/2http://ipnsig.org/2013/07/09/curiosity-rover-the-store-forward-architecture/
6
Figure 2.2: DTN stack
Figure 2.3: DTN transmission comparison
7
Figure 2.4: Mars network
2.2 Saratoga
Saratoga[16][17], named after the USS Saratoga3, was first developed at Surrey Satellite Technology
Ltd (SSTL) to download imagery from its remote-sensing Disaster Monitoring Constellation (DMC) satel-
lites.
Saratoga is a very scalable, peer-to-peer reliable transport protocol, capable of transferring very large
files (exa-exabytes - 2128 B) guaranteeing error-free data delivery using a Selective Negative Acknowl-
edge (SNACK) logic.
Saratoga does not use any congestion control, it just sends the data at the highest possible rate in the
link, with no regard for other flows. This is acceptable since space-links are typically dedicated.
It is a protocol based in User Datagram Protocol (UDP) (can also be used with UDP-lite to minimize
checksum computation overhead), designed to send data packets as fast as possible (since it is de-
signed for private, low congestion links there is no need for congestion control) over several hops on
privately owned links. It is also capable of transmitting over links with very long propagation delays since
no forced timeouts are used.
Saratoga uses a 32B DATA message header, another 8B for the UDP header, 20B for the IP header
and another 18B for the Ethernet header, for transmitting a payload of 1422B (assuming a normal sized
ethernet frame of 1500B), offering a payload ratio of ∼95%.
The Saratoga protocol was developed with design goals of being very fast; to use cheap, reliable and
robust internet technologies; provide more speed than TCP can offer; not caring about congestion, be-
cause it is to be used as a single flow per link with no competition; support very big data file sizes;
simultaneous delivery to multiple receivers. Two example implementations of the Saratoga specification
are being developed, but are still incomplete.
The first one is a Perl implementation4, that was written for interoperability testing, to validate the
3https://web.archive.org/web/20080328084822/http://www.chinfo.navy.mil/navpalib/ships/carriers/
histories/cv03-saratoga/cv03-saratoga.html4http://saratoga.cvs.sourceforge.net/viewvc/saratoga/saratoga-v1-perl/
8
Figure 2.5: Saratoga Message Sequence Chart
Saratoga specification. It was not designed with performance in mind, which means that it is not suitable
for a real scenario deployment, only for testing other implementations.
The second one is a C++ implementation5, it is freely available for use and implements most of the
Saratoga version 1 [18] (the latest one) specification. It is still incomplete but, unlike the perl implemen-
tation, it is intended to be suitable for real world scenarios.
Saratoga uses 5 types of messages:
• BEACON
Sent periodically, the beacon message is used by each peer to declare its EID, its availability for
receiving packets, and the max file descriptor (used to indicate the offset of the data segment)
available (16, 32, 64 or 128 bits).
• REQUEST
Used to request a file, directory listing, or a file deletion from a peer (in other word, a get transac-
tion).
• METADATA
Used to start the sending of a file, (in other words, a put transaction) it describes the file, sets the
translation ID, and advertises the used file descriptor.
• DATA
Used to send data. The file descriptor is used to indicate the offset of the data segment. An
Acknowledge (ACK) message can be requested.
5http://sourceforge.net/p/saratoga/saratoga-vallona/ci/default/tree/
9
Figure 2.6: Saratoga stack
Figure 2.7: Saratoga STATUS message
• STATUS
Used to report lists of missing data offsets (known has HOLESTOFILL) and errors. This imple-
ments the SNACK logic of Saratoga. These are requested periodically by the sender (piggyback-
ing a DATA message) asking the receiver to report the correct reception of the segments, and the
missing HOLES as figure 2.7 illustrates. This means that the pace at which the ACK are sent is set
by the sender. The receiver can also send STATUS messages when he "thinks" these are needed
(e.g. start, restart, end of a transmission).
2.3 SCPS-TP
The Consultative Committe for Space Data Systems (CCSDS)[19][20] organization formulates recom-
mendations to address problems found in space systems. One of this projects is the Space Communica-
tions Protocol Specifications (SCPS), a set of extensions to existing protocols designed to address the
typical problems encountered in space communication systems, with a great focus in promoting systems
interoperability and cost reduction, these are divided in 4 categories:
10
• SCPS File transfer Protocol (SCPS-FP) - A set of extensions to File Transfer Protocol (FTP) to
make it more bit efficient and to add advanced features such as record update within a file and
integrity checking on file transfers. This is an optional protocol.
• SCPS Transfer Protocol (SCPS-TP) - SCPS-TP[21] uses a set of TCP options such as Negative
Acknowledge (NACK), SNACK, modified slow start algorithms, modified congestion avoidance
algorithms, and windows scaling. Additionally, SCPS-TP includes sender-side modifications to
enhance the TCP performance in the stressed environments, such as long delays, high bit error
rates, and significant asymmetries. As a result, the SCPS-TP stack is compatible with other recog-
nized TCP implementations. SCPS-TP refers collectively to the protocols that provide full reliability
(using TCP), best-effort reliability (using TCP with some modifications), and minimal reliability ser-
vices (using UDP).
• SCPS Security Protocol (SCPS-SP) - SCPS-SP is an optional security protocol, and is compa-
rable to IP Security (IPSec).
• SCPS Network Protocol (SCPS-NP) - A bit-efficient network protocol that is analogous to IP.
However, it is not compatible with IP. The protocol is designed for use in space systems. It supports
static or dynamic routing, precedence, and multiple routing options. This is an optional protocol.
In respect of efficiency, SCPS-TP uses a 20B header, plus 20B for the IP header, and 18B for the
Ethernet header, summing up to an overhead of 58B for a 1442B payload (assuming a normal Ethernet
frame). This results in a payload ratio of ∼96%. Some cubesat projects, such as the Dynamic Iono-
sphere Cubesat Experiment (DICE) [22] are using the CCSDS standard. The SCPS-TP was designed
to support a best-effort, minimal reliability strategy; to operate efficiently in a wide range of delay, band-
width and errors conditions; operate efficiently in space-based processing environments; support priority
connections; support connectionless multicasting; and support packet oriented applications.
2.4 T-CSP
CSP is a layer 4 protocol designed for space communication environments, specifically designed for
cubesat systems. Tolerant CSP (T-CSP) is an extension of CSP, capable of performing in a DTN man-
ner, and was designed and implemented by João Ferreira during his master’s degree dissertation in
Communication Networks Engineering at Instituto Superior Técnico [3]. In his dissertation the goal was
to send imagery data over an AX25 link, using CSP for both layer 2 and layer 3. Since AX25 has a very
restrictive Maximum Transmission Unit (MTU) limit (256B), a protocol capable of performing fragmenta-
tion and packet recovery was necessary.
T-CSP relies on a SNACK logic to ensure packet recovery. This approach was chosen due to the typical
throughput asymmetry in space-links (The downlink tends to have more throughput). This means that
a packet is only retransmitted if the receiver reports that it’s missing, otherwise its successful arrival is
assumed.
11
In figure 2.8 a T-CSP segment is shown, where 1B is used for file identification, 2B for the segment
number and another 2B for the total number of segments being transmitted. This structure allows for
255 possible different files and 65 535 segments per file, that results in a 15.44MiB of max file size. Since
space-links are characterized by being low debit this can be a reasonable value.
In the case of overhead, since 29B are used for transporting 247B of payload, we are looking at a pay-
load ratio of ∼90%. T-CSP was designed to perform segment fragmentation, retransmit lost packets,
and be compatible with the AX25 stack.
Figure 2.8: T-CSP segment over CSP packet
Figure 2.9 demonstrates a successful T-CSP file transmission.
In the first step, the GS sends a command in a CSP packet (T-CSP is not a appropriate for sending
commands), requesting the transmission of a file.
In step two and three, the requested file is split into multiple segments that will be transmitted in T-CSP
packets. These include their segment offset, since packets may be lost or not arrive in order. This
process continues for until the n packet, at which point the GS software will be able to process the file
as a whole.
Figure 2.9: T-CSP transmission example
12
2.5 Comparison
Table 2.1: Comparison of the existing solutions
Features Saratoga SCPS-TP T-CSP
Available implementation Yes No Yes
Bidirectional Yes Yes Yes
Auto-retransmission Yes Yes Yes
Transmission error detection Yes Yes No
Prioritization No Yes No
Supported file sizes 256 EiB2 (2128 B) Not limited 15.44MiB
Overhead ∼5% ∼4% ∼10%
Congestion Control No Yes No
Compatible with CSP No No Yes
Distributed No No No
From these three protocols we can get a grasp of the type of problems encountered in space com-
munications and the type of solutions chosen. Nevertheless, none of these are appropriate for resolving
the presented problem. T-CSP was designed to work with the CSP protocol, uses a SNACK logic and
retransmits lost segments but does not have error detection mechanisms. SCPS-TP covers a lot of the
necessary requirements, but it is not compatible with the CSP protocol. Saratoga is also capable of a
lot of what it is required, but is missing CSP compatibility. The major reason for the development of a
new protocol, however, is that none of these are capable of transmitting in a distributed manner. This
means that in all this cases, non of them are capable of taking usage of an available link to resume a
transmission if this is not the original link through which the transmission was established.
13
14
Chapter 3
Architecture
With the distributed DT protocol, contrasting the current solutions, a transmission is not paused or can-
celed when a link is disrupted, and instead, resumes over the next available node, only pausing indefi-
nitely if absolutely necessary. This is specially important in LEO satellites like the cubesat systems that,
due to their fast orbit, fly out of the transmitting GS line-of-sight very quickly. This solution removes the
very time consuming need of waiting for a link re-establishment. Possibility requiring the passage of one
or multiple full orbits, depending on the scenario in question. This is specially relevant when dealing
with transmissions of big data files. Section 3.1.1, 3.1.2 and 3.1.4 will describe the solution for these
scenarios.
Another interesting example is the usage of a Cubesat to deliver a firmware update to a Wireless Sensor
Network (WSN) with devices scattered over large distances. Here the simultaneous transmission of the
data payload to all devices within the Cubesat Line Of Sight (LOS) would offer a major advantage over
the typical Unicast transmission. Section 3.1.3 will describe this solution.
The rest of this chapter will be dedicated to explaining the logical design and operations of the proposed
solution.
3.1 Application scenarios
In this section we will describe the four major scenarios the proposed solution must solve, these are
directly connected to the requirements described in section 3.2. The first two, One-to-one (section 3.1.1)
and Multi-to-one (section 3.1.2), are already solved by current solutions but the last two(section 3.1.3
and 3.1.4), however, are not (with the exception of Saratoga protocol that can perform in a on-to-multi
fashion).
3.1.1 One-to-one transmissions
The most simple scenario is a transmission with only one sender and one receiver. This is what most
current solutions do, where the objective is to send or receive payload data from a system with a dis-
ruptive network connection. Figure 3.1 demonstrates an example with a transmission between node A
15
and C through B with the dotted line representing a disruptive link. This can be achieved successfully
by performing the transmission in a store-and-forward fashion, pausing and resuming the transmission
accordingly to the link availability. In the example shown in figure 3.1 node B would store the data blocks
received from node A, when the link with node C is unavailable, and forward them when it becomes
available again. This is what current solutions do, as it will be further explained in section 3.4.1.
C
B
A
Source
Destination
Figure 3.1: One to one transmission
3.1.2 Multi-to-one transmissions
A slightly more complex scenario is one where several data payloads are transmitted at the same time,
through the same intermediary nodes, and to possibly the same destination. This can be achieved by
creating a protocol that allows multiple transmission to execute in parallel. In figure 3.2, an example is
showed where three different connections are originated in nodes A, B and C and terminate in node F.
An example of this would be a group of distant sensor networks placed in remote areas that need to
deliver their data payloads to a collection server, and do so through a satellite link. This would be a multi-
to-one example and would require a protocol capable of performing multiple parallel transmissions.
The specific details of how this scenario is being addressed by the proposed solution are described in
section 3.4.2.
16
A
Source 1
F
E
D
Source 3Source 2
B C
Destination
Figure 3.2: Multi to one transmission
3.1.3 One-to-multi transmissions
A more complex scenario is the transmission of a data payload from one source to multiple destinations.
This can obviously be done by simply transmitting the data N times for N destinations, but this is very
inefficient. This is particularly true in the case of spacecrafts communicating with earth since, due to
their altitude, they are gifted with a big line-of-sight radius that may cover more than one destination
node.
An example of this, that takes advantage of the large communication line of sight of a LEO spacecraft, is
the transmission of a firmware update to N nodes on the terrain that are sufficiently close to each other
to fit in the line-of-sight cone of the spacecraft. If one were to perform a multicast transmission, instead of
executing N unicast transmissions, one could improve very significantly the overall transmission speed.
This idea consist in bringing the multicast concept into DTNs.
In the example shown in figure 3.3 the data blocks travel from node F to D in the same way they would
in figure 3.1 and then are multicasted to node A, B and C. This offers the advantage of allowing three
nodes to receive a payload while only performing one transmission.
The major challenge in this scenario comes from the fact that each of the multicasted nodes (A, B and
C) have links with different qualities due to their distances, meteorological conditions, Signal to Noise
Ratio (SNR) and other characteristics. This means that packets that are successfully received by one
node may be dropped by another. This challenge and the proposed solution will be explained in more
detail in section 3.4.3.
17
A
Source
F
E
D
Destination
3
Destination
2
B C
Destination
1
Figure 3.3: Multicast transmission
3.1.4 Distributed transmissions
The major feature of this project is its distributed nature. The intent is to allow a single transmission to
be performed over several links, several nodes, with no fixed requirement over any single route. This
works in a similar manner to a typical peer-to-peer network.
In figure 3.4, an example is shown with a single transmission being performed from node C to A over
several B nodes. Node C has several available links of disruptive nature and will use whichever is
available at each moment to deliver the payload data blocks along the network. This results in not
having to wait for the re-establishment of a specific link to resume a transmission, simply continuing
sending data blocks over whatever links are available.
The benefit of this feature is greater when there are multiple low quality links, but is however less relevant
when a good quality link is available.
A LEO cubesat with access to multiple GSs with low quality links is a case for which this approach could
be very beneficial. Another example would be a Sattelite cluster where Inter Satellite Links (ISL)[23] are
used to sent fragments of a big file to each satellite capable of transmitting the fragment to earth, thus
increasing the overall transmission speed. This could be used, for example, for transmitting the ADS-B
signal captured in space to earth in real-time.
18
A
Source
C
B
Destination
BB B
Figure 3.4: Distributed transmission
3.2 Requirements
For the development of this project a list of requirements was defined. These are design decisions that
will guide the development of an adequate solution to the presented problem:
• Distributed protocol
The main goal of the project is to create a distributed protocol, capable of maintaining a single
connection over different GSs or satellites, with different links, to transmit as continually as possible
while the satellite line-of-sight time window passes over different geographical spots.
• Bidirectional communication
Communication originated from both the ground-stations to the satellite (uplink), and from the
satellite to the ground-stations (downlink), should be supported.
• Guarantee data integrity
To guarantee data integrity means to be able to detecte if the received data suffered any alteration
while being transmitted. Data corruption is a very common occurrence in radio communications,
specially in such error prone environments has satellite communications.
Data corruption can be detected with the use of a checksum algorithm. Generally this can be
calculated, either at the transport layer level (calculated for each transmitted segment) or, once the
transmission has ended, to the complete receive data, or both.
• Retransmission of broken segment
To detect data corruption is one thing, to deal with it is another. The protocol should, by itself,
detect and retransmit data that was incorrectly received.
19
• Designed to be used with layer 2/3 CSP Protocol
CSP is a popular protocol designed for nanosats. It supports a connection oriented layer 4 protocol,
network layer 3, data layer 2, and physical layer 1. For the purposes of this project’s distributed
DTN protocol, CSP’s layer 2 and 3 implementation will be used.
• Use non-blocking mechanisms
Since it is expected for the links to be disrupted and reconnected frequently, it is important that the
implementation of the protocol does not leave the remaining program in an active wait state.
• The implementation should work in the same Operating Systems (OSs) where CSP works
CSP implementation currently supports Windows, Linux, FreeRTOS, and MacOSX. The main fo-
cus of this project will be on the Linux and FreeRTOS.
• The development should be performed according to ECSS-E-ST-40C[24]
This standard developed by ESA covers all aspects of space software engineering including re-
quirements definition, design, development, validation and maintenance. By following this stan-
dard, one can produce better quality software, more prepared to be deployed in a real space
environment. This is because this standard reflects the joint experience and knowledge gathered
by ESA during many years of space activity and software development.
3.3 Protocol
The implemented Distributed Delay Tolerant Protocol (DDTP) protocol design was inspired on both the
Saratoga protocol[18] (see section 2) and the peer-to-peer architecture[25].
In this section the term "transmission" will be used to describe a network communication between two
directly connected nodes, and the term "session" to describe a DDTP, distributed, connection that spans
several transmission over several nodes.
The proposed solution must solve the scenarios described in section 3.1. It must outperform the current
solutions, speeding up transmission times by the use of its distributed architecture.
The major purpose of the implementation is the transmission of large data payloads from and to LEO
cubesats, but it is also intended to solve the transmission of data in spacecraft clusters topologies.
3.3.1 Messages
A total of five message are used to communicate between network nodes, these are illustrated in figure
3.5:
• Session Info
This is always the first message exchanged between two network nodes. It is used to describe the
transmission session. Proceeding messages will be related to this session. It is part of the initial
handshake that takes places in the beginning of any communication, that will be explained in more
20
detail in section 3.3.2. It’s contents include the session destination address, source address and
priority.
• Metadata
Describes a collection of data blocks to be transmitted. This includes the offset of this data blocks
in the overall, distributed, transmission, the data blocks count and the checksums of these data
blocks. This message is always sent after the session info message, during the initial handshake.
• Control
This message is the most unique among the rest, since it is used to send session information
between the two endpoint nodes. These are: the node who has all the data blocks and started the
transmission, and the final destination node. This includes fin, snack, ack flags that inform the
sending node of the termination of a session, a list of data blocks that were not received, and a
sequence of data blocks that were received, respectively. This message represents the only direct
communication that is transmitted between the two end nodes, although, it will most likely need to
be relayed through intermediary nodes.
• Status
Serves the purpose of delivering responses to a node. This is used in the final step of the initial
handshake, in order to inform the node requesting a transmission if this is accepted or not. Multiple
status codes are available for rejecting a transmission in order to inform the node of the reason for
this rejection. This include lack of space, power saving mode, and already processing a priority
transmission.
• ACK
This is a simple acknowledge message. It serves the purpose of reporting to the seeding node
that a sequence of blocks have been received. This information is important so that the sending
node can mark these datablocks as successfully sent and reuse the storage space if necessary.
Metadata
Status
Session Info
Control
ACK
Describes a ongoing transmission between two nodes. This
is used to iden�ty the session and consequently the
des�na�on of the proceeding messages.
Describes a collection of data pieces, requesting the
receiving node to accept the pieces and route them along to
the final destination.
Only processed by the end-to-end nodes. It is used to
deliver connection related information to both ends. This
includes retransmission requests, sacks, fin, ack.
Error and status messages, normally sent as a response to
other messages.
ACK piece message.
Figure 3.5: DDTP Message types
21
3.3.2 Algorithm
The protocol algorithm is defined by two state machines one for controlling the reception and another
the transmission procedures. Moreover, one of the core rules of a session is that when a block is ex-
changed, the receiving node takes in the responsibility for that block, meaning it should keep forwarding
it along the route that will eventually reach its final session destination.
For route determination each node looks at a table kept in memory. Each entry in the table contains met-
ric, destination and route values. There are several possibilities for how the routing table values could
be determined. The simplest one consists in static values, with a lot more complex examples being
possible, like dynamically changing the table to reflect the best space-link route computed by analyzing
the orbital parameters of a spacecraft orbit[26]. This is however outside the focus of this dissertation.
Data blocks then flow trough the network, in no particular order, until they arrive at their destination. If a
data block does not arrive after a long time (e.g. node crashed or a particular link has been offline for a
long time) the receiving node sends a control message (figure 3.5) with a SNACK flag for the block in
question.
One of the protocol requirements is that it should be usable in low resource embedded systems, so some
compromises need to be made. Instead of identifying each data block that is going to be sent through
a cryptographic-safe hash, like in the case of a typical peer-to-peer protocol, a sequence number along
with the transmission session ID is sent.
Each active node in the network starts in the standby state. This state is where the protocol remains
when it has no work to do, more specifically when it doesn’t have anything to send or receive. There
are two ways to leave this state, either a network communication is received from another node, moving
therefore to the reception state machine (figure 3.7), or a request for a new transmission of data is
received from the upper layers of software, moving the protocol to the sender state machine (figure 3.6).
This process also moves the receiver into the pre recv session info state and the sender to the pre
send session info state.
In every exchange between two peers the Session info message (see figure 3.5) is always the first one
sent. This message describes the session in question, its identification (the source address, session id
pair) and priority. The transmission and reception of this message moves the receiving peer from the
pre recv session info to the pre recv state, as illustrated in figure 3.7.
If the sender intends to send data blocks, it will move to the pre sending state, but if a control message
is to be sent, it will move to the send control state, as illustrated in figure 3.6.
In the case of a control message, the receiver moves to the control state. At this point the receiver can
send a message either accepting or rejecting the control message. If the message is rejected, it is up to
the sender to find another route for delivering it to it’s destination. If however it is accepted, the receiver
will store the message and forward to its destination. This moves both machine states to the end state.
Control messages are only processed by the end-to-end transmission peers, every other peer will not
look into its contents. The only exception to this rule is control messages with a retransmission flag.
These may trigger a data block transmission in a intermediary peer if the peer contains the data block in
22
storage, either in alive or zombie state (see section 4.1.5).
If a metadata is received, however, the receiver move to the pre recv 2 state, will parse it and retrieve the
information about the data blocks that the sender intends to send. The sender on its hand will move to
the pre sending 2 state. The receiver will respond with a status message reporting if the transmission
is accepted or not. Reasons for rejecting include lack of storage space, no route available for the session
and power saving modes.
If the transmission is accept, however, the sender move to the sending state and starts sending the
data blocks described in the metadata message until completion or the link drops, while the receiver
move to the recv state and receives the mentioned data blocks.
Pre_send
Session_infoPre sending
Pre sending
p2Sending
Send ControlSend Session_Info
Recv Status(reject)
Recv Status(accept)Send Session_info Send Metadata
Standby
End
Send next block
Recv last ACK
End
Recv Status(accept) and queue message
Recv Status(reject)
Figure 3.6: DDTP sender process state machine
Pre_recv
session_infoPre_recv Recv
Control
Recv Session_info
Pre_recv2
Recv Metadata
Sent Status(Reject)
Sent Status(accept)
Standby
Recv Control
Recv next block
End
Sent last ack
End
Send Status(reject)
Send Status(accept) and mark message has sent
Figure 3.7: DDTP receiver process state machine
23
3.4 Application Scenarios operations
Taking in consideration the scenarios described in section 3.1 and the solution proposed in section 3.3,
the operational behavior of the developed protocol will now be explained in each scenario. This is a
theoretical description that will focus on the more technical aspects, experimental results will presented
in chapter 5.
3.4.1 One-to-one transmissions
Even though the first scenario is the most simple one, and it is already solved by the current solutions,
it is still very relevant. This is because it allows one to test the protocol algorithm in a simple case first,
before going to the more complex scenarios, and because it allow one to compare the proposed solution
performance with the current solutions in the same context.
In figure 3.8 a simplified Message Sequence Chart (MSC) of a transmission between two, directly con-
nected, nodes is depicted. In this example, node A initiates the transmission by sending a session info
and metadata messages. The session info describes the session itself. It contains the addresses,
priority and session id (data structures will be explained in section 4.1.5). The metadata message de-
scribes the data blocks that will be sent in this particular transmission.
The receiver (node B) after receiving these messages will have all the information it needs to "decide" if
it is able to accept and forward along the data blocks.
Since in this particular example node B is the final destination, it will sent a status message informing
node A that it accepts the transmission. After this node A starts sending data blocks.
This process is very similar to the one described in section 2.2. This is to be expected since they are
both solving the same scenario. The DDTP exchange however requires a higher network overhead
since more metadata is sent. This may be decremental in this particular scenario, but will result in a
much higher total throughput in the new scenarios.
Session info
Metadata
Status
Data
Data
Ack
A B
...
Figure 3.8: MSC One-to-one transmission
24
3.4.2 Multi-to-one transmissions
A generalization of the previous scenario is the processing of multiple transmissions in parallel. This
means that the protocol implementation should be able to handle the simultaneously reception of mes-
sages from different sessions.
In figure 3.9 an example of this is showed. Nodes A and C have data blocks to send along the network
and have selected node C as the next hop, simultaneously. Just in the previous scenario, they sent a
session info and metadata message to node B, describing the transmission itself and the data blocks
that they intend to send. At this point, node B parses this data and responds with a status message,
ether accepting or rejecting the requested transmission.
In this example, node B has accepted both transmissions (this can be observed since node A and C
proceeded by sending data blocks), but it could also had been the case that transmission A would had
been rejected, or transmission B, or even both. Reasons for a rejection include not enough memory for
storing the offered data, energy conserving modes and a priority transmission is already using the link.
Session info
Metadata
Status
Data
Data
Ack
A B
Session info
Metadata
Data
Data
C
Status
Ack
... ...
Figure 3.9: MSC Multi-to-one transmission
3.4.3 One-to-multi transmissions
In figure 3.10 an example of this scenario is displayed. The major difference in this scenario is the usage
of multicast messages. This means that one cannot rely on the receivers acquiring all the messages
(there is not re-transmissions at the transport layer). This means that the data messages need to be
sent together with the information about the DDTP session. This is displayed in the example figure, over
the RDP layers, the necessary fields are inserted in the DATA message. There is offcourse no way to
ensure that all the data blocks arrive at all the receivers correctly, but for this the retransmission message
explained in section 3.3.1 will be use by the receiver to request a normal, unicast, retransmission of the
mission data blocks.
25
DATA
Session
InfoSequence
RDP Multicast
CSP Multicast
AX25
Figure 3.10: One-to-multi transmission
3.4.4 Distributed transmissions
In this section, while using the concepts introduced in the previous scenarios, a distributed transmission
will be explained. In figure 3.11 an example is illustrated, where each black arrow corresponds to a
one-to-one transmission (explained in section 3.4.1) and each red arrow represents a link drop. The
example is divided in three steps, A, B, and C that correspond to three transmission opportunities in
an orbit. The example corresponds to a spacecraft (ISTSAT-1 in the figure) downlinking to earth a data
payload through multiple GSs. This is the major use case for which this protocol was developed.
When the spacecraft (ISTSAT-1 in the figure) goes over the first GS (GS1 in the figure) it obtains its first
transmission opportunity. During this time windows the ISTSAT-1 will transmit as much as possible of
the data blocks until the link drops and it is forced stop. This pieces are then sent from the GS1 to the
Mission Control Center (MCC) server through the Internet.
An important thing to notice is that even though, in this example, multiple transmission will occur over
different links, in a distributed manner, each individual transmission is performed in the same way as a
one-to-one transmission from section 3.4.1.
This process repeats two more time in step B and C until all data blocks are sent and reassembled in
their final destination, the MCC server.
Another interesting possible example is the transmission of data to earth from a cluster of cubesats. This
is illustrated in figure 3.12, where the blue spacecraft, in order to transmit the data to earth, needs to
forward the data through the remaining spacecrafts. The non-dotted lines represent good quality links,
which are spacecraft-to-spacecraft and earth communications, and the dotted lines represent low quality
disruptive links, which are spacecraft-GS communications.
This example can also be solved by the distributed transmission procedure, where the first spacecraft
(marked with an A in the figure) will first transmit the data to other spacecrafts that can reach earth,
26
who will in turn, according to their transmission opportunities, forward the data to earth. The spacecraft
selection depends on the route algorithm used, as explained in section 4.1. If two or more GSs are at
transmission range of the spacecraft, the selection of which will be used will also depend on the route
algorithm. Transmission collisions are avoided since all packets are addressed to a single receiver.
ISTSAT-1 GS1ISTSAT-1 GS2 GS3 MCC Server
Link Drop
Link Drop
Link Drop
Transmission
Transmission
Transmission
Transmission
Transmission
Transmission
A)
B)
C)
Figure 3.11: MCC Distributed Transmission
27
A)
B)
Figure 3.12: Distributed transmission - cubesat cluster example
3.4.5 POCKET++
Pocket++[27] is a lossless compression algorithm developed by ESA, designed for spacecrafts with
few resources. It is able to achieve a good compression ratio, specially on data payloads with period
information, such as is usually the case with spacecrafts telemetries. For the ISTSAT-1 however, the
POCKET++ compression will be used for compressing the captured ADS-B records, and thus it should
work together with the DDTP protocol.
Since DDTP does not look into the contents of the data payload it is transmitting, there is no need
to adapt its architecure to support Pocket++. There are however some requirements for transmitting
Pocket++ compressed data, of which the major one is that the data being compressed must be divided
into periodic, fixed sized, frames. This means that the usage of Pocket++ over DDTP can impact the
transmission quality, since each frame can only be decoded after it is completely received. The impact
of such combination will be explained in chapter 5.
28
Chapter 4
Implementation
Any software project should enforce a clearly defined set of methodologies, tools and procedures to be
used in its development. This very important for the success of any engineering project, but particularly
in a project like this, that not only features a lot of members working together, from different engineering
and professional backgrounds, but also where any error or mistake can be fatal, like it is typically the
case in space related projects.
In this section the complete methodology employed in the development of this project will be explained
in detail, some originating from the ISTSAT-1 project, others from the ESA FYS programme and some
from the DDTP project itself.The implementation will also be explained in detail. This includes all the
design decisions that where made to accommodate the specified requirements in the best way possible.
4.1 Software Architecture
The software development was divided in three modular sections: protocol, route and storage, as pic-
tured in figure 4.1. These were implemented in the form of independent software libraries, containing
the core protocol implementation, the route calculation, and the payload data storage module, respec-
tively. This approach was used be the DDTP implementation is designed to be used in different kinds of
systems, with architectures and capabilities.
The storage lib creates an abstraction level between the protocol lib and the selected storage medium,
this results in an implementation that is not only designed for one storage method but can be more easily
ported to multiple. In the illustrated example two mediums are available: RAM and flash storage.
The route lib offers an interface for accessing the routing table and dynamically updating its entries if
such is desired, allowing, due to its modular nature, the implementation of multiple route calculation
algorithms.
The protocol lib contains the actual software implementation of the DDTP protocol, including its algo-
rithms and messages.
The application level software can be any software that needs to perform a DDTP transmission. The
only requirement it needs to fulfill is to interface with the protocol library in order to perform a DDTP
29
transmission.
Protocol Lib
DDTP
Application
RAM Storage
Storage Lib
Storage
Flash Storage
Route Lib
Algorithm BAlgorithm A
Routes
Figure 4.1: DDTP modules
In figure 4.2 a simplified version of the software structure is illustrated. The project was divided into
ten major files (more were used but are omitted here for the purpose of clarity and lack of relevance).
These are:
• ddtp
The main file that gives the upper layers of software access to the DDTP protocol. This includes
functions as ddtp_recv and ddtp_send that serve the purpose of queuing data to be sent and
received. In the example figure 4.2 these functions are being called by the main function.
• ddtp_sm
The file that contains all the information related to the DDTP state machines explained in section
3.3.2.
• ddtp_conn
This file defines the structures and functions related to each connection being processed. Together
with ddtp_sm, this two files contain most of the internal state of the program when it is running.
• ddtn_messages
Defines all the protocol messages, its construction, serialization and parsing.
• ddtp_rtable
The protocol routes are stored and processed here. Can use external files as source of the routing
algorithm.
• ddtp_storage
Defines all the functions and structures related to the storage of data blocks. Can use external
files to defines new storage mediums.
• ddtn_net
All access to the lower network layers are defined in this file. The purpose of this file is simply to act
as an interface to the actual files that defined each network layer. Depending on the compilation
options and interface in question different network layers will be accessed.
30
• ddtn_net_rdp
The file that implements the access to the actual RDP protocol. The real RDP protocol processing
is done inside the libcsp C library.
• ddtn_net_inet
The file that implements the access to the TCP/IP protocol.
• ddtn_net_omnet
The file that implements the access to the OMNET simulated TCP/IP and UDP/IP protocol.
app
ddtp.h
ddtp_sm.h
ddtp_net.h
ddtp_inet.hddtp_rdp.h ddtp_omnet.h
libcsp.h omnet_inet.hsocket.h
ddtp_messages.h
ddtp_rtable.h
ddtp_storage.h
ddtp_conn.h
main()
ddtp_create()
ddtp_init()
ddtp_halt()
ddtp_send()
ddtp_recv()
ddtp_close()
ddtp_net_recv()
ddtp_net_send()
ddtp_net_accept()
ddtp_net_create()
ddtp_net_close()
ddtp_net_bindandlisten()
ddtp_net_init()
ddtp_net_halt()
ddtp_accept()
ddtp_task()
... ...
...
...
...
... ... ...
... ... ...
Figure 4.2: DDTP software architecture
4.1.1 Multi platform
One of the DDTP protocol requirements is the support of multiple OSs. To achieve this, an architecture
similar to the Libcsp was used. All system related functionality is isolated in separated modules, one
31
for each different OS. Every time a system related call is required by the software, instead of directly
accessing a particular system interface (which would result in system lock-in), an internal DDTP inter-
face module is called. This interface, called Operating System Abstration Layer (OSAL) in figure 4.3, will
select the module of the particular system being called, make the necessary adjustments according to
the system features and limitations, and execute the system depend functionality. This abstraction layer
offer the possibility of easily porting the software to different platforms by only creating a new platform
specific module that implements the specified interface. Without this technique it would be necessary
to perform deep and unclear modifications to the source code, with a high probability of creating new
problems and bugs. The compilation procedure was designed so that one is able to only include the
modules corresponding to the necessary platforms, thus reducing the compiled executable size and
memory footprint. Despite the OSAL interface, an effort was made to avoid using functionalities that
were not available in all the targeted platforms, since doing so would require the development of this
functionalities in the modules where it was unavailable. Particularity the FreeRTOS platform is much
slimmer in terms of functionality than the remaining platforms.
This design follows the common mentioned software development principle: The code should be de-
signed to be extended, not modified.
Modules for three different platforms were developed, which will be explained bellow, but more could be
easily created.
Protocol Lib
DDTP
FreeRTOS Linux OMNETPP
Operating System Abstraction Layer (OSAL)
Figure 4.3: DDTP software platform architecture
4.1.2 Linux
The first platform for which the DDTP protocol was developed was Linux. This was the most obvious
choice since all the development was done in Linux and it is also the most convenient system for execut-
ing all the tests described in section 4.7, that were essential to maintaining the project quality. Moreover
the ISTSAT-1 GS will running Linux, making it a very important platform for this project.
32
4.1.3 OMNET
For the purpose of testing and performance analysis the OMNET simulator was used. This simulator
interfaces with the DDTP project twice, one for sending and receiving messages through the simulated
network, and another for executing the protocol and the associated application itself. Since a simulator
can create events, artificially accelerate the simulation time and extract data from the execution, such
integration is necessary. For the protocol interface the OSAL level was used, like the other implemented
platforms. The simulator itself was executed in Linux.
4.1.4 Multi Network stack
The DDTP protocol is being developed under three different network stacks: TCP/IP, CSP/RDP and
OMNET’s simulated TCP/IP stack. This is illustrated in figure 4.4 where, in similarity to the approach
used for the multi-platform problem described in section 4.1.1, an abstraction layer was used. This layer
was named Network Protocol Abstration Layer (NPAL) and it abstracts every access between the DDTP
protocol and its lower network stacks. This allows one to easily extend the project with new network
stacks, add and remove stack modules without breaking the project (something very useful for memory
constrained systems) and have multi stack compilations. This modular approach also allows one to test
each module in isolation, something very useful for pinpointing problems early on.
Protocol Lib
DDTP
TCP/IPV4 RDP/CSP OMNETPP
Network Protocol Abstration Layer (NPAL)
Figure 4.4: DDTP - Network Abstraction Layer
Multiple network stacks are necessary for this project since communications will travel over two very
distinct mediums, requiring different protocols. In figure 4.5 the multiple stacks are illustrated, where
depending on the different medium, either the RDP/CSP or TCP/IP stacks will be used.
The CSP/RDP stack is useful for space-link communications, like the ones between GSs and space-
crafts or spacecraft-spacecraft. This is due to its simplicity and low overhead. In this scenario the major
concern is the quality of the link and not its congestion, in contrast with the typical earthly communication
scenario like Internet connections.
33
Data Link
Network
Transport
Bundle
Application
Specific for each
network region
Common over across
all network regions
AX25
CSP
RDP
DDTP
App
Ethernet
IP
TCP
DDTP
App
OMNETPP
Simulator
DDTP
App
Figure 4.5: DDTP Network stack
AX25
CSP
RDP
DDTP
App
AX25
CSP
RDP
DDTP
AX25
CSP
RDP
DDTP DDTP
App
Ethernet
IP
TCP
Eth
IP
TCP
Figure 4.6: DDTP Multi stack transmission
The TCP/IP stack is, on the other hand, appropriate for Internet connections and is being used for the
purpose of inter connecting GSs. Both TCP and RDP protocols offer guaranties of integrity, message
order and delivery acknowledgment. These are the minimal requirements for a network stack running
the DDTP protocol.
The OMNET stack is only used for testing, debuging and performance evaluation purposes. It comprises
of a simulated TCP/IP stack that runs inside the OMNET simulator.
A single transmission can flow trough more than one stack as long there is a multi-stack node, like it is
illustrated in figure 4.6. The nodes with multi stack capabilities are called gateways and are necessary
for enabling communications between the different mediums.
4.1.5 Data structures
Each DDTP session is defined by four parameters:
• Source and destination address
These addresses identify the nodes sending and receiving the data, respectively. These identify
uniquely the nodes and are not related to their OSI Layer 3 addresses. Since the DDTP protocol
was developed to work with big networks, something specially relevant in the case of cubesat
clusters, this address has 32bits in size. Although, since the implementation was designed to be
configurable, the size could be easily changed without requiring software adaptations.
• Session ID
34
This is a counter of how many sessions a node has initiated. Each node stores this value for its
own sessions. Together with the source address the session id identifies uniquely a session. This
pair of values is always the identification used for a session, and is sent in the metadata message
in the beginning of a transmission, as explained in section 3.3.1.
• Priority
Represents the priority value of a session related to other sessions that may be executed in parallel
over a single link. This is an optional field.
In addition to these, a data block counter is used to identify each data block in a session. This is neces-
sary since data blocks do not arrive in an orderly fashion, due to the distributed nature of the protocol,
and to identify block in retransmission requests. In a regular Peer2peer protocol, a cryptography-safe
computational digest would be used for both identifying the block and confirming their integrity. Although
this solution is more elegant, it is not feasible in a cubesat like system where processing power is very
limited, so the mentioned solution was chosen.
Each data block stored in by a peer can be in three different states:
• Alive
A datablock relative to a session is stored. And, as such, needs to be forwarded to the next peer
in route to its destination.
• Dead
The structure is free and can be safely used.
• Zombie
The datablock is free and can be used, but contains a valid data block of a session. When a
node transmits a data block to another one, it no longer needs to store it, but there is however
no cost associated with keeping the data stored. Because of this, the data block state is simply
changed to zombie and all software references to this block are maintained. Besides keeping this
data available, even though it has already been sent, its real advantage comes from the fact that
in the event of a retransmission request from the session destination node, the data block can be
recovered and retransmitted at no added cost.
The project was developed with the intent of being highly customized. For the Linux implementation a
key/value configuration file can be used to configure the software with all the possible parameters. This
allows one to change the configurations without having to recompile the software. This can be very
useful for GS configurations.
For embedded systems however, like the ones using FreeRTOS, this solution is not possible. So the
same parameters needs to be defined by hard-coding them in the software before uploading the resulting
binary executable to the system.
These parameters include the following values:
• interfaces
35
• interface costs
• routes
• gateway
• network stacks
• log level
• addresses
4.2 Programming language selection
One of the first decisions that it was necessary to make, was selecting the programming language to be
used in the project development. A crucial decision that by itself establishes a lot of the project develop-
ment foundations. There are two major requirements in the development of software to be deployed in
a spacecraft: Reliability and low resource usage. Particularly in a Cubesat the resources available are
very limited.
In terms of resource usage, the C programming language as a clear advantage. By being a very low
level language (sometimes even referred as high level assembly) its overhead is minimal, the available
compilers produce very optimized machine code and it allows the programmer to access low level infor-
mation that can be used to write more optimized code in certain instances. All this comes at the cost of
programming complexity however.
For this requirement all the modern, very popular, high level languages as Java, Python and Ruby are
completely out of question. They would bring great benefits, not only by allowing the programmer to
write less complex, more easily understandable and maintainable code, but also due to the security and
robustness that their automatic memory management offers (the majority of errors in C written code
originates during memory and pointers operations). However this advantages come at the cost of a very
high level, resource demanding system, that is not acceptable for a Cubesat like the ISTSAT-1.
In terms of reliability the C programming language is a very well known and tested solution. Its been
constantly used by the industry since it was first developed in 1972, there are many well test, constantly
in use, freely available compilers and has a lot of space heritage. Its lacks however a memory manager,
thus requiring the programmer to perform all the memory accesses itself. This is typically a cause of
many problems. The use of strict coding rules and guidelines (like the ones specified by NASA or ESA)
can however greatly reduce this risk.
The C programming language is also the most portable by far. Whichever combination of Center Pro-
cessing Unit (CPU), Instruction Set Architecture (ISA), and hardware one selects, there will probably be
a C compiler available. Since the ISTSAT-1 is still in development, this is an important factor. This also
allows one to easily share code between different subsystems in the spacecraft and on earth.
Considering the overall scenario, the C programming language was chosen for the development of this
36
project. Not only does it fulfill the needed requirements, but along with its space heritage and portability
it makes a very solid choice.
4.3 Build tools
Since the project needs to be compiled and build in several different configurations, both due to its
modular software approach explained in section 4.1 and supporting multiple platforms a flexible solution
for compiling the project was required. The cmake tool was chosen due to its reliability from being a well
tested and wildly used solution that is also multiplatform. The gitlab automatic test suite was also used,
this not only allows the frequent build of different versions of the software during its development, but
also running code analysis and several tests frequently.
4.4 Code Analysis
There are two types of code analysis that are of great benefit to any software project: Static and dynamic.
Both were used in this project.
4.4.1 Static
For static analysis two tools were used, an Integrated Development Environment (IDE), named Inteljea
Clion, and the clang compiler static analysis tool. These tools offer different advantages and are better
used together.
The clang compiler static analysis tool, however offers a slower but more deep insight into the state
of the written code, checking for more complex bugs, coding styles inconsistencies, and other hard to
track problems. These include checking for out of bounds, memory leaks, uninitialized variables and
functions, cast problems, initialization problems, null pointers and some others.
4.4.2 Dynamic
For dynamic analysis two tools were used, GNU Debugger (GDB) and Valgrind. The GDB, a well known
tool, allows one to monitor, pause, and edit the developed software during its execution. Valgrind on the
other hand is a dynamic analysis tool capable of detecting memory leaks, uninitialized or un-allocated
memory usage. This is extremely important for a project such as this, software bugs can be critical.
4.5 Documentation
A common problem that occurs in many software projects is the lack of documentation. Even a very
well developed project can become unusable if there is no record of how it can be used. Particularly in
complex or critical project (like space related ones) this requirement becomes even more serious. To
37
solve this problem two types of documentations were produced: Software reference documentation and
Design documentation.
4.5.1 Software reference
Software reference documentation is a type of document that describes in detail the full API (ideally
both internal and external to the project) of the developed software. This should detail all the relevant
information about the project functions, variables, directories, files and anything else that may be rele-
vant. This can be a daunting task, but if one uses the proper tools and consistently keeps documenting
the developed software throughout the project, it becomes a simple task that will greatly improve the
project’s quality.
To achieve this purpose the Doxygen1 software was used, a tool that generates reference documenta-
tion from analyzing annotated C source code. This results in both consistently documented code, in a
standard format, and the production of a very detailed reference documentation.
4.6 Source Code Version Control
Using a source code version control tool is not necessary for any software project, but it was also
highly recommended by the ESA FYS program. It allows one to keep a commented history of all the
modifications done throughout the project, allows a hole team to work collaboratively while keeping a
backup copy in a server.
Initially subversion was used and latter the project was moved to git. Both tools are fine to use for
this project, with the git tool being a more optimized, distributed solution and subversion working in a
classical, centralized, server-client model. A git related tool called gitlab was used to complement the
source version control with some other features like automatic testing capabilities that will explained in
more detail in section 4.7.
4.7 Testing
Constant and thorough testing is a fundamental part of any software project, but it is even more important
when it is space related. For the development of the DDTP protocol two types of tests were used Unit,
and load tests.
• Unit Test
Unit testing is a technique for checking and maintaining the developed code quality. For each unit of
code, normally this can be functions, files or modules, one or more tests are developed. These will
execute the unit with different input values, test its execution intermediary values through assert
statements, and test the resulting output. This simple concept allows one to test the software
with a very small granularity. This is very important in order to find problems early, precisely, and
1https://www.stack.nl/~dimitri/doxygen/
38
throughout the development of the project. New problems and bug can be discretely introduced
each time one changes to a project.
To perform this task an open-source C lib was used called munit 2.
• Load tests
To ensure that the software is capable of functioning correctly when its operational limits are
reached, load test are used. Theses tests run several executions of the software in limit push-
ing situations (eg. over maximum connection requests, very high delays, very small delays) and
assert whether the software is able to deal with this situations without deriving from following the
specification (eg. crashing).
To achieve this a simulator was used. Due to the exotic nature of this protocol (distributed radio
communications) fully non simulated executions are challenging, and since an important factor of
any kind of software test is to be able to execute it frequently, OMNET a network simulator was
used. The usage of this simulator and its related environment will be explained in section 5.
4.8 Development Environment
All the software development was done in a Linux OS, with the remaining platforms and network stacks
being implemented at a second stage. The first execution corresponded to simply running unit tests,
which progressed to localhost transmissions until it was necessary to use a network emulator to execute
more complex scenarios.
For the purpose of executing the complete developed protocol (as opposed to the unit tests, that execute
small unit individually), the OMNET3 was used. This software simulates a network typology, allowing one
to prepare big and complex networks, both wired and wireless, with programmable event, while allowing
the nodes to execute arbitrary C++ software. This means that it was possible to adapt the DDTP protocol
to run in the simulated nodes of a OMNET network.
Other possibilities were also considered, like the Netkit4 simulator, that even though it is a less powerfull
solution compared to OMNET it does not require the implementation of a new network interface, as
uses the TCP/IP Linux interface. This however proved not to be a sufficiently capable solution for the
scenarios that needed to be emulated.
In term of programming practices. a Keep It Simple and Stupid (KISS) approach was used, the clang-tidy
tool was used for setting and keeping a code format and the ESA coding standard for space software
was followed, in order to keep the software at a level of quality necessary for space deployment. One of
the key requirements of this standard is that one should not use any type of dynamic memory allocation.
All the memory resources should be allocated at the beginning of program execution. This is not only to
avoid the typical memory allocation bugs but also to allow one to easily calculate how much memory the
software will use, an important metric since spacecrafts tend to be very resource constrained.
2https://github.com/nemequ/munit3https://www.omnetpp.org/4http://wiki.netkit.org/man/man7/netkit.7.html
39
40
Chapter 5
Experimental evaluation
In order to validate and quantify the reliability, functionality and quality of the designed solution, a set
of tests were performed. These are divided into the scenarios described in section 3.1 so that one
can confirm and analyze the core features of DDTP. Since the characteristics necessary for testing a
distributed DT protocol (such as high latency, high error rate, link disruptions, distributed network) are
difficult to emulate in laboratory, the OMNET network emulator was used to create a simulated network.
To add more value to the obtained values and compare them to the current available solutions, a publicly
available saratoga reference implementation was evaluated in the same scenarios. An unmodified copy
of its C++ implementation1 was used at revision 56b9f1.
In figure 5.1 the simulated topology used for the following tests is illustrated. Each node of the network
is connected to each other by a bidirectional channel with a datarate of 10KB/s, where links can be
disconnected during a transmission, through scheduled events, or even before the simulation (forcing a
transmission to follow a particular route). All nodes in the topology are running the same DDTP imple-
mentation, with same the upper layer software (see figure 4.3). They do however contain different route
and network configurations, since they are placed in different network positions. It should be noted that
node A and B are the only two nodes that contain more than one link, and so, have the possibility of
using another network route.
1https://sourceforge.net/p/saratoga/saratoga-vallona/ci/default/tree/
41
Link α
Link βA
B
C
DLink δ
Link γ
Figure 5.1: Testing simulation topology
5.1 One to one
In order to measure the performance of both protocols in a one-to-one scenario, using the topology
illustrated in figure 5.1, multiple transmissions were performed between node A and B, exclusively over
link α (link β was disconnect), with multiple data payload sizes. All the values obtained in this section
were captured in the network interface of node B.
In figure 5.2 the DDTP and Saratoga transmissions are illustrated, where one can observe that the trans-
mission time increases linearly with the payload size. In this scenario the Saratoga protocol is clearly
faster, which is to be expected since it uses less metadata due to its decreased functionality compared
with DDTP. However, even thought the difference is relatively small, this means that in a exclusively
(combinations are possible) one-to-one scenario, a simpler protocol like Saratoga is preferable.
In table 5.1, the values of the transmission are displayed, with DDTP achieving an average throughput
of 8.909KB/s and Saratoga of 9.295KB/s.
Table 5.1: One to one transmission
Time (s) Throughput (kB/s)
Size (KB) DDTP Saratoga DDTP Saratoga
100 11.9 10.6 8.4 9.4
200 22.8 21.4 8.8 9.3
300 33.1 32.1 9.1 9.4
400 44.1 42.9 9.1 9.3
500 55.0 53.4 9.1 9.4
1000 111.7 109.4 8.9 9.1
2000 222.1 218.7 9.0 9.1
42
0
10
20
30
40
50
60
0 100 200 300 400 500 600
Tim
e (
s)
Data payload (KiB)
DDTP
Saratoga
Figure 5.2: DDTP vs Saratoga - One to one
5.1.1 One to one - Delay tolerant
Another feature that requires verification is the DDTP DT capabilities. Again, the transmissions were
performed between node A and B (5.1), exclusively through link α (link β was disconnect), with a 200KB
data payload, and all traffic was captured at the network interface of node B.
In figure 5.3 the DDTP transmission throughput is displayed in function of the time, where each point in
the chart corresponds to a 1 s average of the throughput. A link disruption was scheduled at the 5 s mark,
causing an unexpected interruption in the transmission. This forces the transmission to pause, waiting
for the link re-connection, since no other routes are available. As can be observed in the mentioned
figure, from the 5 s mark to 53 s no data was transmitted. At that point the link reconnection is detected
and the transmission is resumed, finishing at 72 s.
As was desired, the property of being able to pause and resume a transmission according to the link
availability was validated. This is one of the core requirements of the developed protocol.
In figure 5.4 the same experience, in the same conditions, was repeated with the Saratoga imple-
mentation. Here the link is once again disconnected at the 5 s mark, reconnecting at 20 s. Similarly to
DDTP the transmission was able to pause and resume accordingly to the availability of the link.
43
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
0 10 20 30 40 50 60 70 80
Th
rou
gh
pu
t (B
/s)
Time (s)
Route A
Figure 5.3: DDTP DT transmission
0
2000
4000
6000
8000
10000
12000
0 5 10 15 20 25 30 35 40
Th
rou
gh
pu
t (B
/s)
Time (s)
Route A
Figure 5.4: Saratoga DT transmission
44
5.2 Multi to one
In order to validate DDTP abilities in a multi-to-one scenario two transmissions were performed. Using
the same topology from the previous section (see figure 5.1), two transmissions of a 200KiB data pay-
load were performed simultaneously. One from node B to D, over link γ, and another from node C to D
over link δ. It should be noted that in this experiment node D is receiving both transmissions at the same
time. Link α and β were disconnected to ensure that no traffic goes through node A.
The experiment was performed with both DDTP and Saratoga separately, as ilustrared in figure 5.5. The
DDTP took a total of 44.796 s and Saratoga 42.783 s. It is to be expected that Saratoga achieves a better
result in this case since, similarly to the results in section 5.1, it uses less metadata in its messages.
The real advantage of DDTP will be observed in the experiments performed in more complex scenarios
that will be described in the following sections.
0
2000
4000
6000
8000
10000
12000
0 5 10 15 20 25 30 35 40 45 50
Th
rou
gh
pu
t (B
/s)
Time (s)
DDTP
Saratoga
Figure 5.5: Multi to one - DDTP vs Saratoga
5.3 One to multi
Using the topology illustrated in figure 5.6, with a 10 kB/s channel that directly connects node A, B and
C, a 200KiB transmission was performed. This transmission originated from node A to, simultaneously
node B and C. This corresponds to a one-to-multi transmission as defined in section 3.1. The transmis-
45
sion is illustrated in figure 5.7. It is important to note that this case is very different from the remaining
scenarios. This is because, since messages are sent in multicast, there is not confirmation from the
recipients of their correct arrival (i.e. an ACK message). To solve this problem, recipients will send
retransmission requests for missing or corrupt data blocks.
In this particular case, two copies of the same 200KiB data payload were sent to node A and B in a total
of 20.887 s. The alternative to this woud be to perfom 2 one-to-one transmissions.
Looking at table 5.1, one can see that DDTP took 22.8 s to perform a one-to-one transmission of a
200KiB payload. This means that to solve this scenario with only one-to-one transmission, it would be
necessary to perform two equal transmissions taking a total of 45.6 s.
In this scenario DDTP’s multicast mechanism clearly offers a much more optimized solution.
A
B
C
Figure 5.6: One to multi topology
46
0
2000
4000
6000
8000
10000
12000
0 5 10 15 20 25
Th
rou
gh
pu
t (B
/s)
Time (s)
Figure 5.7: DDTP - One to multi transmission
5.4 Distributed
To validate DDTP main feature, its distributed property, a transmission of 200KiB was performed be-
tween node A and D, using the topology illustrated in figure 5.1. All links and nodes were connected at
the beginning of the simulation, with a link disruption scheduled for link α. Node A has two routes for
reaching node D: Route A through link α; and route B through link β. Unlike what happened in section
5.1.1, where only one route was available and so node A was forced to pause the transmission, when
the disruption occurs the transmission should resume through route B.
In figure 5.8 this experiment is illustrated, where each point in the chart represents a 1 s throughput
average. When the link α is disconnected, at the 9 s mark, the transmission through route A stops. At
this point any current solution would proceed by simply pausing the transmission and waiting for the link
restablishment. This is the behavior observed in figure 5.4 by Saratoga. Unlike Saratoga, however, the
DDTP resumes the transmission through route B as soon as it detects that route A is no longer avail-
able. The result of this is the full transmission finishing in under 30 s. The simpler DT approach used
in section 5.1.1, with the same data payload, resulted in a 72 s transmission time, this corresponds to
a huge improvement on DDTP part. It should also be taken into account that for bigger link disruption
times this difference is higher, increasing linearly with the link disruption time.
47
0
100
200
300
400
500
600
700
800
0 5 10 15 20 25 30
Th
rou
gh
pu
t (B
/s)
Time (s)
Route A
Route B
Figure 5.8: DDTP distributed transmission
5.5 Compression
In section 3.4.5 the Pocket++ compression scheme was introduced. Since the ISTSAT-1 will compress its
mission data with this software, the DDTP should be tested with it, not only to confirm its compatibility but
also to measure its performance. To this effect, a sequence of 2262 spacecraft telemetry messages were
transmitted between node A and B, in the topology illustrated in figure 5.1, first in their original format
and then compressed by Pocket++. The original message sequence accounted to a total of 277KiB,
and the compressed version 68KiB, resulting in a ∼4.07 compression ratio. Since the messages used
correspond to samples from a spacecraft telemetry, a high compression ratio is to be expected. This is
because it is normally the case for telemetry to contain repetitive or similar values over time. The results
of this transmission are displayed in figure 5.9. The original messages took 30.975 s to finish while the
compressed version only took 8.382 s.
This test results can be extended to the, more complex, distributed scenario. This is because it makes
no difference to DDTP the kind of data payload it is transmitting (whether it is compressed or not).
In the case of ADS-B messages, their content is very repetitive by nature. This is because their
fields (e.g airplane ID, geographical coordinates, flight number) contain information within a fixed range
of values. Moreover, this is specially true for a set of airplanes geographically close to each other, which
will be the case for the ones tracked by ISTSAT-1. Due to this, it is expected that, much like the example
above, Pocket++ will provide a high compression ratio when applied to ADS-B messages.
48
0
2000
4000
6000
8000
10000
12000
0 5 10 15 20 25 30 35
Th
rou
gh
tpu
t (B
/s)
Time (s)
Uncompressed (277KB)
Pocket++(68KB)
Figure 5.9: Pocket++ transmission
If one were to configure the ISTSAT-1 mission module to only track a particular airplane, or a small
collection, the compression ratio would be even higher, since not only would some of the message field
remain constant, such as the airplane and flight identification, but fields like the geographical coordinates
would change in a incremental fashion. Thus increasing their recursive information.
49
50
Chapter 6
Conclusion and future work
A Cubesat is a very resource constrained type of spacecraft, with low energy reserves, small solar pan-
els, and low communication and processing capabilities. Due to this, the development of a Cubesat
offen requires very optimized solutions. The ISTSAT-1 mission is to perform a feasibility study of the
use of cubesats for receiving ADS-B signals from traveling airplanes. A safety system that broadcast
information from each airplane. It then needs to transfer the processed ADS-B information to earth,
through one or more GSs. Since cubesats are typically launched in LEO, they have a very fast traveling
speed (close to 8 km/s), that results in a very small time window (under 20min) to communicate with the
GSs on earth. Current solutions solve this problem by operating in a DT fashion.
This project proposes a new approach, the development of a distributed DT protocol, capable of per-
forming the same tasks already being solved by the current solutions but in a much more optimized way,
increasing the overall speed at which data can be received. This solution is not only appropriate for
spacecrafts like the ISTSAT-1, that simply wish to transmit mission data to a server, but can also be used
for more complex networks like spacecraft clusters with ISL and the simultaneous transmission of data
to multiple receivers on earth, like a WSN.
Four application scenarios were defined, and these encompass the major requirements necessary. From
these four, only two are currently being solved by the existing solutions: one-to-one and multi-to-one
transmissions. The developed solution was designed to not only solve the first two but also the new
ones: one-to-multi and distributed transmissions.
The experimental results showed that using DDTP will result in much faster transmissions in two of the
scenarios.
When multiple links are available (distributed scenario) DDTP is able to make use of these links and
avoid completely pausing the transmission when a disruption occurs. This offers a very significant per-
formance improvement over current solutions which simply operate in a DT fashion in these scenarios.
This is the use case of the ISTSAT-1 project.
When multiple transmission destinations are available over a single link, DDTP also manages to reduce
very significantly the transmission times. While current solutions simply perform several one-to-one
transmissions to solve this problem, DDTP multicast capabilities allow the transmission to reach all des-
51
tinations simultaneously. This is the use case of a WSN whose nodes are receiving a firmware update
over a space link.
The first two scenarios, one-to-one and one-to-multi are solved by both DDTP and the current solutions,
but due to the increased metadata usage in DDTP current solutions are actually slightly faster, in the
one-to-one test Saratoga was able to finish the transmission 1.8 s faster than DDTP. However, purely
one-to-one or one-to-multi scenarios are not the use case for which DDTP is designed for.
Future work should include the testing of DDTP in real world scenarios, including its integration with the
remaining systems of the ISTSAT-1 cubesat. More complex routing algorithms could also be developed,
potentially increasing transmission speeds even further. One possibility would be having DDTP collect
information from the module responsible for determining the ISTSAT-1 location (the Altitude Determina-
tion and Control System module). This would allow DDTP routing algorithm to predict when links would
become available/unavailable and make choices based on this information. DDTP power usage onboard
of ISTSAT-1 should also be analyzed and, if necessary, optimized.
52
Bibliography
[1] CalPoly, “Cubesat design specification”, The CubeSat Program, California Polytechnic State . . .,
2009, ISSN: 1110662X. DOI: http://dx.doi.org/10.1155/2015/681901.
[2] Secure World Foundation, New Actors in Space. 2011, p. 159, ISBN: 9781618398055.
[3] J. Ferreira, “ISTNanosat-1 Heart”, Master’s thesis, IST - Instituto Superior Técnico, 2012. [Online].
Available: https://fenix.tecnico.ulisboa.pt/downloadFile/395144736837/MScThesis-
JFerreira_vFinal.pdf.
[4] A. Mccook, “On the right path”, Nature, 2011, ISSN: 00280836. DOI: 10.1038/nj7353-667a.
[5] P. Coelho, “ISTNanosat-1 Quality Assurance , Risk Management and Assembly , Integration and
Verification Planning”, no. May, p. 107, 2016.
[6] H. Weiss, “Delay-Tolerant Networking Architecture”, RFC Editor, RFC 4838, Apr. 2007, pp. 1–61.
[Online]. Available: https://tools.ietf.org/html/rfc4838.
[7] A. Schlesinger, B. M. Willman, L. Pitts, S. R. Davidson, and W. A. Pohlchuck, “Delay/Disruption
Tolerant Networking for the International Space Station (ISS)”, IEEE Aerospace Conference Pro-
ceedings, no. October 2010, 2017, ISSN: 1095323X. DOI: 10.1109/AERO.2017.7943857.
[8] O. N. Challa and J. McNair, “CubeSat Torrent: Torrent like distributed communications for CubeSat
satellite clusters”, Proceedings - IEEE Military Communications Conference MILCOM, 2012. DOI:
10.1109/MILCOM.2012.6415828.
[9] R. F. Afonso, “Sistema de Gestão e Determinação de Atitude do Engenharia Electrónica Júri”,
p. 74, 2016.
[10] P. M. Gameiro, “Distributed Delay Tolerant Protocol”, 15th International Symposium on Wireless
Communication Systems (ISWCS), 2018, p. 6, ISBN: 9781538650059.
[11] M. C. Chuah, P. Yang, B. D. Davison, and L. Cheng, “Store-and-forward performance in a DTN”, in
IEEE Vehicular Technology Conference, 2006, ISBN: 0780388879. DOI: 10.1109/VETECS.2005.
1543275.
[12] K. Scott and S. Burleigh, “Bundle Protocol Specification”, RFC Editor, RFC 5050, Nov. 2007,
pp. 1–50. [Online]. Available: https://tools.ietf.org/html/rfc5050.
[13] W. Beech, D. Nielsen, and J. Taylor, “AX . 25 Link Access Protocol for Amateur Packet Radio”,
no. July, 1998.
53
[14] R. F. T. Berners-Lee and L. Masinter, “Uniform Resource Identifier (URI): Generic Syntax”, RFC
Editor, RFC 3986, Jan. 2005, pp. 1–61. [Online]. Available: https://www.ietf.org/rfc/rfc3986.
txt.
[15] S. E. Alaoui and B. Ramamurthy, “Routing optimization for DTN-based space networks using a
temporal graph model”, 2016 IEEE International Conference on Communications, ICC 2016, 2016.
DOI: 10.1109/ICC.2016.7510733.
[16] L. Wood, C. Smith, W. M. Eddy, W. Ivancic, and C. Jackson, Taking Saratoga from space-based
ground sensors to ground-based space sensors, 2011. DOI: 10.1109/AERO.2011.5747332. [On-
line]. Available: dx.doi.org/10.1109/AERO.2011.5747332.
[17] L. Wood, W. M. Eddy, W. Ivanèic, J. McKim, and C. Jackson, “Saratoga: A Delay-Tolerant Net-
working convergence layer with efficient link utilization”, in 2007 International Workshop on Satel-
lite and Space Communication, IWSSC’07, 2007, ISBN: 142440939X. DOI: 10.1109/IWSSC.2007.
4409410.
[18] L. Wood, “Saratoga: A Scalable Data Transfer Protocol”, RFC Editor, RFC Draft, Mar. 2012, pp. 1–
53. [Online]. Available: https://tools.ietf.org/html/draft-wood-tsvwg-saratoga-11.
[19] “CCSDS BUNDLE PROTOCOL SPECIFICATION Recommendation for Space Data System Stan-
dards PROTOCOL”, no. September, p. 94, 2015.
[20] T. de Cola, H. Ernst, and M. Marchese, “Performance analysis of CCSDS File Delivery Protocol
and erasure coding techniques in deep space environments”, Computer Networks, 2007, ISSN:
13891286. DOI: 10.1016/j.comnet.2007.04.015.
[21] CCSCS, “Space Communications Protocol Specification (SCPS)—Transport Protocol (SCPS-TP)”,
2006. DOI: CCSDS714.0-B-2. [Online]. Available: http://public.ccsds.org/publications/
archive/714x0b2.pdf.
[22] G. Crowley, C. Swenson, C. Fish, and A. Barjatya, “Dynamic ionosphere cubesat experiment”,
Tech. Rep., Oct. 2009. [Online]. Available: http://digitalcommons.usu.edu/smallsat/2011/
all2011/84/.
[23] Z. S. Gibalina, “Estimation of capabilities of cooperative CubeSat systems based on Alamouti
transmission scheme”, 2018.
[24] R. ESA and S. Division, “Space Engineering - Software ECSS-E-ST-40C”, Tech. Rep., Mar. 2009.
[Online]. Available: http://wwwis.win.tue.nl/2R690/doc/ECSS-E-ST-40C(6March2009).pdf.
[25] E. N. W. G. G. Camarillo, “Peer-to-Peer (P2P) Architecture RFC”, RFC Editor, RFC, 2009, pp. 1–
26. [Online]. Available: https://tools.ietf.org/html/rfc5694.
[26] Y. Song, L. Liu, and J. Chen, “A Temporal Graph Model based Power Aware Routing Algorithm in
Deep-space Networks”, 2017.
[27] D. Evans, O. Chattlain, and R. Vitulli, “The ESA POCKET Housekeeping Telemetry Compression
Algorithm”, no. February, p. 10, 2015.
54
