AAA - InDECT Deliverable D2.2 Revised Version

European Seventh Framework ProgrammeFP7-218086-Collaborative Project

2.2. Description of systems architecture

The INDECT Consortium

AGH — University of Science and Technology, AGH, PolandGdansk University of Technology, GUT, PolandInnoTec DATA GmbH & Co. KG, INNOTEC, GermanyIP Grenoble (Ensimag), INP, FranceMSWiA — General Headquarters of Police (Polish Police), GHP, PolandMoviquity, MOVIQUITY, SpainProducts and Systems of Information Technology, PSI, GermanyPolice Service of Northern Ireland, PSNI, United KingdomPoznan University of Technology, PUT, PolandUniversidad Carlos III de Madrid, UC3M, SpainTechnical University of Sofia, TU-SOFIA, BulgariaUniversity of Wuppertal, BUW, GermanyUniversity of York, UoY, Great BritainTechnical University of Ostrava, VSB, Czech RepublicTechnical University of Kosice, TUKE, SlovakiaX-Art Pro Division G.m.b.H., X-art, AustriaFachhochschule Technikum Wien, FHTW, Austria

©Copyright 2011, the Members of the INDECT Consortium

2.2. Description of systems architecture ©INDECT Consortium — www.indect-project.eu

Document Information

Contract Number 218086

Deliverable Name Description of systems architecture

Deliverable number 2.2.

Editor(s) Grzegorz Sobański, PUT

Author(s) Grzegorz Sobański, PUTPaweł Lubarski, PUTMikołaj Sobczak, PUT

Reviewer(s) Plamen Vichev, Technical University of SofiaZulema Rosborough, Police Service of Northern IrelandNikolai Stoianov, Technical University of Sofia

Dissemination level public

Contractual date ofdelivery

31-10-2010

Delivery date 27-10-2010

Status final

Keywords architecture, integration, system model, communication, nodes

This project is funded under 7th Framework Program

D22 public 3/45


Contents

Document Information 3

Executive Summary 7

Introduction 9

1 Logical structure 111.1 Overview of the system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111.2 Kinds of nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2 Communication Subsystem 132.1 General idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.2 Connecting non-IP segments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.3 Communication between modules . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3 Communication model 153.1 Sample communication cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.1.1 Tracking device reporting its position . . . . . . . . . . . . . . . . . . . . 163.1.2 Client requesting information about tracked device . . . . . . . . . . . . . 16

3.2 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

4 Programming environment 174.1 Sensor nodes (F2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.2 Processing nodes (F3, F4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.3 End-user (desktop) applications (F5, F6) . . . . . . . . . . . . . . . . . . . . . . 17

5 Physical nodes 185.1 UAV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

5.1.1 Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195.1.2 Airframe with propulsion . . . . . . . . . . . . . . . . . . . . . . . . . . . 195.1.3 On-board computers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205.1.4 Stabilized gimbal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225.1.5 General purpose sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225.1.6 Power system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225.1.7 UAV Communication subsystem . . . . . . . . . . . . . . . . . . . . . . . 23

5.2 UAV Ground Segment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235.3 Command room . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265.4 Mobile command node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275.5 Server - responsible for operations of all nodes, possibly distributed. . . . . . . . 285.6 GSM-GPS tracker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305.7 Mobile tracking sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325.8 Multipurpose sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

6 Interfaces and communiaction protocols 336.1 UAV communication with WP2 Server . . . . . . . . . . . . . . . . . . . . . . . . 34

6.1.1 UAV Video stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346.1.2 UAV and payload telemetry . . . . . . . . . . . . . . . . . . . . . . . . . . 346.1.3 UAV mission planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

6.2 Multipurpose station communication with servers . . . . . . . . . . . . . . . . . . 356.3 Mobile Tracking sensor communication with Multi Purpose Sensor . . . . . . . . 356.4 GSM-GPS Tracker communication with WP2 Server . . . . . . . . . . . . . . . . 366.5 WP2 Server communication with WP6 Portal . . . . . . . . . . . . . . . . . . . . 36

7 Integration 367.1 Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

7.1.1 Tracking UAV operations . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

D22 public 5/45


7.1.2 UAV offline operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377.1.3 UAV mission planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377.1.4 License plate recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377.1.5 GSM-GPS tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

7.2 Data types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387.2.1 Position . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387.2.2 Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387.2.3 Speed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387.2.4 Waypoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387.2.5 UAV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397.2.6 UAV Mission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397.2.7 Tracked object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397.2.8 Vehicle data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

7.3 Data provided (out) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407.3.1 Video streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407.3.2 Objects descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407.3.3 Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

8 Future work 41

Conclusions 41

6/45 public D22


Executive SummaryIn this document we describe a proposed logical structure and architecture of the developedsystem.

Deliverable 2.2 is the extension to the Deliverable 2.1 according to the Description of Work.In the Deliverable 2.1 entitled "Preliminary report on proposed logical structure of the systems"we proposed the architecture which we verified on many internal meetings and we decided notto change it in Deliverable 2.2. Furthermore we added the new section on integration and datatypes.

The objective of the system is to support operational activities of police officers and otherpublic services. This goal is to be realized by integration of various devices that can trackmobile objects. Both existing devices and specially designed ones will be used.

The system is physically split into nodes, which communicates between themselves. Wehave logically categorized nodes for easier orientation. Every node can perform one or morefunctions from the list: provide communication, gather data, process data, present data tousers. In Section 1 we give a detailed description of the functions and relations between them.

The second important aspect is the network that connects nodes. We can discriminatethree types of connections. First a high bandwidth cable network with low latencies is usedas a backbone network to connect geographically distant locations and provide a means tocommunicate with remote nodes. Additionally a high bandwidth, long range, point to pointwireless links can be used to extend the backbone to remote locations, where a cable connectionis not available. And last a short range wireless links will be used to connect to mobile nodes.A more detailed description with a suggested technologies can be found in Section 2.

Section 3 provides a model of the communication that rest of the system is built on. De-pending on the use case scenario on of two models can be used. First, a stream-based TCP thatis widely used in the Internet can be used. It is designed to support communication betweentwo nodes that can have a stable network connection for the time they need to communicateand is generally used in case of a huge amounts of data. Examples include a video streamingor camera pictures. The second model is message based. When one node wants to communi-cate with the other, it prepares a message, which is a self-contained pack of data, that othernode can understand and act upon without additional communication. This message can bethen propagated into network in a store and forward mode. This allows a communication witha remote nodes that connects to the network only occasionally. A message can be stored onan access point node and it will be transferred to the mobile node when in enters the rangeof the wireless link of the access point. In the same manner a message from such node can betransferred into the permanent network or even to other mobile nodes. Such message can bedelivered to its destination(s) long after the originating node lost or closed its connection tothe access point.

To develop modules in the most reliable and consistent way, a programming environment’srequirements and recommendations are gathered in Section 4. Processing (server) nodes arerequired to support both Linux and Microsoft Windows environments. End-user applicationsmust support Microsoft Windows environment. C# and .NET platform is recommended.

Section 5 specifies the top-level requirements for all nodes in the system. The followingnodes were introduced: UAV, UAV ground segment, command room, mobile command, serverand various sensors.

UAV is an autonomous system that can accept a mission assignment and realize it onits own without a further supervision. It can on its own be split into submodules, that aredesigned in cooperation. Starting from the airframe that must support all the weight andpower requirements, through on-board computers, to sensors and communication subsystems.UAV ground segment is directly interacting with the UAV, providing a way for the users tocontrol the UAV.

Command room and mobile command are an entry points for the users to maintain andtake advantage of the system. It contains a wall-displays to present a situation overview tocommanding officers and other people in control. Consoles for operators allow to input datainto system, to control all the nodes and to send commands to them. It also allows to submitqueries to the system to control what is presented on the wall-displays.

Server nodes are responsible for persisting all the data that is gathered in the system and

D22 public 7/45


allow to access it offline in the future. Server nodes also assist in processing queries issued fromthe command rooms.

Sensors list includes a mobile tracking devices and multipurpose sensors. Mobile trackingcan be realized based on different localization methods. First solution is to use the GPS receiverto acquire the current positions and to transfer it via a wireless link to the system. Short rangededicated solutions or a GPRS network can be used for this communication. Second solutionis to use only a short range wireless, mesh network for both localization and communicationwith the system.

A multipurpose sensor can be additionally equipped with more devices to acquire a widespectrum of data. Examples include video-cameras to detect objects of interest and providea stream of video, temperature sensors, bio- and chemical-sensors to detect contamination andpossibly many other solutions.

Section 7 describes a technical details of the data that is accepted as the input by nodesand data that is produced and available as the output.

The document is concluded in Section 8 with a summary of future work to be done in taskT2.1.

8/45 public D22


IntroductionTask 2.1 in work package WP2 concerns the high level overview of system being developed in thiswork package. The main goal of WP2 is to provide police officers with a prototype system foracquiring, processing and displaying information about observation of various mobile objects.To fulfill those requirements three interrelated modules of the system must be constructed.

First a multi-mode system for positioning and tracking movable objects is needed. Specialdevices and spatial algorithms are needed to determine a precise location of objects. Second isa Unmanned Aerial Vehicle as a autonomous patrol and observation platform. The last elementis a wireless network infrastructure needed to provide those two former goals.

All of these, connected together in various ways, form an integrated network-centric system.This system, which is one of the core subsystems of the Indect Project, is the eyes and theears of the entire project. It supports other Work Packages with different data which canbe processed further. Thats is why WP2 subsystem fits exactly into the idea of ’Intelligentinformation system supporting observation, searching and detection for security of citizens inurban environment’.

D22 public 9/45


1. Logical structure1.1. Overview of the system

The objective of the system is to support operational activities of police officers and other publicservices with a techniques and tools for the observation of various mobile objects. This goal isto be realized by integration of wide spectrum of both current available and specially designeddevices that can track mobile objects thus creating a heterogenous network-centric system withvarious attached nodes.

Figure 1: System overview

The INDECT system will consist of many modules, that realize many different functions.Because of that, for the effective realization of stated goals the system is being designed asa distributed, event-driven system. Every node of the system can be seen as an independentcomponent that realizes its function and communicates to other nodes information about oc-curred events. This approach will ensure that the system is modular, open and scalable. Thusfurther development of the system or addition of new nodes can be done fairly easy. The en-sures that the entire system is robust and it improves information sharing. Information sharingenhances its quality and helps build situation awareness for police officers. By sharing situa-tion awareness with other end-users they can collaborate more easily and this increases workeffectiveness.

1.2. Kinds of nodesFrom an infrastructure point of view nodes in the system can be divided into groups based ontheir functions:

F1 Nodes that ensure communication capabilities of the system (routing nodes).

F2 Nodes that gather information from its sensors.

D22 public 11/45


F3 Nodes that process information, including: detection of events occurrence, detecting con-nections between events (possibly events from different sources), etc.

F4 Nodes that answer to end-user specified tasks and queries, on-demand gather informationfrom other nodes, prepare data to be displayed to the user, etc.

F5 Nodes that display information about events to the end-user (passive client).

F6 Nodes that allow the end-user to query the system for specific information (active client).Of course every node of the system can at the same time perform more then one function. Forexample one node can gather information and at the same time (pre)process them, lookingfor events and so on. Some functions will be combined more often then others, most commoncombinations include:

• gathering and processing data (F2, F3),

• processing data and processing end-user tasks (F3, F4), those nodes can be viewed as“servers” in a typical client-server design,

• displaying events and an entry point for end-user (F5, F6), those nodes can be viewed as“clients” in a typical client-server design.

Every node needs a way to communicate with other nodes, so basically it can be said thatevery node performs F1, but we will say that a node performs F1 if it acts as a communication(routing) node for other nodes.

From a high level point of view, we can distinguish following nodes:• communication nodes that compose the mesh network and communication infrastructure,

• detectors:

– cameras,– wireless detectors of tracking devices,– and others,

• servers, that gather data from detectors and share them with clients (end-users),

• end-users’ client applications that acts as an entry point into the system.

Sensors node (F2)

Client node(F5, F6)

Processing node

(F3, F4)

Sensors node

(F2)

Processing

node (F3, F4)...

...

...

Events

Query/

Control

Data

...

Figure 2: Node cooperation structure

Figure 2 shows a general flow of data between different kinds of nodes. It is important toremember that many logical functions (ie. F2 and F3) can be performed on one physical node.

12/45 public D22


2. Communication Subsystem2.1. General idea

Designing a distributed system emphasizes the network part of the system which is the core ofevery network-centric system. It is crucial for the efficient work that every node in the systemcan possibly communicate with any other node in the system.

Existing networks

Backbone

network

Wireless

backbone

Wireless

and mesh

networks

Figure 3: Network

Figure 3 presents a overview of the network design. It is obvious that the designed networkmust be heterogeneous to be able to connect different nodes that are scattered in different areasand with different communication capabilities. We can divide available connections into groups:

• a fast and efficient backbone - a cable network with a high bandwidth (10Mbps+, possibly1Gbps between most important node), it can be both dedicated network where possible, orusing existing Internet or other dedicated infrastructure with proper security mechanisms,

• a wireless backbone - a long range, high bandwidth, mainly point-to-point connection toremote stations (potentially using WiMAX technology),

• a short-range wireless links - show range, low or high bandwidth link between nearbynodes.

The goal of backbone connections is to build a stable network that connect all static nodesinto one system. Then those static nodes can act as a distribution points for other nodes usingshort-range wireless links. Because the cost of building and operating backbone connectionscan be high it is natural to put one such node in an area and use short-range and cheaper linksto connect nearby nodes to the network.

Based on the connection we can divide nodes into four classes:

• backbone nodes that make up the backbone network (providing only F1),

• nodes connected directly to the backbone, they are always (excluding failures) connectedto the network,

D22 public 13/45


• static (non-mobile) nodes connected with a short range links to the backbone nodes orother static nodes; they will create a mesh network that can cover a wide area with onlya cheap (money cost and energy consumption), short range links,

• mobile nodes that connect with a short range links to other nodes.

Mobile nodes can be further divided into two classes:

• mobile nodes that stay connected to the network for a long time,

• mobile nodes that initiate a short connections to send data, and stay disconnected mostof the time.

On top of those physical connections an IP network will be built. This will allow use to usestandard and well-known algorithms and protocols. Every node can be then assigned an uniqueIP address, and every node can then initiate TCP or UDP connection to every other nodeaccessed with an IP.

2.2. Connecting non-IP segmentsUnfortunately not every device can be made to support IP. There will be nodes without thecapability of supporting IP connections, especially the ones that stay disconnected, and connectsto the network to transfer and receive short bursts of data.

For such nodes access points (AP) must be built to allow for data transfers to and fromthem. In such case, the node which will act as the AP will have IP address, and will be availablefor other nodes in network. If node wants to send data to disconnected-node it will contactthe AP and leave a message that will be transferred to the destination node, when it will bepossible.

There may be many AP node for the same disconnected modules. In such case they shouldsynchronize between themselves message queues, so that it will not be relevant to which APthe node connect, it should always get the awaiting messages.

Examples of such links includes:

• hand-crafted short and long-range radio communication,

• short-range bluetooth communication,

• SMS communication in cellular networks.

Surely such connections will be used in INDECT system.

2.3. Communication between modulesCommunication between modules in Work Package 2 susbsytem will be realized according tothe model found in figure 4.

Servers are the heart of a network-centric system and they are connected to the commu-nication backbone - fiber, cable, wireless broadband, narrowband or all together. Servers canexchange information between each other and communicate with other nodes directly or indi-rectly in order to gather all the requested information that will be saved in the database. Thisis the only part of the system that has direct access to the database. For example GPS/GSMTrackers communicate with the servers directly through GSM network and UAV communicateswith the servers through a UAV Ground Segment or through Multipurpose Sensors. Serversalso interpret requests from the command room, wheather it is a WP2 application, Indect Por-tal or any other authorized application. There it is presented to the end-users. Because this isa network-centric system the information does not always have to go to the server and to thedatabase. The UAV could ask a Multipurpose Sensor what are the weather conditions in itsarea, like wind speed, in order to adjust flight trajectory or speed. The Mobile Tracking Sensorcould be placed under a suspects car, while it is moving through the city it the sensor willcommunicate with Multipurpose Sensor and it will forward the information about the suspectsposition of to the UAV. This information will be also sent directly to the Command Room,there the police officers could aks for a video stream from the UAV. This stream would be sent

14/45 public D22


WP2 Server Multipurpose Sensor

GPS/GSM Tracker

Mobile Tracking

Sensor

Command Room

UAV

UAV Ground Segment

Database

Indect Portal

Figure 4: Communication between modules in WP2 subsystem

through UAV Ground Segment or if it is out of range through the nearest Multipurpose Sensor,the servers and straight to the Command Room. This is the idea of a network-centric systemwhere everything is integrated and this approach is being used in the WP2 subsystem.

3. Communication modelCommunication between nodes can be generally realized with two models: one which we cancall stream-based, and second is message driven.

In a stream-based model nodes opens a connection (ie. TCP) between themselves and theyexchange a long stream of data. This type of systems are common in Internet. Client opensa connection to the server, and they exchange a two-way stream of data. It works well whenthe network is stable, and the connection can be kept open for a whole time of operation: clientasking for data, server preparing the response, and server transfers the response. The longer ittakes for the server to prepare the response, and weaker the stability of the network is, the moreproblems appears in this model. Most often, when the connection is broken, a whole proceduremust be repeated, including collection of the data by the server.

In a message driven system a whole communication between nodes is based on messages.Nodes do not communicate with a stream of data, but with messages. Every client request,server response, etc is wrapped in an envelope and seen as one message. In this model the sameseries of operations as before can be seen as this:

• client prepares a message describing its request,

• it sends the message to the server,

• when the server receives the message it process the request, and prepares the answer ina form of another message,

• then send the message to the requester,

• client reads the answer to its query from the message.

D22 public 15/45


It is important to note, that this whole operation is asynchronous. The transfer of messages canspan among many connections between those two nodes, if nodes queue messages this model isresistant to temporary disconnections, nodes unavailability, etc. Moreover messages does notneed to be transferred directly between two interested nodes, they can be proxied, queued andforwarded by intermediate nodes.

This model is especially useful in distributed systems, where nodes can be disconnected forlong time or available only through very heterogeneous network. When we consider nodes thatconnect to the network only for short transfers of data, we can see that stream-based modelbehaves very poorly then. In case of message based system, AP can queue messages that areintended for such disconnected node, and hand them to the node when it connects. The sameis true for messages originating from such node - they can be queued by AP and forwarded tothe recipient long after the sender has disconnected.

In a message based system we define:

Message bus an abstract place that is a central point of messages exchange. Creators of themessage put them into one of the message buses. It is the responsible for delivering thosemessages to proper recipients. A message bus is responsible for routing the message indistributed system and for securing access to the messages. It can optionally persist themessages so they are never lost.

Subscription node interested in receiving specific information can request the message bus tosend him messages that meet his expectations. For example a server node can requestall information about positions of tracked devices. When a tracked device put a messageabout its position into message bus it will be delivered to this server node.

Message modes a message in a bus can be processed in few distinct ways: it can be deliveredto only one recipient or to all those that expect it; it can be queued or discarded whenthere is no subscription for this message; it can be deleted after delivery or kept (possiblyfor limited amount of time) for future subscriptions.

3.1. Sample communication cases3.1.1. Tracking device reporting its position

1. Tracking device gets its position from a sensor, ie. GPS (F2).

2. The device decides it should report the position to the system (F3).

3. It builds and queues a new message.

4. When it reaches a point where communication is possible it relays its queued messages tothe AP.

5. When AP gets the messages it forwards it to destination node (F1).

(a) optionally AP can put it in local message bus (or queue) and wait for other node toactively ask for this message.

6. A processing node (server) gets the message and acts accordingly (persist it in database,inform clients, etc) (F3).

3.1.2. Client requesting information about tracked device1. Client application sends a message to a server subscribing for information about a tracked

device (F5, F6).

2. The server can then send his own subscriptions to other nodes to be able to fullify client’srequest (F4).

3. Every time when the server receives new piece of information about the tracked device itwill construct and send a message to the client (F4).

16/45 public D22


4. Client receives this information and display it to the end-user, for example on a map (F5,F6).

3.2. SecurityBecause the nodes are transferring between themselves sensitive data, all network traffic in thesystem must be encrypted. Known, tested and widely used encryption protocols should be usedto minimize the possibility of unauthorized persons accessing the data.

The second problem is authentication. As many of the nodes are operating remotely, systemmust be immune to an attempt of some nodes impersonating others. A PKI infrastructureshould be used to uniquely identify nodes in the system.

4. Programming environmentBecause of the heterogeneous nature of the system, a large number of programming languagesand environments will be used. Here we gather a set of rules that should be followed.

4.1. Sensor nodes (F2)Environment highly depends on the hardware used.

4.2. Processing nodes (F3, F4)• Must work both in MS Windows and Linux operating systems.

– In specific and proven situations they may work in only one of them.

• Should be developed in .NET, preferably C# language.

– Mono project must be used to launch them on Linux and other *nix systems.

• If those applications must run on embedded systems, they may be written in other envi-ronment, specific to that environment (ie. Ansi C, C++).

• If performance is critical, and it is proven that unmanaged (C++) version will run muchfaster, an unmanaged language can be used.

Managed environment provides a runtime environment for an application that performsmemory management for application. Modern environments includes .NET, Java, Python,Ruby and others.

4.3. End-user (desktop) applications (F5, F6)

• Must work in MS Windows environment (XP or newer).

• It is convenient if it would work in other operating systems (Linux, Mac), but not required.

• Should be developed in .NET, preferably C# language.

D22 public 17/45


5. Physical nodesWithin INDECT project following physical nodes will be designed, prototyped and tested:

1. Unmanned Aerial Vehicle- for covert surveillance.

2. UAV Ground Segment - a base for UAV operations and control.

3. Command room - a center for commanding officer controlling, gathering and displayinginformation from all nodes from WP2.

4. Mobile command node - provides similar functions as a command room.

5. Server - responsible for operations of all nodes, possibly distributed.

6. GSM-GPS Tracker - a device able to track a movement of a vehicle and reporting it’sposition to system.

7. Mobile tracking sensor - a device moving in a mesh network.

8. Multipurpose sensor - a node connected to the system with a capabilities of connectingdifferent sensors and other devices, build a mesh network.

All nodes are described here in more details, while figure 5 shows how those nodes accomodatein a overview shown in 2.1.

Existing networks

(5) Server

(3) Command

room(s)

(1) UAV

(8) Multipurpose

nodes

(2) UAV Ground

Segment

(4) Mobile

command node(s)

(7) Mobile tracking

sensors

(GSM)

(6) GSM-GPS

Tracker

(GPS)

Figure 5: Nodes overview

5.1. UAVRealizes: F2, possibly F3.

18/45 public D22


5.1.1. ComponentsThe main functional components of UAV are:

• Airframe with propulsion.

• On-board computers with auto-pilot.

• Stabilized gimbal with sensors (visual sensors).

• General purpose sensors (eg. chemical).

• Communication system.

For intra-component relations please refer to figure 6.UAV requires ground segment to operate (see 5.2):

• Ground control system. Is used to plan missions for UAV and to control on-board sensors.

• Ground communication system. Is necessary to gather information from UAV sensors.

• Start and landing support system.

Design and implementation of ground segment is as important as UAV itself.

Flight control computer

CommunicationSystem

Flight control surfaces Engine

Ultrasound sensor

Height

Speed

Angles

Pressuresensor

HeightAirspeed

Angle of attack

Mission planUAV status

Communication status

Camera

Stabilized gimbal

AnglesZoom

Image processingComputer

Image

Zoom

Image features

Power management

Batteries

Power consumptionRemaining

power

Power consumption

Navigationcomputer

Mission computer

GPS

Position WaypointsPosition

Settings

General purpose sensor

Figure 6: UAV’s components

5.1.2. Airframe with propulsionThis component consists of:

• propeller or rotor

• engine (electric or petrol)

D22 public 19/45


• engine power source (power cells or petrol)

• electric generator for petrol engine

• airframe with actuator engines

Main factors which influence design of the airframe is maximum speed and endurance.

UTWORZONY PRZEZ PROGRAM EDUKACYJNY FIRMY AUTODESK


UTW

OR

ZONY

PRZ

EZ P

ROG

RAM

ED

UKAC

YJN

Y FI

RM

Y AU

TOD

ESK U

TWO

RZON

Y PRZEZ PRO

GRAM

EDUKACYJNY FIRM

Y AUTO

DESK



UTW

OR

ZONY

PRZ

EZ P

ROG

RAM

ED

UKAC

YJN

Y FI

RM

Y AU

TOD

ESK U

TWO

RZON

Y PRZEZ PRO

GRAM

EDUKACYJNY FIRM

Y AUTO

DESK

Figure 7: Technical drawings of proposed airframe

5.1.3. On-board computersOn-board computers have to perform these operations to maintain UAV autonomous flight:

1. engine and actuator control,

2. flight control,

3. navigation,

20/45 public D22


4. maintain mission status and way-points.

On-board computers additionally can:

• Process data from sensors (including advanced visual algorithms).

• Archive data from sensors (in a limited manner).

• Compress data before sending to ground segment.

Burzyk UAV is equipped with 3 computers:Flight control computer. It is a real-time computer, built on the ARM architecture. It is

responsible for engine and actuator control, flight control and sensor fusion (combining datafrom different sensors in order to archieve a consistent state). It will be designed and built byPUT.

Figure 8: Visualizations of the proposed airframe

D22 public 21/45


Figure 9: Kinematic diagram of the gimbals

• Performance computer is an x86 architecture mobile module. It is responsible for naviga-tion, mission, and processing visual data.

• Video encoding module is a computer designed to compress video stream.

On-board computers communicate with each other with CAN bus and Ethernet. The firstprotocol guarantees real-time responce, the second gives very high bandwidth. Both communi-cation technologies are used allowing taking the best form each technology.

The architecture is based on message passing between computers and sensors. This approachis very open, it allows easy replacement of modules, hardware upgrade or adding modules formthird parties.

5.1.4. Stabilized gimbalStabilized gimbal is an important part of most UAV’s. It is responsible for moving visual sensorsand stabilizing the image (reducing vibrations).

Main architectural trade off for stabilized gimbal is the size of carried sensors anddegrees offreedom of the gimbal, the more degrees the smaller the sensor can be.

Sensor range for stabilized gimbals include:

• optical cameras,

• night cameras (close infra red),

• thermo-vision cameras (infra red),

• laser rangefinders.

Optical and thermo-vision gimbals will be built for the Burzyk UAV. However any manufacturercan build a gimbal as long as mount and communication will be compatible.

Designed gimbals have 2 degrees of freedom: the roll and pitch axes. It is shown on thefigure 9. The weight of the gimbal is 650g.

5.1.5. General purpose sensorsUAV can have general purpose sensor mounts enabling installation of sensors designed byexternal manufacturers. This type of system have to have published interface (hardware andsoftware).

5.1.6. Power systemUAV is equipped with Power management center, which measures power consumption of allsystems and available power. It can turn off any system or device in UAV allowing intelligentpower usage. It is also very important in case of malfunction - components can be shut downsaving energy for emergency action. It allows to calculate time remaining of UAV operation (indifferent modes).

22/45 public D22


5.1.7. UAV Communication subsystemIn definition from section 1.2 UAV can be a (F1, F2, F3, F4) node.

Communication system in UAV has 3 main links:UAV - Ground segment UAV gathers large amounts of data - especially visual data. This

link requires broadband communication. Of course UAV can conduct a mission withoutcommunication, but its uncommon in civilian applications. Ground segment sends thedata to the network-centric system.

UAV - UAV communication (F1) can be done in 2 ways - retransmission of full broadbanddata or retransmission of narrow band. At the moment we asume, that there will be nobroadband retransmission.

UAV - ground nodes communication (F1) allows UAV to extend its sensing capabilities. Itcan be also used to extend communication capabilities of the system. UAV broadbanddata can be received by authorized ground users. Ground sensors can send narrow datato UAV.

Ground station

Network

UAV

Ground node

UAV

Narrow transmission

Broadband transmission

Ground node

Figure 10: UAV’s communication system

5.2. UAV Ground SegmentRealizes: F5, F6

D22 public 23/45


UAVCommunication

System

NavigationComputer

Mission planningSystem

Operator Console

Take off system

Landingsystem

Mission planWaypoints

Image Data Mission validation

and preprocessing

Current position

Command Room

Data important for mission planning

Figure 11: Architecture of the ground segement

Ground segment is a part of the system which allows UAV’s to operate from ground.It might be mobile (mounted on a vehicle or carried by personnel).These are main duties of the ground segment:1. Planning UAV missions or transmitting missions from Command Room to UAV.

2. Communicating with UAV - ground segment is the main point of communication.

3. Controlling UAVs on missions:

(a) getting live feed of video and telemetry(b) display mission parameters(c) control mission parameters

4. Aiding UAV start and landing - most UAV’s need specialised take off and landing equip-ment.

Ground segment might be mobile or stationary. Stationary ground segment is integrated withcommand room and is a software module.

Mobile version of ground segment consists of:• Antenna tracker with communication system. Proposed design of a tracker system is

shown on figure 12.

• Rugged computer with operator software.

• Visual landing system - an innovative system which uses images from camera to trackUAV and landing site.

This equipment is carried in a specially designed backpack. Allowing 2 people to carry UAVand ground segment. Antenna tracker and visual landing system will be developed by WP2.

Ground segment also has to be able to do maintenance duties including:1. Maintenance of vehicles - ie. cleaning, refilling, replacing parts.

2. Upgrading software and hardware in vehicles.This is not required during standard operation.

24/45 public D22


Figure 12: Visualization of proposed antenna tracker

D22 public 25/45


Figure 13: General Idea of Cammand Room

5.3. Command roomRealizes: F5, F6

A center for commanding officer controlling, gathering and displaying information from allnodes from WP2. It must provide:

1. Operator console(s)

2. Wall-display for situation overview

The general idea of Command Room is presented at the Figure 13. The room contains a fewoperators’ consoles, multiple wall displays and other printing/communication devices.

Operator console should provide:

1. Insight into system operations:

(a) general status of the system (operational, problems, down)(b) status of the nodes including:

i. state of the servers, performance tracking,

26/45 public D22


ii. state of the disconnected nodes - last seen, expected next communication.(c) list of requested processing.

2. Controlling nodes:

(a) a module for every kind of node should be provided allowing:i. controlling behavior of that nodes, specific to that node,ii. sending orders to remote nodes,iii. gathering info from nodes.

(b) registering new nodes to the system,(c) deregistering removed nodes (decommissioned nodes, compromised nodes, etc)

3. Display of tactical situation:

(a) map with a position and state of all nodes,i. possible with a predicted now and future positions.

(b) alerts about events discovered in the system (examples includes but is not restrictedto: movement of some objects, leaving/entering areas, spotting objects),

(c) summary of conducted operations.

Each operator is equipped with advanced application for command support which enables toinspect the tactical situation. This software is able to be cutomized to fit each operator’s needs.It is possible to create own working enviroment, which consist all necessary information in realtime. It enables also to combine different layers and objects at one view. It is fully integratedwith whole network-centric system presented in this deliverable. This enables to collect andanalyze larche amount of data form different types of nodes. What is more, two operators canwork together on one case and interact with each other.

User

Authentication

Nodes manager

GUIVideo display

2D display 3D display

Objects manager

Authorization & Security

Message bus

Local data providers Video providers Communication with server

Figure 14: Architecture of desktop application

A system of authorization for users should support different levels of access. For examplea user and password access should be more restricted then access for the same user authorizedwith a smart card. Generally a smart card authentication should be preferred.

5.4. Mobile command nodeRealizes: F6, rarely F5

Mobile command node should provide as much of abilities of command room as possible.However it should be optimized for displaying information and controlling specific nodes asrequired by the specific operator (or operation). Display of tactical situation should be reducedto display only relevant information to increase readability.

Overview of the system can be skipped on mobile command node if it is ncd ..ot convenient.

D22 public 27/45


5.5. Server - responsible for operations of all nodes, possibly distributed.Realizes: F3, F4

As a server we define a infrastructure required for the server software to work with a highperformance and high availability.

Server should:

1. Be able to handle CPU usage of all processing software needed to gather and processinformation from all nodes.

2. Not have one point of failure - CPUs, disks, power, network connections should be redun-dant.

3. Be secure (specific requirements to be defined later).

Server software should provide the capabilities:

1. Accept connections and messages from all nodes in the system.

2. Deliver those messages to modules responsible for nodes, that should:

(a) store all/most of the received information,(b) detect event occurrences as requested from end-users,(c) report events in form of alerts that will be sent to users.

3. Answer to client queries for current state of the nodes, last received information, historicinformation.

Database

managerNodes manager Video storage

Events analyze

GIS Data

storage

and provider

Objects tracking

Authorization & Security

Message bus

... SMS communication module TCP communication module

Figure 15: Server’s architecture

WP2 server is built around Return architecture developed on PUT. Return is an applicationserver written using .NET framework and running on different platforms — Linux andWindows.Usage od Return allows for modularity and easy deployment of server. Based on the frameworkarchitecture, the WP2’s server is divided into modules. Every module is responsible for its partof functionality. Modules exchange messages when in need of data and processing available inother modules.

Return Server provides all the necessary infrastructure for the message passing, securitymechanisms and communication — based on messages — with other nodes. Additionally othermodules can provide its own methods of communication if needed. Functions in server are splitinto following modules.

The center system of Return is LightBus — a high performance message bus that is respon-sible for providing communication between modules. All communication between modules isbased on messages and publish–subscribe model. Every module is running independently ofothers, using its own pool of threads and processes. It subscribes to the message bus listeningfor messages that it is interested in. When a module has new data or results of processingit then publishes this data as a message to the message bus. Message bus then delivers thismessage to all modules that have subscribed for this information.

28/45 public D22


Light Bus

Message Bus

GIS Provider Objects Provider Video Manager

Webservice Network

Figure 16: Message bus

This model of communication allows for much independence between modules. This givestwo most important advantages. Firstly, no thread is running code from different modules. Thisgreatly increases reliability of modules, reduces problems with concurrent access to variablesand data, and allows for easy testing. Secondly, modules are decoupled, because they do notcommunicate directly, but with a help from the message bus. Modules publish data theyproduce and subscribe to data they need. Publishers and subscribers do not have to know theothers. Figure 16 presents this idea in graphic form.

Functionality of the server is split into modules:

1. GisProviderGisProvider is a multipurpose module, responsible for the access to the geography database.It is a utility module that provides GIS data for other modules and for the command centerto display maps to the user.

2. ObjectsProviderObjectsProider is an extendable, base module for modules that are responsible for pro-viding, processing and managing data about real-world objects. In Indect list of thoseobjects include, but is not restricted to: UAVs, tracking sensors, blue force units, trackedobjects.This module is responsible for persisting data about objects in the database, and provid-ing access to them in the future.Objects are identified by the URI string. URI is a hierarchical structure, where everyobject is uniquely identified and placed in a tree structure of all objects. Hierarchy levelsand object identifiers are string based, and delimited with a “/” character.

3. IndectWebServiceData gathered and processed by the WP2 systems must be available outside of WP2 —mainly in the WP6 portal. Communication with the portal will be available througha SOAP WebService. Implementation of this webservice will be in a return moduleIndectWebService. Contract of the interface is in the discussion with WP6.

4. Video StorageVideo streamed from UAV and possibly from other nodes is stored on dis on the server.This module provides access to it for offline viewing by authorized users. It allows foraccessing videos based on date and source and streams the video for user application inCommand Center, and possibly WP6 portal.

D22 public 29/45


GPRS transmission

EDGE transmission

communication via SMS

ExistingGSM networks

GSM-GPSTracker

GSM-GPSTracker

GSM-GPSTracker

Server

Figure 17: GSM-GPS tracker communication system

5. Nodes managersEvery tracking node must be managed by the server. A list of tracking nodes with itsidentifications and certificates must be stored and made available to authorize nodes toprovide information. Especially revocation list of certificates must be managed.Additionally configuration for all Multi sensor nodes must be accessible for users andchanges to it dispatched to nodes when possible.

5.6. GSM-GPS trackerRealizes: F2

A small device equipped with a GPS receiver to track a position of the object it is attachedto.

The device uses a high-performance GPS module along with a passive, omni-directional,state-of-the-art antenna which is immune to many types of interferences, including multi-pathsignals. The module is capable of working in an autonomous low-power mode, i.e. to periodicallywake up from sleep mode to report a position.

The GSM communication module is integrated with an application processor which executesall the device code. This allows for a higher integration and makes it possible to fully control thecommunication hardware and software stack. The integrated antenna is a penta-band designsupporting up to 5 GSM frequency bands so that a module upgrade to 3G technology is possiblein the future. Currently however, GPRS/EDGE transmission is sufficient as far as data speed isconcerned and has an important advantage over 3G — lower power consumption. The trackercan optionally use SMS transmission instead of GPRS/EDGE when such connection cannot berealized. The communication architecture of the GSM-GPS tracker is shown in Figure 17.

As the aim is to have the GSM-GPS tracker operational for as long as possible, the devicecan work in different modes to manage energy consumption, e.g.:

1. Stay on-line with continuous or periodic data transmission over GPRS/EDGE. Objectpositions can be saved in internal persistent memory for periodic batch synchronizationwith the server. Positions are not sent while the object is not moving in order to savepower and mobile data usage.

2. Stay on-line only when the position of the object is outside a certain area — so calledanchor mode. This mode can further improve operation time.

30/45 public D22


Figure 18: GSM-GPS tracker architecture

Server

Mobile trackingsensor


Multipurposenode


Multipurposenode

Short-range communication

Figure 19: Mobile tracking sensor communication system

3. Disable radio and connect only on events:

(a) periodically, to report a position, e.g. every 6 hours,

(b) when movement of object occurs and exceeds some pre-programmed threshold.

4. Stay in off mode for a pre-programmed time. This allows to reach a standby time in termsof a few months.

5. Send detailed GPS satellites signal information only in setup mode which is used whenattaching the tracker to the object. In other modes only the most important data is sentautomatically (e.g. current location) to lower the mobile data usage. Detailed data maybe sent on request.

The modes can be changed remotely at any time. All other configuration options, e.g. whetherto allow data transfer while roaming, are also accessible remotely.

The device is equipped with an integrated power management module which is responsiblefor charging the battery, monitoring its state and assuring that the GSM module (the mostpower consuming element in the device design) is powered properly, even in low temperatureconditions when battery has worse performance. The information concerning battery state isperiodically transferred to the server so that it can easily be monitored by an operator.

Diagram with architecture of the GSM-GPS tracker is presented in Figure 18.

D22 public 31/45


Figure 20: Mobile tracking sensor architecture

5.7. Mobile tracking sensorRealizes: F2A mobile tracking sensor is a small device used to track objects. The device has to be designedin a way that ensures small size and simultaneously operational time as long as possible withoutbattery charging.

Unlike GSM-GPS tracker the mobile tracking sensor uses a mesh network built with multi-purpose nodes (see Section 5.8) for both communication with the system and position estima-tion.

The device has radio transceiver for transmitting data and receiving commands. It usesa short-range communication subsystem to send and receive data from nodes of the meshnetwork or with other mobile nodes. The communication scheme is presented in Figure 19.

Position of the mobile tracking sensor is obtained by analysing information which masterstations’ area it is entering and leaving. Optionally the device can be equipped with a GPSreceiver to get much more precise location. In case of using GPS receiver the device may alsohave accelerometer to turn off the GPS when it is not moving and this way minimize powerconsumption and extend working time.

Architecture diagram of the mobile tracker sensor is shown in Figure 20.

5.8. Multipurpose sensorRealizes: F1, F2, and possibly others with addons.

A general purpose node. It is equipped with a short range communication subsystem toconnect to other such nodes. They are placed around the city. One of the aims of suchmultipurpose node is traffic monitoring and recognizing of every license plate of passing vehicles.It should also cooperate with neighbor nodes via communication backbone. In this way a set ofmultipurpose nodes create a mesh network, where all nodes can be accessed with a mutli-hopmessage passing. The scheme of such network can be seen at Figure 19

Every node maintains the list of vehicles that are wanted, for example stolen ones. Theplate recognition system operates in two modes ALL and WANTED. In ALL mode it reportsappearance of every vehicle, whereas in WANTED mode it reports only appearances of thevehicles from the list.

Additionally other subsystems can be added to the node, including:

1. Long-range communication - a node can be connected to the backbone, with a broadbandwireless connection or a broadband wired connection, acting in this way as a entry pointfor the system to the mesh network. More then one such entry point is possible, andencouraged for reliability of the mesh network.

2. Video-camera to observe and detect different kinds of objects.

3. Sound detectors.

4. Meteo stations.

32/45 public D22


CO

C

Image

computer

Node

motherboard

Communication

module

CT

Figure 21: Multipurpose Sensor

The scheme of the node is presented at the Figure 21The core unit of the Multipurpose Sensor consists of:

Node Motherboard responsible for the power management and package routing of the datapackets within the node. It can be extended by different types of sensors.

Image Computer responsible for image processing tasks performed by the sensor

Communication module a set of wireless devices that enable communication with othernodes in the network, as well as with the moving objects

6. Interfaces and communiaction protocolsIn order to ensure proper and reliable communication in Work Package 2 a series of inter-faces and communication protocols has been developed. The following section covers interfacesbetween devices in WP2 subsystem:

UAV↔WP2Server

Network: COFDM Data LinkSoftware: Interface SI.Log2, Interface SI.Video

Multi Purpose Sensor↔WP2Server

Network: ethernet/IPSoftware: Interface SI.1

Mobile Tracking Sensor↔Multi Purpose Sensor

Network: Short range radio

D22 public 33/45


Software: Interface SI.2

GPS/GSMTracking device↔WP2Server

Network: GPRS/EDGE/HSPA

Software: Interface SI.3

WP2Server↔WP6Portal

Network: Ethernet/IP

Software: Interface SI.WS1 SOAP WebService

Since these protocols will be used in devices for the police force only general information willbe given. The exact protocols are confidential in order to ensure safety of the solution and willbe given to authorized personnel on request. All communication is encrypted.

6.1. UAV communication with WP2 Server

6.1.1. UAV Video stream

Video signal from camera is compressed by the video encoder module. A commercial moduleis being used it can compress into MJPEG and MPEG-4 streams. MJPEG compression is lessbandwidth efficient but it is robust and does not generate delay. Because UAV in flight isvery dynamic and the image is changing very fast most of benefits of the MPEG-4 compressionare lost. Therefore the compression system is focused on MJPEG. The delays of the MJPEGstream are around 50 ms. The resolution of the camera signal is 640x480 or 704×480 dependingon the camera.

6.1.2. UAV and payload telemetry

The telemetry protocol is a connectionless protocol based on UDP datagrams. Structure of theprotocol is based on messages. At this moment messages of such types are planned:

Position and state this message is used to get position and state of the UAV including ori-entation and speed.

Control surfaces this message carries the state of the control surfaces of the UAV includingengine speed, flaps, etc.

Gimbal status this message defines orientation and mode of operation of the gimbal.

Gimbal control basic this message sets basic control of movement of the gimbal.

Gimbal control visual - allows control of the gimbal in visual mode.

Gimbal visual mode status - gets status messages from visual tracking system.

Gimbal control in object tracking mode - control messages sent to gimbal in mobile ob-ject tracking mode.

Energy consumption this message is used to get curent and total consuption of the UAV.

Information from these messages is available for UAV operator.

34/45 public D22


6.1.3. UAV mission planningMission is defined by a tree-like structure in which task is either a concrete task (e.g., flying toa waypoint) or a composition of subtasks. Mission plan messages are defined recursively: onemessage can contain other messages.

TaskSequence This message shall be used to define a task that consists of subtasks that willbe executed in the given order.

TaskSet this message shall be used to define a task that consists of subtasks that will becompleted in the order specified by a planner. The order can be chosen to i.e., minimizetraveled distance.

FiniteStateMachineTask this message shall be used to define a task that consists of subtasksthat will be completed in the order that depends on the runtime events, possibly multipletimes.

WaypointTask this message shall be used to define a task of flying to a waypoint.

SegmentTask this message shall be used to define a task of flying along a segment.

LoiterTask this message shall be used to define a task of circular loiter at given position.

PolygonCoverTask this message shall be used to define a task of flying over a polygon withthe intention of covering it with a visual sensor.

Position this message shall be used to define a geographical position.

Altitude this message shall be used to define a required altitude.

Speed this message shall be used to define a required speed.

StateTransition this message shall be used to define a transition in the FiniteStateMachine-Task

6.2. Multipurpose station communication with serversThis protocol is used by multipurpose sensors for communication with each other. The messagescan contain some steering information for the nodes, as well as the data collected by the system.

Standard header frame this frame encapsulates every other frame sent or received by mul-tipurpose nodes.

Flash frame this frame enables to reprogram the node over the air.

Mobile tracking this frame is used for transmission of data from devices used for vehicletracking.

Plate recognition frame frame used for the communication with plate recognition module.Can be used for reporting appearances of the vehicles, or can carry commands like “addplate to wanted list”.

6.3. Mobile Tracking sensor communication with Multi Purpose SensorThis protocol is used for sending messages to and receiving messages from Mobile Trackingsensor via short-range radio.

Beacon this frame is sent periodically by master stations to inform tracker in range of whichmaster stations it is

Request this frame allows tracker to send information (without the proper request mobiletracking sensor does not emit any frames)

Status frame which encapsulates information sent by tracker

D22 public 35/45


6.4. GSM-GPS Tracker communication with WP2 ServerThis protocol is used to communicate with GSM-GPS Tracker via existing GSM networks(GPRS/EDGE/SMS).

Configuration this frame is sent to tracker in order to set appropriate working mode

Acknowledge this frame is sent as an acknowledge to the working mode change request

Failure this frame indicated that some problem occurred while changing working mode

Setup this frame contains detailed data of all the GPS satellites seen by the device

Position this frame is sent by tracker with its current and past location

Status this frame contains GSM and GPS detailed signal status information and data con-cerning BTS stations seen by the device in its neighborhood

Low battery this frame is sent when the internal battery is low

Low prepaid this frame is used when a prepaid SIM card is installed in GSM-GPS tracker

Start moving this frame indicates that the object has just started moving

Stop moving this frame indicates that the object has stopped moving

6.5. WP2 Server communication with WP6 PortalThis protocol is used for communication with WP6 portal. In the future, when the integrationof the entire project matures, there will be a set of messages designed for communication withIndect Portal. Following messages are examples of the protocol that will be used to communicatewith Indect Portal.

PutObjects for providing data about GIS objects from portal to WP2.

GetObjects for listing objects in the specified geographical region and with specified criteriamet, objects can then be listed on the portal.

7. IntegrationThis section is describing methods of integration between system from WP2 and systems fromother WP. In following section by external system we understand applications and systemsexternal to the WP2.

7.1. Scenarios7.1.1. Tracking UAV operationsThe user of some external system is tracking some vehicle of interest. He/she knows thatan UAV is operating in the same area. WP2 provides to the external system a live feed fromthe UAV and end-user can for example use the portal developed in WP6 to watch the videostream from the UAV.Information provided by WP2 system:Video a life stream of the video from the UAV displayed in video control

Auxiliary the telemetry of the UAV: position, gimbal angle, gimbal properties, etc.

Input from user:None Portal operator is not authorized to control the UAV - this officer can communicate with

UAV operator with other means (e.g. phone) to change UAV actions.

36/45 public D22


7.1.2. UAV offline operationEnd-user is gathering a set of data to be presented in the court. He/she is interested in datafrom some time range and area. Using the portal the end-user determines if any UAV performeda mission in the specified location and time. If there was any related mission end-user can accessthe recorded video stream.Input from user:Time as described in 7.2.2.

Place (position type as described in 7.2.1) It can be a single point or 2 points describingcircular area between them.

Information provided:UAV flights in that time and area

Path UAV was following (described as a list of time and position pairs)

Gimbal direction

Based on this information operator can select a time and then start watching UAV flightin replay mode, but with ability to control the time and speed, similarly to video recorders.

7.1.3. UAV mission planningEnd-user is using an external system. In case the external system or end-user decides it wouldbe beneficial to launch an UAV mission, an external system can compute a path with waypointsand submit it to the WP2 system. WP2 system can queue it, suggest it to the UAV operatoror discard, depending on the constraints.Input from the external system:Mission description consisting of list of waypoints described in 7.2.4.Information provided:Result if the mission was accepted, queued, or discarded.

Validation is this mission executable in technical terms (e.g. no speed limits are exceeded).

Availability is UAV available at that time.

Reservation it is informing all personnel that mission is being planned at that time - thismight be an input to a more complex workflow.

7.1.4. License plate recognitionThe plate recognition station is located near the road and reads all license plates of vehiclesthat pass by. There are two operation modes possible: SUBMIT ALL and SUBMIT WANTED.In SUBMIT ALL mode station transmits all license numbers that were possible to be read. InSUBMIT WANTED mode station informs of the appearance of the vehicles predefined by theuser.Input from user:Wanted numbers license plate numbers of searched cars, defined by the user.

Information provided:Number license number of the vehicle

Image photo of the vehicle corresponding to the license number

Time the exact time of the recognition

Color (optional) the average color of the vehicle

D22 public 37/45


7.1.5. GSM-GPS trackingGSM-GPS tracking device supports various modes to provide a good compromise between powerconsumption and tracking quality. The modes can be changed in real time to accommodatecurrent requirements.Input from user:Frequency frequency of GPS position retrieval

Synchronization frequency frequency of batch position synchronization between device andserver

Information provided:Time time of the position retrieval

Position position of the tracked object

Accuracy accuracy of current position

Speed speed of the tracked object

Battery level current level of the device battery

GSM cell current GSM Base Transceiver Station (BTS) details

7.2. Data types

7.2.1. PositionFields:Latitude represented in decimal degrees (52.1234°), depending on the data can be represented

by string or fixed-point value,

Longitude same as latitude,

Altitude meters above sea-level or meters above ground, will be marked by metadata.

Geodetic reference system WGS84

7.2.2. TimeThis type describes time in a unequivocal way. Implementation will be discussed later.

This and other similar data types are described here as placeholders for future discussions.Allowing partners in other WPs to be aware of data available in WP2.

It will have to include date, time and offset to the UTC, probably a timezone info.

7.2.3. SpeedThis data type describes speed in a unequivocal way. Implementation will be discussed later.At the moment we propose meters per second.

7.2.4. WaypointEach waypoint describes a point in mission when state of UAV is going to change. It containsthese fields:

Time as described 7.2.2.

Place as described 7.2.1.

Action is describing mission plan for that waypoint e.g. start recording video.

38/45 public D22


7.2.5. UAVThis data type describes All parameters of UAV while it is in motion.Fields:Position as described in paragraph 7.2.1

Time as described in 7.2.2

Speed as described in 7.2.3

Gimbal direction angle in which camera is looking

Observed area area which is observed by camera (position + radius)

Gimbal stabilization mode mode of stabilization

Camera properties e.g. zoom level, or mode of operation

Recording is the recording from UAV camera available.

Action any special actions UAV is performing e.g.. drop

Battery level measured voltage of main battery of UAV

Mission type a brief description of mission (an item out of a predefined list)

Next waypoint (waypoint type as described in 7.2.4)

7.2.6. UAV MissionDetailed description of mission. It consists of a list of waypoints as described in 7.2.4.

7.2.7. Tracked objectDescribes a snapshot of data about object. A series of this entity can be streamed updatingthe current information. System can be queried about a list of history snapshots for an object.

Parameters always present in the snapshot:

Uri string identifier associated with the object, unique key

Position see 7.2.1

Time timestamp of the current record, see 7.2.2

Text description entered by the end-user

Optional parameters:

Speed as described in 7.2.3

Icon uri of the icon that can represent the object on screen

7.2.8. Vehicle dataData structure that describes the vehicle detected by the automatic plate recognition station.Fields:Number license number of the vehicle

Image photo of the vehicle corresponding to the license number

Time the exact time of the recognition, see 7.2.2

Color (optional) the average color of the vehicle

D22 public 39/45


7.3. Data provided (out)WP2 provides three kinds of data: video streams, descriptions of tracked objects and events.

7.3.1. Video streamsThe whole system has many nodes equipped with video cameras. These might be the UAVor license plate recognition station. The man objective is to analyze the video stream locallyand make all decisions autonomously. In fact situations that require presentation of live videostream to the operator may arise. All nodes of the system analyzing video data can also transmitthem to the remote operator. The UAV can produce Video streams in mJPEG or in MPEG4formats. The plate recognition station sends only the static images during normal operation. Itis also possible to transmit live video in mJPEG or MPEG4, however it is optional feature. Allvideo streams can be stored and downloaded later if needed. The video storage server supportsmJPEG and MPEG4 formats. It can be queried for the stored video. Query can contain thesource (like for example particular UAV), desired time period and quality.

7.3.2. Objects descriptionsWhen the system is tracking objects snapshots of objects’ information are available from thesystem. External services can submit queries in four scenarios:

• current information about object(s) identified by uri(s) – in response it will get a streamof snapshots (see 7.2.7) for this object(s)

• current information about objects from a desired area (to be determined if the area mustbe a rectangle or can have arbitrary shape) – in response it will get a stream of snapshots(see 7.2.7) for objects that are present or moving in this area

• historic information about object(s) identified by uri(s) and time range – in response itwill get a list of snapshots for this object(s)

• historic information about objects from a designed area and time range – in response itwill get a list of snapshots for objects that were present in this area in the specified timerange.

API of this service is to be determined in cooperation with the consumers of the data.

7.3.3. EventsUAV State EventThis event occurs when UAV reaches a state. Possible states are:

• Reached a waypoint (including start / landing)

• Sensor found an object (Sensor threshold reached)

• Low battery

• Emergency

State frame format:

Position as described in 7.2.1

Time as described in 7.2.2

State_Description information about state reached e.g. “Waypoint 10 reached”

Plate Recognition Vehicle AppearanceThis event occurs when the vehicle appears in front of the camera. This event is triggered by:

• Appearance of any vehicle, when working at SUBMIT ALL mode

• Appearance of “wanted” vehicle, when working at SUBMIT WANTED mode

The standard data for plate recognition station are transmitted in such a case.

40/45 public D22


8. Future workTasks left to complete in the task 2.1. top-level architecture design are:

• choice of a middle-ware platform for message-based communication,

• detailed description of all nodes of the system,

• detailed description of integration with WP6 to allow for implementations,

• detailed specifiaction of the encryption protocols and authorization.

ConclusionsIn this document we have described an architecture of a network-centric system being designedin Work Package 2 of the INDECT project.

The presented design is highly modular, scalable, open and secure. It is split into two mainparts. Presented first is a communication subsystem, consisting of hardware infrastructureand communication models and strategies. Second is a product built on top of the networkto deliver the required functionality - UAV, UAV Ground Segment, Command Room, Servers,GPS-GSM Trackers, Mobile Tracking Sensors and Multipurpose Sensors. Both parts can beextended, modified or replaced independently as long as they conform to the specification.

Everything built in Work Package 2 is modular – thanks to this new nodes can be added tothe network infrastructure, new nodes can be added to the product layer, etc. Design of bothserver and client software architecture is designed with modularity in mind, too. Software isdesigned to be easily extended with new modules and plugins, to support easy addition of newfunctionality and features.

D22 public 41/45


References[1] A. Ryan, X. Xiao, S. Rathinam, J. Tisdale, M. Zennaro, D. Caveney, R. Sengupta,

and J. Hedrick, “A modular software infrastructure for distributed control of col-laborating UAVs,” Proceedings of the AIAA Conference on Guidance, Navigation,and Control, pp. 1-10.

[2] A. Puri, “A Survey of Unmanned Aerial Vehicles (UAV) for Traffic Surveillance,”Department of Computer Science and Engineering, University of South Florida,2005.

[3] R. W. Pratt, "Flight Control Systems - Practical Issues in Design and Implemen-tation", Institution of Engineering and Technology, 2000.

[4] Sonka, Hlavac, Boyle, “Image Processing, Analysis and Machine Vision, secondedition”, Thomson Asia, 2004.

[5] G. Mühl, L. Fiege, P. Pietzuch, “Distributed Event-Based Systems”, Springer-Verlag Berlin Heidelberg 2006.

[6] O. Eini, “Unnatural acts on source code”, http://ayende.com/Blog/, online.

[7] M. Fowler, “Patterns of Enterprise Application Architecture”, Addison-Wesley,2003.

[8] M. Fowler, “Closure”, http://www.martinfowler.com/bliki/Closure.html, Septem-ber 2004.

[9] M. Fowler, “Inversion of control containers and the dependency injection pattern”,http://www.martinfowler.com/articles/injection.html, January 2004.

[10] G. Jeffrey, "System Requirements Analysis", Elsevier, 2006.

[11] J. Armstrong, “Programming Erlang”, armstrongsoftware, 2007.

D22 public 43/45


Version

Date

Upd

ates

andrevisio

nhistory

Edito

r2.0

29/03/2010

ToC

forD2.2

GrzegorzSo

bański,P

UT

2.1

30/08/2010

Add

edintegrationsection

Pawe

łLub

arksi,PU

T2.2

08/09/2010

Internal

review

GrzegorzSo

bański,P

UT

2.3

13/10/2010

Rem

arks

aftersecu

rityreview

GrzegorzSo

bański,P

UT

D22 public 45/45

Documents

AAA - InDECT Deliverable D2.2 Revised Version