C2017/3-8 RELIANCE architecture specification (final) WP2

C2017/3-8 RELIANCE

Deliverable 2.5 – RELIANCE reference

architecture specification (final)

WP2 – Verticals, Use Cases, Requirements,

Architecture and Business Models

Working Version of Deliverable

• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •

Dissemination level: Public

Version: 1.0

Authors: SII Concatel (CCTL)

Contributors: RELIANCE CONSORTIUM

C2017/3-8 RELIANCE REsiLIent ANd scalable sliCing over multiplE domains https://projectreliance.com/ Supported by

Spain – CDTI Sweden – VINNOVA Turkey – TÜBITAK

Participants in project (RELIANCE) are:

• SII Concatel S.L. (CCTL)

• Keyland Sistemas de Gestión S.L. (KEYLAND)

• Saint Patrich Technology S.L. (TyP)

• RISE Research Institutes of Sweden AB (RISE)

• Bombardier Transportation Sweden AB (BOMBARDIER)

• Westermo Network Technologies AB (WESTERMO)

• Eduro AB (EDURO)

• Turkcell Teknoloji (TURKCELL)

• Netas Telecommunications A.S. (NETAS)

• ULAK Communication Inc. (ULAK)

Project coordinator: Elio Saltalamacchia, SII CONCATEL

Disclaimer

This document contains material, which is copyright of certain PARTICIPANTS and may not be reproduced or copied without permission. The information contained in this document is the proprietary confidential information of certain PARTICIPANTS and may not be disclosed except in accordance with the regulations agreed in the Project Consortium Agreement (PCA).

The commercial use of any information in this document may require a licence from the proprietor of that information.

Neither the PARTICIPANTS nor CELTIC-Plus warrant that the information contained in this document is capable of use, or that use of the information is free from risk, and accept no liability for loss or damage suffered by any person using the information.

1 Preface

Providing a range of services with varying requirements on a common mobile operator platform is considered a crucial factor to maximize the addressable markets. The capability to stretch service provisioning across multiple technical and administrative domains further enables operators to increase the coverage and capacity of such services. To meet the specific requirements of verticals, 5G systems have to be highly scalable and customizable, providing at the same time high-reliability levels, regardless of the heterogeneous underlying infrastructures.

The RELIANCE project extends the 5G network architecture with new functionalities needed for multi-service management, with the related abstractions, interfaces, and mechanisms to support needed resilience, security, and scalability. A key innovation of the project is the highly scalable and robust slice choreography plane that enables the offering of vertical-tailored slices “as-a-Service”, ensuring the required scalability, security, and resilience features. It facilitates the mapping of service requirements from the verticals with their predicted traffic demands into dedicated networking and cloud resources through automated multiparty negotiation. Scalable slicing is complemented with protocols for deployment and runtime adaptation of resilience and security mechanisms over heterogeneous infrastructures, so to enable a unified reliability framework.

The architectural work of the project is driven by the diverse needs and requirements stemming from vertical industries (e.g., automotive, industry 4.0, enhanced multimedia broadband, train control & management, and public protection & disaster relief). The RELIANCE framework will be implemented and validated on selected use-cases through ‘proof-of-concept’s, aiming at demonstrating the envisioned benefits of the RELIANCE framework.

This document, Deliverable D2.5, is linked to Task 2.2 RELIANCE functional architecture. This document is the final version of the RELIANCE reference architecture specification. Deliverable 2.5 is an extended version of Deliverable D2.2 (RELIANCE reference architecture specification (intermediate)). The objective of Task 2.2 is to define a high-level architecture addressing the existing limitations for the provision of slicing capabilities across multiple domains concerning the management, control, access, leasing, federation and interconnection of slates for a set of well-defined and business-driven use cases. Task 2.2 aims also at defining the system architecture requirements and system functional architecture taking into account the use case requirements. As part of the system architecture work, relevant security requirements are also defined. Task 2.2 provides inputs to WP3 and WP4 to build detailed specifications and to WP5 for the design and development of vertical-specific use cases and the testbed definition.

CELTIC-Plus Dissemination Level: Public page 4 (52)

RELIANCE Project ID: C2017/3-8 REsiLIent ANd scalable sliCing over multiplE domains

2 Executive Summary

The purpose of WP2 is to compile a portfolio of use cases originating from Verticals using the RELIANCE infrastructure, to infer from them the functional and non-functional requirements for offering slicing services -, and finally to define the high-level architecture for the RELIANCE project. Besides, WP2 will perform a techno-economic analysis of the business models derived from the interrelation of all the actors in the RELIANCE value chain. This document, Deliverable D2.5 is linked to Task 2.2 RELIANCE functional architecture. The objective of T2.1 is to detail and complement the initial set of use cases and extract service-level requirements that should be addressed by the RELIANCE end-to-end architecture. The objective of Task 2.2 is to define a high-level architecture addressing the existing limitations for the provision of slicing capabilities across multiple domains concerning the management, control, access, leasing, federation and interconnection of slates for a set of well-defined and business-driven use cases. Task 2.2 aims also at defining the system architecture requirements and system functional architecture taking into account the use case requirements.

Task 2.2 provides inputs to WP3 and WP4 to build detailed specifications and to WP5 for the design and development of vertical-specific use cases and the testbed definition. As part of the system architecture work, relevant security requirements are also defined.

Task 2.2 takes in input from Task 2.1 (Verticals, use cases, requirements and targeted 5G services) the diverse needs and requirements of RELIANCE use cases for defining the RELIANCE high-level architecture. More specifically, Task 2.2 takes in input from Deliverable D2.1 (RELIANCE business-driven use cases and requirements definition) describing the four main RELIANCE use cases:

• Train Control and Management.

• Industry 4.0.

• Smart Facility Management.

• Enhanced Multimedia Broadband.

2.1 Scope of Deliverable

This deliverable is related to Task 2.2. Its goal is to define at a high level of the architecture. This document will be used as the main reference for WP3 and WP4 to build detailed specifications and to WP5 for the design and development of vertical-specific use cases and the testbed definition. As part of the system architecture work, relevant security requirements will be defined. Deliverable D2.5 aims to provide a final release of specifications for the RELIANCE architecture, as well as, details about the 3GPP 5G Core and the Turkcell 5G SA Core testbed.



3 Table of Contents

1 Preface ........................................................................................................................................ 3 2 Executive Summary .................................................................................................................... 4

2.1 Scope of Deliverable ............................................................................................................ 4 3 Table of Contents ........................................................................................................................ 5 4 List of Figures and Tables ........................................................................................................... 6 5 Abbreviations .............................................................................................................................. 7 6 Introduction ................................................................................................................................. 8 7 Current limitations and challenges ............................................................................................ 10

7.1 Summary of existing approaches and reference architectures ......................................... 10 8 Requirements from verticals ..................................................................................................... 11 9 Integration with verticals ........................................................................................................... 12

9.1 Slice request and creation ................................................................................................. 12 9.2 Monitoring and support ...................................................................................................... 13 9.3 Security considerations ...................................................................................................... 13

10 Domain-Specific Orchestration Layer ................................................................................... 15 11 5G-Enabled Services Repository and Composition Architecture Layer................................ 18

11.1 Current Technologies for High-level Composition ......................................................... 18 11.2 ML-enabled Vertical-focused Orchestration Mechanisms ............................................. 26 11.3 Visual Composer Diagrams ........................................................................................... 32

12 Realizing RELIANCE architecture ......................................................................................... 36 12.1 LTE Radio Slicing ........................................................................................................... 36 12.2 LTE Radio some dedicated EPC elements .................................................................... 36 12.3 NSA architecture: LTE radio and New Radio with all dedicated EPC elements ............ 37 12.4 SA architecture: Standard-based network slicing with New Radio and 5G Core .......... 38

12.4.1 Observability of KPIs in network slicing implementation ............................................ 43 12.4.2 Network Slicing in LTE Testbed ................................................................................. 46

13 Conclusion ............................................................................................................................. 50 References ....................................................................................................................................... 51



4 List of Figures and Tables

Figure 1 Slice creation use case diagram ........................................................................................ 12 Figure 2 Package diagram for use cases ........................................................................................ 13 Figure 3 Variety of communication service instances provided by multiple NSIs ............................ 15 Figure 4 Provisioning for a network slice subnet instance (NSSI) .................................................. 15 Figure 5 Network Slice Related Management Functions ................................................................ 16 Figure 6 RELIANCE Slice Creation Steps ....................................................................................... 17 Figure 7 Simple explanation of the basic concepts of FBP [5] ........................................................ 18 Figure 8 The browser-based editor of NodeRed .............................................................................. 19 Figure 9 Juju server compatibility [10].............................................................................................. 19 Figure 10 ARCADIA overview [13] ................................................................................................... 21 Figure 11 Docker Container Architecture [14] .................................................................................. 21 Figure 12 Docker and Microsoft Bring Containers to Windows Apps .............................................. 22 Figure 13 Development history leading to Kubernetes [13] ............................................................ 23 Figure 14 RedHat Open Shift Architecture ...................................................................................... 24 Figure 15 YANG Data Modelling Language [25] .............................................................................. 26 Table 1 GSL areas overview ............................................................................................................ 28 Figure 16 PaaSword Overview [9] .................................................................................................. 30 Figure 17 MATILDA project architectures overview [11] ................................................................. 31 Figure 18 Service Composition Wireframe ..................................................................................... 32 Figure 19 Basic field Wireframe ...................................................................................................... 32 Figure 20 Array Wireframe .............................................................................................................. 33 Figure 21 Object Wireframe ............................................................................................................ 33 Figure 22 GUI WIREFRAME FOR CONTAINER COMPOSITION .................................................. 34 Figure 23 LAMBDA ARCHITECTURE-BASE SERVICE COMPOSITION ..................................... 34 Figure 24 KAFKA NODE CONFIGURATION 1 .............................................................................. 35 Figure 25 KAFKA NODE CONFIGURATION 2 ............................................................................... 35 Figure 26 Instant messaging service and its network slicing implementation ................................. 36 Figure 27 A VoLTE service example ............................................................................................... 37 Figure 28 Network slicing architecture: LTE radio and New Radio ................................................ 37 Figure 29 RELIANCE architecture: Standard-based network slicing with New Radio and 5G Core 39 Figure 30 5G system architecture (Cisco Ultra 5G Packet Core Solution, White paper, 2018) ...... 40 Figure 31 Turkcell 5G SA Core Testbed Architecture ...................................................................... 41 Figure 32 System overview of Turkcell Testbed for Video Conference Use Case (NETAS) .......... 41 Figure 33 System overview of Turkcell Testbed for Industry 4.0 Use Case (KEYLAND) ............... 41 Figure 34 System overview of Turkcell Testbed for Remote Maintenance with Volumetric Streaming Use Case (CONCATEL)................................................................................................................... 42 Figure 35 System overview of Turkcell Testbed for Train to Cloud (Wayside) Use Case (BOMBARDIER) ............................................................................................................................... 42

https://aunia-my.sharepoint.com/personal/juan_leandro_ingubu_io/Documents/AUNIA-RELIANCE/FINALREVIEW/deliverables_1/RELIANCE--D25-Reference_architecture_specification_(final)%20v2.0.docx#_Toc69243947



5 Abbreviations

3GPP 3rd Generation Partnership Project

BSS Business Support Systems

CSP Communication Service Provider

CSMF Communication Service Management Function

DN Data Network

DRB Data Radio Bearer

DSCP Differentiated Services Code Point

eNB evolved Node B

gNB Next Generation Node B

HSS Home Subscriber Server

LTE Long-Term Evolution

MANO Management and Orchestration

MME Mobility Management Entity

NFVI Network Functions Virtualization Infrastructure

NSI Network Slice Instance

NSSI Network Slice Subnet Instance

NSMF Network Slice Management Function

NSSMF Network Slice Subnet Management Function

OSS Operations Support Systems

PCRF Policy and Charging Rules Function

P-Gw Packet data network Gateway

QCI Quality of Service Class Identifier

QoS Quality of Service

RAN Radio Access Network

SDN Software-Defined Networking

S-Gw Serving Gateway



6 Introduction

The CELTIC-Plus RELIANCE project extends the 5G network architecture with new functionalities needed for multi-service management, with the related abstractions, interfaces, and mechanisms to support needed resilience, security, and scalability. A key innovation of the project is the highly scalable and robust slice choreography plane that enables the offering of vertical-tailored slices “as-a-Service”, ensuring the required scalability, security, and resilience features. It facilitates the mapping of service requirements from the verticals with their predicted traffic demands into dedicated networking and cloud resources through automated multiparty negotiation. Scalable slicing is complemented with protocols for deployment and runtime adaptation of resilience and security mechanisms over heterogeneous infrastructures, so to enable a unified reliability framework.

The RELIANCE project conducts research activities on the concept of new resource abstraction based on the slate concept, which provides isolated resource share offerings within a domain for the slices. This new abstraction requires dynamic management procedures via which resources can be aggregated and partitioned to introduce unified control and management. The architectural work of the project is driven by the diverse needs and requirements stemming from vertical industries (e.g., automotive, industry 4.0, enhanced multimedia broadband, train control & management, and public protection & disaster relief). The RELIANCE framework will be implemented and validated on selected use-cases through proof-of-concepts, aiming at demonstrating the envisioned benefits of the RELIANCE framework.

This document, Deliverable D2.2 is linked to Task 2.2 RELIANCE functional architecture. The objective of T2.1 is to detail and complement the initial set of use cases and extract service-level requirements that should be addressed by the RELIANCE end-to-end architecture. The objective of Task 2.2 is to define a high-level architecture addressing the existing limitations for the provision of slicing capabilities across multiple domains concerning the management, control, access, leasing, federation and interconnection of slates for a set of well-defined and business-driven use cases. Task 2.2 aims also at defining the system architecture requirements and system functional architecture taking into account the use case requirements.

Task 2.2 provides inputs to WP3 and WP4 to build detailed specifications and to WP5 for the design and development of vertical-specific use cases and the testbed definition. As part of the system architecture work, relevant security requirements are also defined.

Task 2.2 takes in input from Task 2.1 (Verticals, use cases, requirements and targeted 5G services) the diverse needs and requirements of RELIANCE use cases for defining the RELIANCE high-level architecture. More specifically, Task 2.2 takes in input from Deliverable D2.1 (RELIANCE business-driven use cases and requirements definition) describing the four main RELIANCE use cases:

• Train Control and Management.

• Industry 4.0.

• Smart Facility Management.

• Enhanced Multimedia Broadband.

This deliverable, D2.2, provides a general description of the RELIANCE architecture (Standard-based network slicing with New Radio and 5G Core). The next D2.5 deliverable (RELIANCE reference architecture specification (final) – extended and final version of RELIANCE architecture) will provide more detailed information about the RELIANCE architecture (Standard-based network slicing with New Radio and 5G Core).

In this extended deliverable, we have presented the final version of the RELIANCE architecture. We have in particular (i) provided more detail about the 3GPP 5G Core and (ii) explained the Turkcell 5G SA Core testbed

The deliverable is organized as follows: In Section 7 we summarize existing limitations for the provision of slicing capabilities across multiple domains; in Sections 8 and 9 we provide an initial overview of the creation phase in network slicing - offering vertical-tailored solutions; in Section 10 we describe how the RELIANCE architecture is based on the 3GPP Network Slicing Management concept; in Section 11 we provide an analysis of current technologies for high-level service



composition; Section 12 provides (i) an overview of RELIANCE architecture and 3GPP 5G Core and (ii) explain the Turkcell 5G SA Core testbed. In Section 13 we present the conclusions.



7 Current limitations and challenges

Summary of existing limitations for the provision of slicing capabilities across multiple domains concerning the management, control, access, leasing, federation and interconnection of slates.

7.1 Summary of existing approaches and reference architectures

- 3GPP based network slicing is only available with 5G SA architecture - Existing orchestrators still misses a dynamic - QoS based prioritization mechanisms (QCI, DSCP, etc.)

o Limited to radio and transport. o Requires manual planning and operation. Difficult to maintain. o Per UE QoS is supported while network slicing allows up to 8 different slice

assignments per UE - Mainly for physical network functions (PNF) - There is no real end-to-end network slicing available in the current architectures where each

domain has an awareness of slice.



8 Requirements from verticals

RELIANCE will conduct research activities on the concept of new resource abstraction based on the slate concept, which provides isolated resource share offerings within a domain for the slices. This new abstraction requires dynamic management procedures via which resources can be aggregated and partitioned to introduce unified control and management.

The architectural work of the project is driven by the diverse needs and requirements stemming from new vertical enhanced services from several industries (e.g., automotive, industry 4.0, enhanced multimedia broadband, train control & management), designed to take advantage of the new capabilities derived from the new 5G context. The RELIANCE framework will be implemented and validated on selected use-cases through “proof-of-concept”s, aiming at demonstrating the applicability, feasibility, resilience, and benefits of slice management in the creation, provisioning and seamless management of a single domain in an operator network.

We analyzed the following use cases:

- Train Control and Management. - Industry 4.0. - Smart Facility Management. - Enhanced Multimedia Broadband.

Based on these requirements we identified the following recurring requirements of these use cases:

- security - latency, jitter, predictability - guaranteed bandwidth

A detailed description of the use cases (and their requirements) can be found in Deliverable D2.1.

When we reviewed 3GPP TR 28.801 and see similar characteristics for network slices as follows:

- RAN technology selection - bandwidth, latency requirements - guaranteed vs. non-guaranteed QoS - security

We use these requirements to define the final architecture.



9 Integration with verticals

RELIANCE extends the 5G network architecture with new functionalities needed for multi-service management, with the related abstractions, interfaces, and mechanisms to support needed resilience, security, and scalability. A key innovation of the project is the highly scalable and robust slice choreography plane that enables the offering of vertical-tailored slices “as-a-Service”, ensuring the required scalability, security, and resilience features. It facilitates the mapping of service requirements from the verticals with their predicted traffic demands into dedicated networking and cloud resources through automated multiparty negotiation. In this section, we provide and an initial overview of the creation phase in network slicing - offering vertical-tailored solutions.

9.1 Slice request and creation

Slice creation procedures involves steps from customer to deployment within the operator infrastructure. The customer needs to relay its SLA requirements in a specified template required by the operator. There are several standardization activities to define these templates to provide interoperability between verticals and operators. This way a customer can create a single template and share this with multiple operators without any modification.

Figure 1 Slice creation use case diagram

Error! Reference source not found. shows actors and their use cases for slice creation. Here video conferencing applications asks from network slice management function (NSMF) to create a slice with the specified SLAs. NSMF requests the creation of a network slice from the operator. The operator sets up the required infrastructure including VNF creation, networking configuration, radio resource allocation, etc.



Figure 1 Slice creation use case diagram

Error! Reference source not found. shows the package diagram for slice creation for multiple verticals.



Figure 2 Package diagram for use cases

9.2 Monitoring and support

Applications require to be updated with the status of the assigned slice. Such monitoring data may include VNF based (CPU, memory, etc.), network-based (latency, throughput, jitter, etc.), or application based (created PDU sessions, active users, etc.) status reports.

Standards do not yet define how such metrics could be accessed by third-party applications. However, we plan to register metrics and logs into a central database and make them available to third party applications.

9.3 Security considerations

Certain use cases such as banking, train remote control, etc. require utmost security due to the critical nature of their operations. In such cases, slice isolation will ensure that data confidentiality for those use cases. To achieve data confidentiality, the security of network, computation and storage resources should be considered. Computation resources include VNFs and those critical services can be assigned to a dedicated slice without sharing any VNFs with other slice users.

Network intrusion detection (IDS) and prevention systems (IPS) are also critical for the secure operation of the slices. In D4.2 we investigate such detection and prevention mechanisms using an SDN controller. In our study, a DoS attack is simulated and with the use of a ULAK MAYA SDN controller together with a third party IDS system (sFlow-RT) the DoS attack is taken under control and the attacker is blocked from the whole network.



10 Domain-Specific Orchestration Layer

In this section, we describe how RELIANCE aims at developing its unique architecture by adhering to the 3GPP Network Slicing Management concept.

Network slicing is seen as one of the key features for 5G, allowing vertical industries to take advantage of 5G networks and services. 3GPP SA5 adopts the network slice concept as defined in SA2 and addresses the management aspects. Network slicing is about transforming a PLMN from a single network to a network where logical partitions are created, with appropriate network isolation, resources, optimized topology and specific configuration to serve various service requirements.

As an example, a variety of communication service instances provided by multiple Network Slice Instances (NSIs) are illustrated in the figure below. The different parts of an NSI are grouped as Network Slice Subnets (e.g. RAN, 5GC and Transport) allowing the lifecycle of a Network Slice Subnet Instance (NSSI) to be managed independently from the lifecycle of an NSI.

Figure 3 Variety of communication service instances provided by multiple NSIs

According to 3GPP’s specifications and reports CSPs (Communication Service Provider) need the four following phases which are related to the lifecycle management: Preparation, Instantiation, Configuration and Activation; Run-time; Decommissioning.

Figure 4 Provisioning for a network slice subnet instance (NSSI)

The overall requirements operators need having been identified by 3GPP are listed below:

https://www.3gpp.org/images/articleimages/SA5_5g_article_2018/2018_03_SA5_002_kf.jpg



- Communication Service Management Function (CSMF):

• Responsible for translating the communication service related requirement to network slice related requirements.

• Communicate with Network Slice Management Function (NSMF).

- Network Slice Management Function (NSMF):

• Responsible for management and orchestration of NSI.

• Derive network slice subnet related requirements from network slice related requirements.

• Communicate with the Network Slice Subnet Management Function (NSSMF) and Communication Service Management Function.

- Network Slice Subnet Management Function (NSSMF):

• Responsible for management and orchestration of NSSI.

• Communicate with the NSMF.

Figure 5 Network Slice Related Management Functions

RELIANCE aims at developing its unique architecture by adhering to the 3GPP Network Slicing Management concept.

Figure 6 shows the RELIANCE Network Slice creation methodology. The MANO actor may include CSMF, NSMF and NSSMF or some functions can be delegated to OSS/BSS level.



Figure 6 RELIANCE Slice Creation Steps

MA

NO

NF

VI

SD

N

Co

ntr

oll

er



11 5G-Enabled Services Repository and Composition Architecture Layer

At a high level, the vertical owner should be provided with an interface for service composition, based on different configurations, nodes and modules. Added to this, a workflow must be defined to provide communication between the vertical owner and the 5G infrastructure provider. This should be bidirectional communication. First, a mechanism should allow the communication to deploy a slice as it has been defined through the interface by the vertical owner. Second, once the slice has been deployed, the vertical owner should get information related to the level of fulfilment related to KPIs. This could also open new opportunities for data analytics and the application of machine learning techniques to FCAPS.

In this chapter, an analysis of current technologies for high-level service composition is defined. Second, an analysis of ML-Enabled Vertical-focused Orchestration mechanisms is shown. Finally, a set of preliminary diagrams for a visual composer are shown.

11.1 Current Technologies for High-level Composition

1) NodeRED - https://nodered.org/

NodeRed was originally developed by IBM’s Emerging Technology Services [1] but now forms part of the OpenJS Foundation [2]. The key functionality of NodeRed is to wire together “hardware devices, APIs and online services in new and interesting ways” [3]. NodeRed does this by providing a browser-based interface to create “flows” that help connect multiple IoT connections.

NodeRed uses Flow-Based Programming (FBP) to achieve this – i.e. a visual programming tool based on the server-side Java Scripting platform, Node.js. FBP is a break from Imperative and Functional programming and allows even non-programmer to understand code. Invented by J. Paul Morrison in the 1970s, and described in his 1994 book “Flow-Based Programming: A New Approach to Application Development” [4], FBP contains a series of Black-boxes (“Nodes” in NodeRed) which each have a well-defined function. These black boxes take data as an input, process it and then pass on the processed data as an output.

Figure 7 Simple explanation of the basic concepts of FBP [5]

“A, B and C are processes executing code components. O1, O2, and the two INs are ports connecting the connections M and N to their respective processes. It is permitted for processes B and C to be executing the same code, so each process must have its own set of working storage, control blocks, etc. Whether or not they do share code, B and C are free to use the same port names, as port names only have meaning within the components referencing them (and at the network level, of course).

https://nodered.org/



M and N are what are often referred to as "bounded buffers", and have a fixed capacity in terms of the number of IPs that they can hold at any point in time.” [4].

As shown in Error! Reference source not found., NodeRed allows users to edit Flows in a browser-based editor, connecting multiple flows and then using a single click to deploy them to the runtime.

A text editor is available to edit flows in Javascript and each flow can then be saved to a built-in library, along with useful functions and templates, for later reuse or sharing.

Figure 8 The browser-based editor of NodeRed

The flows are stored in JSON meaning that they can be easily exported or shared. What’s more, the NodeRed repository contains over 225,000 modules allowing users to extend the capabilities of nodes easily.

Each runtime is built on Node.js, making them lightweight, event-driven and non-blocking, meaning that are ideal for running on simple low-power hardware (such as Raspberry Pi) situated at the edges of the network.

NodeRed can be run locally using Docker and is perfect for:

• Raspberry Pi, [5],

• BeagleBone Black, [6]

• Arduino, [7]

• Android, [8]

NodeRed uses a JavaScript array to represent the “flow”, where each object is a node containing a set of core type-specific properties. NodeRED does not manage the deployment, scaling or orchestration of these nodes only handling the correct deployment of the flow nor does it allow for the specification of (network) parameters for different component links.

2) Juju - https://github.com/juju

Juju is an open-source application modelling tool that reduces operation overhead by enabling quick deployment, configuration, integration and scaling of complex cloud-based software topologies. Juju introduces a common controller which operates all machines in the user’s running models and therefore simplifies operational tasks on cloud services. It works on public cloud solutions such as AWS, GCE, and Azure or private solutions such as MAAS, OpenStack, and VSphere, as well as on local deployments.

Figure 9 Juju server compatibility [10]

https://nodered.org/docs/hardware/raspberrypi

https://nodered.org/docs/hardware/beagleboneblack

https://nodered.org/docs/hardware/arduino

https://nodered.org/docs/platforms/android

https://github.com/juju



Developed by software company Canonical, the company behind Ubuntu, Juju is open source but is also available as a hosted solution in the form of JAAS, Juju as a Service. [10]

An application is the central construct of Juju, accessible all over the network it is usually a long-running service. Canonical uses the following analogy to explain the concept of application:

“It’s easiest to think of the term “application” in Juju in the same way you would think of using it day-to-day. Middleware such as database servers (PostgreSQL, MySQL, Percona Cluster, etc, …), message queues (RabbitMQ) and other utilities (Nagios, Prometheus, …) are all applications.” [10]

Examples of Juju applications include:

• A Ruby on Rails web application deployed behind Apache2 and Phusion Passenger.

• All workers within a Hadoop cluster are considered a single application, although each worker unit [10]

Another central aspect of Juju is a “Charm”, which tackles the service metamodel in this case. Charms are software packages that ensure the compute node that they’re being deployed to satisfies the size constraints specified by the user. This includes installation and scaling as well as networking and storage capacity rules.

Charms are grouped in “Bundles”, addressing the service graph metamodel in this case. Bundles can include particular configurations and relations between deployed software and are used to link various services together in order to allow the user to deploy large parts of their app infrastructure in one go.

Charm’s and Bundle’s main functionalities are Installations, Configurations, Connections, Upgrades and updates, Scaling out and scaling back, health checks, operational actions and Benchmarking.

3) ARCADIA - http://www.arcadia-framework.eu

ARCADIA was a 2016 H2020 funded project aiming to tackle the design and development of distributed applications “consisting of micro services in an emerging pattern, considering the advantages associated with the management of self-deployable and components to be orchestrated as well as the adoption of a DevOps culture in cloud applications deployment and management.” [11]. Put another way, the goal of the ARCADIA project is to design and validate a novel reconfigurable-by-design highly distributed applications development paradigm over programmable infrastructure. [12] and adopts principles and techniques already available in novel software engineering paradigms including Network Function Virtualization (NFV) and Software-Defined Networking (SDN).

ARCADIA presents a context model which represents the entire lifecycle of reconfigurable-by-design distributed applications, consisting of micro services and denoted in the form of a service graph. ARCADIA describes a policy management framework that is targeted at service providers who managing deployment and orchestration aspects of such applications over programmable infrastructure.

http://www.arcadia-framework.eu/



Figure 10 ARCADIA overview [13]

The most granular executable unit of an ARCADIA application is called the ARCADIA Component Model. The Component Model deals with component and service-graph models and has a set of interconnected components producing the service graph. The ARCADIA Service Graph Model can be described as a collection of ARCADIA Component Models forming a directed graph.

Highly Distributed Applications (HDAs) are another central component of ARCADIA and are instantiations of a complex service graph where each component is described with several properties. This information is enclosed in an XML file, generated by a specific XML Schema Definition (XSD).

Finally, the service graph model encapsulates the description of each component, as well as a description for each virtual link. This also contains information regarding monitoring metrics that express the whole service graph.

4) DOCKER - https://www.docker.com/

Docker is an open-source tool used to offer developers more freedom in regards to how they create, deploy and run applications and is based on the concept of “Containers”. A container allows a developer to package up an application with all the necessary libraries and dependencies so that it can be deployed as one package.

Figure 11 Docker Container Architecture [14]

Docker was created so that the package will run on any other Linux machine regardless of any customized settings the developer or sysadmin may have applied. Conceptually it is similar to a virtual machine however rather than creating an entire operating system, Docker enables applications to be able to use the same Linux kernel as the system that they're running on which means they can be distributed only with the components not already running on the host computer, greatly reducing the size of the application and in turn, giving a significant performance boost.

https://www.docker.com/



The success of Docker on the Linux platform brought about a partnership with Windows so that Docker can also run on Windows Server and Desktop – often referred to as Docker Windows Containers [12]. The goal is to provide a consistent user experience with all Windows Server 2016 and later versions shipping with Docker Engine – Enterprise, as well as native support for Windows 10 via Docker Desktop. Docker Windows containers use the same Docker CLI, API, image format and content distribution

services as they do on Linux, providing a consistent experience across platforms. That is to say that developers can use the same commands and UI in Windows as on Linux environments. Another benefit is that different versions of IIS/.NET can coexist on a single system with container isolation to eliminate conflicts.

5) Docker Compose - https://docs.docker.com/compose/¨

Docker Compose is a tool for defining and running multi-container Docker applications. Simply put, Docker Compose allows a complex stack to be defined in a file and then I can be run with a single command. Compose uses a YAML (YAML Ain't Markup Language [14]) file to configure the application services, which can then be created and started from this configuration with a single command.

6) The Open Container Initiative (OCI) - https://www.opencontainers.org/

Launched on June 22nd 2015 by Docker, CoreOS and other leaders in the container industry and formed under the auspices of the Linux Foundation, the Open Container Initiative (OCI) is a lightweight, open governance structure developed explicitly to create open industry standards around container formats and runtime.

The Runtime Specification (runtime-spec) and the Image Specification (image-spec) are the two current specifications defined by the OCI [13].

• The Runtime Specification outlines how to run a “filesystem bundle” that is unpacked on disk.

• This specification defines how to create an OCI Image and output an image manifest, a filesystem (layer) serialization, and an image configuration

Docker has donated runC, its container format and runtime to serve as the foundations of this project and can be accessed at: https://github.com/opencontainers/runc.

Projects associated with the Open Container Initiative can be found at https://github.com/opencontainers and developers can interact with the community at https://www.opencontainers.org/community.

7) Kubernetes by Google – http://kubernetes.io

Figure 12 Docker and Microsoft Bring Containers to Windows Apps

https://docs.docker.com/compose/¨

https://www.opencontainers.org/

https://www.opencontainers.org/community

http://kubernetes.io/



Hosted by the Cloud Native Computing Foundation (CNCF) [12], Kubernetes is a portable, extensible open-source container orchestration engine for automating deployment, scaling, and management of containerized workloads, services and applications, facilitating both declarative configuration and automation.

Figure 13 Development history leading to Kubernetes [13]

Companies traditionally ran applications on physical servers however this often leads to problems with resource allocation. For example, if multiple instances of an application were running on one server, it may be the case that one instance would consume most of the resources, causing the other application to underperform.

This leads to the development of Virtualised Deployment and the concept of a Virtual Machine (VM). Multiple instances of virtual machines can run on a server and provide greater control over resource allocation, greater scalability and greater security, as virtual machines are isolated and one application cannot freely access another application running on a different virtual machine. The downside to VMs is that they are a complete machine, each with all the components needed to run the application – including their operating system.

As its name implies, Container Deployment involves the concept of Containers. A “container” is like a virtual machine however it allows the application to access and share the actual kernel of the machine it is running on making it much more lightweight than a virtual machine.

Kubernetes.io attributes the popularity of containers in software development to the following features [13]:

• Agile application creation and deployment: Increased ease and efficiency of container image creation compared to VM image use.

• Continuous development, integration, and deployment: Provides for reliable and frequent container image build and deployment with quick and easy rollbacks (due to image immutability).

• Dev and Ops separation of concerns: Create application container images at build/release time rather than deployment time, thereby decoupling applications from infrastructure.

• Observability Not only surfaces OS-level information and metrics, but also application health and other signals.

• Environmental consistency across development, testing, and production: Runs the same on a laptop as it does in the cloud.

• Cloud and OS distribution portability: Runs on Ubuntu, RHEL, CoreOS, on-premises, on major public clouds, and anywhere else.

• Application-centric management: Raises the level of abstraction from running an OS on virtual hardware to running an application on an OS using logical resources.

• Loosely coupled, distributed, elastic, liberated micro-services: Applications are broken into smaller, independent pieces and can be deployed and managed dynamically – not a monolithic stack running on one big single-purpose machine.



• Resource isolation: predictable application performance.

• Resource utilization: High efficiency and density.

8) OKD

Another distribution of Kubernetes is OKD (referred to as Origin in GitHub and in the documentation) which is optimized for continuous application development and multi-tenant deployment and extends it with security and other integrated concepts. OKD has been designed to add tools to Kubernetes which are developer and operations-centric, thus enabling rapid application development, easy deployment and scaling, and long-term lifecycle maintenance.

The full and current roadmap can be found at: https://ci.openshift.redhat.com/roadmap_overview.html

9) RedHat Open Shift

Moving away from the open-source model, RedHat Open Shift is a hybrid cloud, enterprise Kubernetes application platform and provides a suit of containerisation software used by over 1000 large companies. An open shift is comprised of 4 key products:

• OpenShift Container Platform: this is Red Hat's private platform-as-a-service (PaaS) product, built around a core of application containers powered by Docker, with orchestration and management provided by Kubernetes, on a foundation of Red Hat Enterprise Linux [18]and Red Hat Enterprise Linux CoreOS [18].

• OpenShift Origin: Also known as OKD and was described previously in this section.

• Red Hat OpenShift Online: is the public cloud application deployment and hosting platform from RedHat, providing on-demand access to OpenShift allowing developers to build, deploy and manage scalable containerized applications. [20]

• OpenShift Dedicated: is a Single-tenant, high-availability Kubernetes cluster, managed by Red Hat on Amazon Web Services

Figure 14 RedHat Open Shift Architecture

10) PUPPET - https://puppet.com/

https://ci.openshift.redhat.com/roadmap_overview.html

https://puppet.com/



Puppet is an open-core software configuration management tool running on Linux and Windows systems. It includes its declarative language to describe system configuration and is focused on automating Continuous Delivery, Continuous Compliance, Incident Remediation and Configuration Management. Puppet also has a community of 1000s of developers working to improve the puppet ecosystem.

Puppet offers 3 key products:

• Puppet Enterprise – automation of infrastructure delivery and management offering continuous enforcement of security and compliance policies with a single source of configuration across teams and servers.

• Puppet Remediate – automation of vulnerability management to improve how InfoSec and IT Ops teams find and fix vulnerabilities

• Project Nebula – automation of continuous deployment of applications to multiple cloud-native targets. Users author workflows in YAML, upload them to a source control repository and Project Nebula will take care of the rest

Puppet is powered by their open-source projects Bolt and Puppet Open Source.

Bolt is a language and OS agnostic open-source task orchestrator to make the orchestration of complex workflows more manageable through its simple, fast, agentless multi-platform approach. Developers can use scripting languages like YAML, PowerShell, Bash, Python or Ruby, or can even reuse content from the

Puppet Forge [21] – Puppet’s repository of 6,447 modules for Puppet and Puppet Enterprise.

Open Source Puppet is a powerful configuration management tool focusing on compliance, baseline, drift remediation, and deployment needs. It is open-source, with freely downloadable operating-system-specific agent packages, a

massively scalable server, and data warehousing capabilities via PuppetDB. [22].

11) TOSCA

It is a service automation language for orchestrating procedures and is created by OASIS [23]. Several orchestrating tools are based on TOSCA including OpenStack HEAT, Cloudify and Ubicity.

TOSCA is compatible with both XML and YAML.

TOSCA aims to enable the interoperable description of application and infrastructure cloud services, the relationships between parts of the service, and the operational behaviour of these services (e.g., deploy, patch, shutdown)--independent of the supplier creating the service, and any particular cloud provider or hosting technology. TOSCA will also make it possible for higher-level operational behaviour to be associated with cloud infrastructure management. [23]

TOSCA contains two types of entities: nodes and relationships. Networks, servers, clusters of servers, as well as software components, (for example, services or runtime environments) are described as Nodes in TOSCA. How these nodes are interconnected is described as a relationship.

12) YANG

Developed by the IETF NETCONF Data Modeling Language Working Group (NETMOD) [24], YANG is a high-level data modelling language for the NETCONF protocol which focuses on network



services and network device configuration. YANG is essentially a replacement for SNMP MIBs [25]. YANG models define the schema for configuration and state data of networking devices, and network management tools use the NETCONF protocol to manipulate these data. It is defined in RFC 7950 as a modular language that represents information in an XML tree format.

Figure 15 YANG Data Modelling Language [25]

11.2 ML-enabled Vertical-focused Orchestration Mechanisms

Below as follows, a set of different solutions that could be used as reference are shown:

1) ADMB Project - http://www.admb-project.org/

Automatic Differentiation (AD) is a set of techniques based on the mechanical application of the chain rule to obtain derivatives of a function given as a computer program and finds it use in Numerical Methods, Sensitivity Analysis, Design Optimization and Data Assimilation & Inverse Problems. [1]

Every program written works by executing a sequence of elementary arithmetic operations (additions or elementary functions such as exp()), applying the chain rule of derivative calculus repeatedly to these operations, AD can then compute derivatives of arbitrary order automatically.

Run by the ADMB Foundation and licensed under Berkeley Software Distribution (BSD), the AD Model Builder (ADMB) project provides a programming framework based on AD and aimed at highly nonlinear models with a large number of parameters using C++ classes and a native template language. In other words, it supports the application of automatic differentiation (AD) for solutions to non-linear statistical modelling and optimization problems. [2]

Computational efficiency and high numerical accuracy are both benefits of using AD and are both factors that are crucial in many practical problems. ADMB offers functionalities not found in other software such as the generic implementation of Laplace approximation of high-dimensional integrals for use in latent variable models. [3]

The main advantages of ADMB are flexibility, speed, precision, stability and built-in methods to quantify uncertainty. [3]

http://www.admb-project.org/



Template Model Builder (TMB), also from the ADMB foundation is a similar software to ADMB in that it offers Automatic Differentiation and Laplace Approximation, however, it users R where ADMB uses C++. The user defines the joint likelihood for the data and the random effects as a C++ template function and all other operations are carried out using R. [4]

http://www.autodiff.org/?module=Introduction

2) ASCEND

The Advanced System for Computations in ENgineering Design (ASCEND) was developed at Carnagie Mellon University as a rapid model building environment for complex models comprising large sets of simultaneous nonlinear algebraic equations. [5] to respond to the fact that current technologies do not adequately support mathematical modelling “in the large.” In this paper, we discuss a technology called ASCEND (Advanced System for Computations in Engineering Design), which addresses this issue. We describe two aspects of the technology: a modelling language and an interactive model-building environment. The ASCEND language is structured, declarative, and strongly typed, and incorporates object-oriented extensions. The interactive environment is based on the notion of a concurrent set of tools that reflect the various phases of ASCEND modelling. These tools do not enforce a strict sequence of operations, but rather have been designed to support the flexible access implied by declaratively specified models. We claim that ASCEND offers solutions to several of the issues raised by Arthur Geoffrion in his article, “Computer-based modelling environments,” [21] and use categories introduced by him to frame this discussion.

3) CUTE / CUTEr / CUTEst - https://github.com/ralna/CUTEst/wiki

Constrained and Unconstrained Test Environment (CUTE) was initially released in early 1993 as an open-source testing environment for optimization and linear algebra solvers. It was described in detail in 1995 by Bongartz ET, al. [6], who provided an overview of the environment, and full documentation of the available tools and interfaces at the time.

It was first superseded by CUTEr, (Constrained and Unconstrained Test Environment revisited) [7] in the early 2000s, being enhanced with new features such as new tools, including performance reporting for optimization packages being tested, full backward compatibility with CUTE, new interfaces to additional optimization packages, Fortran 90/95 support (improving on the dynamic memory management issues posed by Fortran 77), and an integrated C/C++ Application Programming Interface. [7]

CUTE was a widely used testing environment for optimization software. The CUTEr test set has become the de facto standard benchmark for research and production-level optimization solvers and has since grown to include a test set of over 1,150 examples as well as providing provide access to other test sets such as those from Netlib, Maros and Meszaros. The latest version is CUTEst (Constrained and Unconstrained Test Environment with safe threads for Mathematical Optimization), [8] providing updated features including dynamic memory allocation, thread-safe Fortran modular design, a new Matlab interface and a revised installation procedure integrated with GALAHAD.

4) ALGLIB – https://www.alglib.net/ ALGLIB is a cross-platform numerical analysis and data processing library. It comes as a free edition licensed under GPL or Personal/Academic license or as a commercial product. The commercial product offers high-performance

C++ version (SMP, SIMD) two C# versions, managed and HPC one (native code, SMP/SIMD) as well as the expected commercial support and warranties.

ALGLIB supports several programming languages (C++, C#, Delphi) and several operating systems (Windows and POSIX). ALGLIB allows users to perform features such as Data analysis (classification/regression, statistics), Optimization and nonlinear solvers, Interpolation and linear/nonlinear least-squares fitting, Linear algebra (direct algorithms, EVD/SVD), direct and iterative linear solvers. It also allows for Fast Fourier Transformation and many other algorithms and is widely used from “nuclear research to aerospace”. [11].

http://www.autodiff.org/?module=Introduction

https://github.com/ralna/CUTEst/wiki

https://www.alglib.net/



5) GSL - http://www.gnu.org/software/gsl/ The GNU Scientific Library (GSL) is a numerical library for C and C++ programmers and is offered as free software under the GNU General Public License.

The library provides over 1000 functions and mathematical routines such as random number generators, special functions and least-squares fitting, as well as an extensive test suite.

To provide an idea of the complexity of GSL, the complete range of subject areas covered include:

Table 1 GSL areas overview

Complex Numbers Roots of Polynomials Special Functions

Vectors and Matrices

Permutations Sorting BLAS Support Linear Algebra

Eigensystems Fast Fourier Transforms Quadrature Random Numbers

Quasi-Random Sequences

Random Distributions Statistics Histograms

N-Tuples Monte Carlo Integration Simulated Annealing

Differential Equations

Interpolation Numerical Differentiation Chebyshev Approximation

Series Acceleration

Discrete Hankel Transforms

Root-Finding Minimization Least-Squares Fitting

Physical Constants IEEE Floating-Point Discrete Wavelet Transforms

Basis splines

Running Statistics Sparse Matrices Linear Algebra

Alternatives - Python

6) SciPy – http://scipy.org Built on the NumPy extension of Python, SciPy (pronounced “Sigh Pie”) is a collection of mathematical algorithms and convenience functions that adds significant power to the interactive Python session made available under the BSD licence. It offers high-level commands and classes for

manipulating and visualizing data and creating data-processing and system-prototyping environments, like its competitors MATLAB, IDL, Octave, R-Lab, and SciLab.

As SciPy is based on Python, it can benefit from Python’s use in developing sophisticated programs and specialized applications (from parallel programming to web and data-base subroutines and classes) as well as the numerous additional modules in numerous niches of the software landscape created by developers across the world. SciPy includes algorithms and tools for tasks such as optimization, clustering, discrete Fourier transforms, linear algebra, signal processing and multi-dimensional image processing1.

1 https://stackoverflow.com/tags/scipy/info

http://www.gnu.org/software/gsl/

http://scipy.org/

https://stackoverflow.com/tags/scipy/info



7) SelfNet - https://selfnet-5g.eu

The SELFNET project was focused on designing and implementing an autonomic network management framework to achieve self-organizing capabilities in managing network infrastructures. It achieved this by automatically detecting and mitigating a range of common network problems which are currently still being manually addressed by network operators. As a result, operational costs were significantly reduced and the overall user experience was improved.

The SelfNet project aimed to create a novel intelligent network management framework capable of assisting network operators in key management tasks using a smart integration of state-of-the-art technologies in Software-Defined Networks (SDN), Network Function Virtualization (NFV), Self-Organizing Networks (SON), Cloud computing, Artificial intelligence, Quality of Experience (QoE) and Next-generation networking. In doing so, it provides automated network monitoring, Health of Network metrics and autonomic network maintenance. [6]

It largely deals with the SON paradigm SDN/NFV Orchestration with its use cases being focused on Self-Monitoring, Self-Protection, Self-Healing and Self-Optimization.

8) CogNet - http://www.cognet.5g-ppp.eu/

The CogNet project was focused on developing solutions that provide highly automated intelligent network monitoring and management. The project aimed to enable the larger and more dynamic network topologies necessary in 5G, improve end-user

QoS and lower capital and operational costs through improved efficiencies and the use of node, link and function virtualisation. [7]

CogNet applied Machine Learning algorithms to available network data to improve autonomic management of telecoms network infrastructure and yield insights, network conditions and to recognise and respond correctly to events when they occur. The project goal was to develop solutions that would provide a higher and more intelligent level of automated monitoring and management of networks and applications, improve operational efficiencies and facilitate the requirements of 5G.

The CogNet project presented 4 key pillars:

• Data collection - Data collection from network nodes.

• Service prediction & provisioning - service demand prediction and provisioning using ML algorithms.

• Network resilience – using Machine Learning algorithms.

• Security - anomaly detection algorithms to identify serious security issues.

Project research areas included data gathering, machine learning, virtualisation, data analytics and autonomic network management.

9) PaaSword - https://paasword.io/

The main aim of the PaaSword project is to protect users’ sensitive cloud data using holistic data privacy and security-by-design framework. The framework enhanced with context-aware access control mechanisms and is based on a searchable encryption scheme. PaaSword capitalises on recent innovations in virtual database middleware technologies, introducing a scalable secure cloud database abstraction layer with sophisticated data distribution and encryption methods, in order to extend the Cloud Security Alliance‘s cloud security principles [8].

https://selfnet-5g.eu/

http://www.cognet.5g-ppp.eu/

https://paasword.io/



Figure 16 PaaSword Overview [9]

PaaSword developed a novel approach to context-aware access control mechanisms which incorporate dynamically changing contextual information into access control policies and context-dependent access rights to data stored in the cloud to implement enterprise security governance in cloud environments. Another important feature of the PaaSword project is that it allows developers to specify the varying level of protection for the application‘s data. The PaaSword project states that it aims to provide a holistic PaaSword framework, Reference architecture, searchable encryption schemes for secure queries, policy access & context-aware security models, policy enforcement middleware and a dedicated IDE plug-in Thus allowing European enterprises to reap the economic and operational benefits of migrating to the cloud [10].

10) MATILDA - https://www.matilda-5g.eu/

Funded by Horizon 2020 5G-PPP Innovation, the goal of the MATILDA project was to design and implement a holistic 5G end-to-end services operational framework which tackled the lifecycle design, development and orchestration of 5G-ready applications and network services. The MATILDA project designed and developed a holistic framework that would improve the tight

https://www.matilda-5g.eu/



interconnection among the development of 5G-ready applications, the creation of the on-demand networking and computational infrastructure.

Figure 17 MATILDA project architectures overview [11]

This took the form of an application-aware network slice with the appropriate networking mechanisms to enable the correct support for industry vertical applications. MATILDA was focused on bringing a radical shift in virtual and physical network functions and network services as well as reconsidering the development of software for 5G-ready applications. The project aimed to do this by adopting a unified programmability model, the definition of proper abstractions and the creation of an open development environment that may be used by an application as well as network functions developers. [11] Automated placement of the 5G-ready applications will be applied through intelligent and unified orchestration mechanisms. Which will also handle the creation and maintenance of the required network slices. Optimisation mechanisms provide deployment and runtime policies enforcement, offering deployment plans based on high-level objectives. A multi-site NFV Orchestrator (NFVO) is employed to provide lifecycle management of supported Virtual Network Functions Forwarding Graphs (VNF-FGs) and network management activities.The key technological concepts and artefacts comprising the proposed MATILDA framework and laid out in the project as [11]

• The provision of 5G end-to-end services which tackle the overall lifecycle of design,

development and orchestration of 5G-ready applications and 5G network services over

programmable infrastructure.

• Meta-models to represent industry applications’ components and graphs as well as virtual -

and physical- network functions and forwarding graphs.

• Collaborative development environment for the design and development of 5G-ready

applications and VNF-FGs.

• A web-based IDE, verification and graphs composition mechanisms.

• Deployment and orchestration orchestrator for developed applications over the available

programmable.

• a multi-site virtual infrastructure manager and multi-site NFVO to support the multi-site

management of the allocated resources per network slice.

• Machine Learning enabled analytics and profiling framework.

• Applications and virtual network function marketplace.



11.3 Visual Composer Diagrams

As a first approach, got from the analysis of needs and requirements derived from Task 2.1, a set of wireframes, features and navigation options have been drafted. This will be the basis for the user interface to be implemented in WP5:

Figure 18 Service Composition Wireframe

The first approach is based on container-based service composition. This diagram (Service Composition) would read and write configuration files of the service. It would show every component and the relations. It would allow the composition of every node that would part of the final service scheme.

Figure 19 Basic field Wireframe



To generate the nodes, a set of basic fields will be provided, as also array and nested fields:

Figure 20 Array Wireframe

Figure 21 Object Wireframe

Below as follows, a sample of the GUI for the composition of a container is shown. On the right side, a sample of the configuration file generated is provided.



Figure 22 GUI WIREFRAME FOR CONTAINER COMPOSITION

The next diagram is showing the composition of a service based on lambda architecture. The service is based on five containers, that can be configured individually:

Figure 23 LAMBDA ARCHITECTURE-BASE SERVICE COMPOSITION



The next diagram is showing how a Kafka node could be configurated:

Figure 24 KAFKA NODE CONFIGURATION 1

Figure 25 KAFKA NODE CONFIGURATION 2

This would allow the generation of a universal user interface to compose services. Eventually, the config field could be used to create a RELIANCE slice intent.



12 Realizing RELIANCE architecture

3GPP based end-to-end network slicing is only available for 5G standalone (SA) architecture. Since 5G nodes have not been available from the beginning of the project we decided to follow a progressive approach from LTE to 5G.

12.1 LTE Radio Slicing

As an example, an instant messaging platform with integrated voice and video capabilities can be served by using the following architecture as shown in Figure 26. DRB and QCI parameters are used to implement slicing in the access network. eNB organizes and manages the traffic load depending on each slice concerning assigned parameters. Then, DSCP and pbit parameters are used in the transport network. Network slicing is partially implemented in the core network. This network slicing architecture enables service level priority setting at the radio layer for LTE networks.

Figure 26 Instant messaging service and its network slicing implementation

12.2 LTE Radio some dedicated EPC elements

VoLTE service can be provided using the following architecture. Network slicing is organized by using DRB & QCI, DSCP & pbit parameters for access and transport network, respectively. Dedication of some EPC elements can be seen in such network slicing implementa tion. MME, S-Gw and PCRF are used commonly for each slice. On the other hand, dedicated P-Gw are deployed in the core network.



Figure 27 A VoLTE service example

12.3 NSA architecture: LTE radio and New Radio with all dedicated EPC elements

As it is illustrated in the following architecture, network slicing is organized by using DRB & QCI, DSCP & pbit parameters for LTE radio, New Radio and transport network, respectively. Fully dedication of EPC elements depending on each slice is implemented in NSA architecture. PCRF, DNS and HSS are commonly used for network slices.

Figure 28 Network slicing architecture: LTE radio and New Radio



12.4 SA architecture: Standard-based network slicing with New Radio and 5G Core

Fifth-generation (5G) mobile systems shall enable operators to host vertical industry segments by introducing new services and enhance business collaborations between operators and third parties. The success of 5G hinges on the creation of new markets whereby operators can deliver customized assured services towards vertical segments in a cost-efficient manner. These vertical segments often have diverse and sometimes conflicting requirements. For instance, services involving self-driving vehicles require low latency and high bandwidth while an application for industrial internet can utilize a massive number of sensors and actuators which are low data rate but need to have long battery life. RELIANCE builds on the premise that future deployments need to stretch across different players that do not have an existing business relationship yet, thus calling for the creation of new dynamic trust models among them. Furthermore, to meet the specific requirements from vertical markets, the software network functions have to be highly adaptable and customizable, providing at the same time high service availability and reliability levels, regardless of the heterogeneous underlying infrastructure. With this motivation, new automated mechanisms for coordination between domains will be necessary, driving research work on multi-domain-supporting control, management and orchestration planes. New business models among infrastructure providers, service providers, vertical industries and relevant stakeholders will need to be defined, covering aspects relevant to multi-tenancy and network sharing. If instances of network and computing resources with related control functions can be grouped into a dedicated and customized network, an operator can approach new market segments through a “Slice as a Service” model. Accounting for the challenges to deploy network functions over different infrastructures, a unified reliability framework is required, developing a set of tools and strategies able to accommodate the expected service availability and reliability even with limited visibility into the underlying infrastructure. As an initial step to address the above challenges, the RELIANCE project conducts research activities on the concept of new resource abstraction based on the slate concept, which provides isolated resource share offerings within a domain for the slices. This new abstraction requires dynamic management procedures via which resources can be aggregated and partitioned to introduce unified control and management. The architectural work of the project is driven by the diverse needs and requirements stemming from new vertical enhanced services from several industries (e.g., automotive, industry 4.0, enhanced multimedia broadband, train control & management), designed to take advantage of the new capabilities derived from the new 5G context. A shown in Figure 29, the RELIANCE framework will be implemented and validated on selected use-cases through “proof-of-concept”s, aiming at demonstrating the applicability, feasibility, resilience, and benefits of slice management in the creation, provisioning and seamless management of a single domain in an operator network.



Figure 29 RELIANCE architecture: Standard-based network slicing with New Radio and 5G Core 3GPP has specified 5 possible configurations or ‘Options’ for connecting to an EPC or new 5G core network (6 if the current 4G system is included) for the introduction of 5G. The features that distinguish each option can be categorized as below:

• Usage of Dual Connectivity • RAT’s role as master node • Core Network usage

The options using dual connectivity are grouped under the term “Nonstandalone” (NSA) to indicate that 5G radio access technology (NR) and LTE are used simultaneously to provide radio access. On the other hand, the options where only one radio access technology is in use are referred to as “Standalone” (SA). It is widely expected that mobile operators will initially deploy 5G using Option 3 allowing the re-use of existing EPC Core functionality. Option 3 has been fully specified in an early drop of 3GPP Release 15. The other deployment options already fully specified by 3GPP are Option 2 and Option 5, both standalone options differing in the type of radio access technology connected to the new mobile core network 5GC. These options were completed in June 2018. Besides, 3GPP has finalized the two remaining Options, Option 4 and Option 7, completed in March 2019. Operators have planned their network deployment strategies based on what is included in the 3GPP specifications, and therefore reduction of the support of these options after standardization is not feasible. The 5GC has also been designed based on a fully modular Service Based Architecture (SBA) providing a cloud-optimized framework for flexible service creation, automation, scalability and resilience. The service-based interfaces for 5G System Architecture are illustrated in Figure 30:

Namf: Service-based interface exhibited by AMF.

Nsmf: Service-based interface exhibited by SMF.

Nnef: Service-based interface exhibited by NEF.

Npcf: Service-based interface exhibited by PCF.

Nudm: Service-based interface exhibited by UDM.

Naf: Service-based interface exhibited by AF.

Nnrf: Service-based interface exhibited by NRF.

Nnssf: Service-based interface exhibited by NSSF.



Nausf: Service-based interface exhibited by AUSF.

Nudr: Service-based interface exhibited by UDR.

Nudsf: Service-based interface exhibited by UDSF.

N5g-eir: Service-based interface exhibited by 5G-EIR.

Nnwdaf: Service-based interface exhibited by NWDAF.

The 5G System Architecture contains the following reference points:

N1: Reference point between the UE and the AMF.

N2: Reference point between the (R)AN and the AMF.

N3: Reference point between the (R)AN and the UPF

N4: Reference point between the SMF and the UPF.

N6: Reference point between the UPF and a Data Network.

N9: Reference point between two UPFs.

Figure 30 5G system architecture (Cisco Ultra 5G Packet Core Solution, White paper, 2018)

The 5G core network will be based on what is called “Service-Based Architecture” (SBA), centred around services that can register themselves and subscribe to other services. This enables a more flexible development of new services, as it becomes possible to connect to other components without introducing specific new interfaces. The new system architecture is specified in 3GPP technical specification 23.501. Service-based architectures have been in use in the software industry to improve the modularity of products. A software product can be broken down into communicating services. With this approach, the developers can mix and match services from different vendors into a single product.



Turkcell has deployed a testbed containing 5G SA core network to provide end to end network slicing implementation. The architecture is illustrated in

Figure 31.

Figure 31 Turkcell 5G SA Core Testbed Architecture

Use case execution based overall system architectures are illustrated in Figure 32, Figure 33, Figure

34, and Figure 35.

Figure 32 System overview of Turkcell Testbed for Video Conference Use Case (NETAS)



Figure 33 System overview of Turkcell Testbed for Industry 4.0 Use Case (KEYLAND)

Figure 34 System overview of Turkcell Testbed for Remote Maintenance with Volumetric Streaming Use Case (CONCATEL)



Figure 35 System overview of Turkcell Testbed for Train to Cloud (Wayside) Use Case (BOMBARDIER)

The testbed has the following features:

• Lifecycle Management of 5G NF

Instantiation on Kubernetes via Helm/kubectl

• Resiliency and Scaling of 5G NF with cloud-native architecture

• Resiliency Testing via chaosKube/VMWhackr - random termination of containers and hosting VM

Scale-in and Scale-out via Kubernetes

Recovery of the application after a catastrophic failure of Kubernetes cluster - time the

duration

• First call – AMF, SMF, UPF, NRF, NSSF (some Dev Sol and simulators will be used)

Service Integration with other lab nodes (3GPP interface integration, APN-DNS, etc)

Test Call

• Observability demonstration

OAM integration for counters/alarms/CALEA, etc.

Validate service healtchecks/liveness using Kubernetes built-in functionality

Prometheus Integration and config

Logging and Troubleshooting

• Software upgrade, Rollbacks and canary testing

Rolling software upgrade of an independent microservice via kubernetes

Software rollback of an independent microservice via kubernetes

Canary testing during upgrades

Install CI/CD environment and validate the full lifecycle of a release deployment/test

The testbed has a cloud-native architecture. Cloud-native is a fundamental change to the way

networks and applications are designed, implemented, deployed and operated. This architecture

provides the following capabilities:



• Cloud Native Observability Stack: We will monitor the KPIs by prometheus based API and

as a result of using these performance data, Grafana illustrates end to end monitoring. These

are typical examples: gauges, counters, histogram, debug, etc. For the tracing, we use

Jaeger.

• Centralized KPI Monitoring (Observability): All 5G KPIs monitored with Grafana. All

microservices, containers and PaaS elements ship stats to Prometheus. Visualization is

realized through Grafana. Alarms and threshold crossings can be used as an Integration

point to ONAP

• Distributed Tracing (Observability): Distributed transaction monitoring, Performance and

latency optimization, Root cause analysis, Service dependency analysis, Distributed context

propagation

• Centralized Logs (Observability)

12.4.1 Observability of KPIs in network slicing implementation

This section provides detailed information for the observability of KPIs in network slicing implementation. This can be useful for our vertical owners. For more information, ETSI TS 128 552 V15.3.0 (2019-06) document can be used as a reference document. Virtualised resource usage measurement

a. This measurement provides the mean usage of the virtualised resource (e.g. processor, memory, disk) in a single network slice instance during the granularity period.

b. OM c. This measurement is generated with .sum suffix for the usage of each virtualised NF

(see 3GPP TS 32.426) related to a single network slice instance by taking the weighted average. The algorithm of the weighted average is vendor-specific.

d. Each measurement is a real value (Unit:%). e. MeanProcessorUsage

MeanMemoryUsage MeanDiskUsage

f. Performance measurement service. g. Packet Switched. h. 5GS

Monitoring of UL and DL user plane latency in NGRAN Satisfying low latency expectations for 5G services, such as URLLC, is one of the key tasks for the operator to meet service performance expectations. As the performance in UL and DL differs, operators need to be able to monitor the UL and DL user plane latencies separately. With performance measurements allowing the operator to obtain or derive the UL and DL user plane latency information separately, the operators can pinpoint the services performance problems to specific problems in UL or DL. The DL IP latency monitoring in NG-RAN refers to the transmission within gNB of IP packets arriving when there is no other prior data to be transmitted to the same UE in the gNB. To further pinpoint performance problem detected, separate counters may be provided per mapped 5QI (which are particularly useful when the mapped 5QI is used by a few services and users and the packet size does not vary much). Monitoring of UL and DL packet loss in NG-RAN Keeping track of UL and DL packet loss in the NG-RAN is essential since for certain services packets that are lost along the way through the system may have a noticeable impact on the end-user. UL



and DL packet loss measurements can be useful for evaluation, optimization and for performance assurance within the integrity area (user plane connection quality). UL packet loss is a measure of packets dropped in the UE and the packets lost on the interfaces (air interface and F1-U interface). If parts of the gNB are deployed in a virtualized environment, it is important to measure also the F1-U UL interface packet loss in a separate measurement, to be able to pinpoint the reason for high packet loss.

Monitoring of DL packet drop in NG-RAN Keeping track of DL packet drops in the NG-RAN is essential since for certain services packets that are dropped along the way through the system may have a noticeable impact on the end-user. DL packet drop measurements can be useful for evaluation, optimization and for performance assurance of the network. For gNBs that are deployed in a split architecture, e.g. when parts of a gNB are deployed in a virtualized environment, the DL packet drops may occur in two parts; the gNB CU-UP and the gNB DU. Therefore, it is important to measure this separately.

Monitoring of UL and DL user plane delay in NGRAN Satisfying low packet delay is of prime concern for some services, particularly conversational services like speech and instant messaging. As the performance in UL and DL differs, operators need to be able to monitor the UL and DL user plane delay separately. With performance measurements allowing the operator to obtain or derive the UL and DL user plane delay information separately, the operators can pinpoint the services performance problems to specific problems in UL or DL. The DL delay monitoring in gNB refers to the average delay of any packet within NG-RAN, including air interface delay until the UE receives the packet. A gNB deployed in a split architecture, the user plane delay will occur in gNBCU-UP, on the F1 interface, in gNB-DU and on the air interface. Therefore, four gNB related delay measurements need to be monitored for the DL delay to pinpoint where the end-user impact from packet delay occurs.

Monitoring of UE Context Release Request (gNB-DU initiated) In order to monitor the stability of the network and detect the service/connection interruption caused by NGRAN, monitoring the UE Context Release Request initiated by gNB-DU is an effective method. Collecting the measurement information of the message and analysing the releasing cause conveyed in the message, operators could detect the stability of NG-RAN, and could decide a specific means to improve the NG-RAN performance.

Monitoring of physical radio resource utilization The physical radio resource utilization measurements could provide operators with the load information of the radio network during the measurement time. The physical radio resource utilization measurements should reflect the average usage and the usage distribution of the radio resource of the physical layer. The measurements can make the operator be aware of whether a cell has ever experienced high load or not in the monitoring period, and is a key input to network capacity planning and load balancing. Monitoring of RRC connection number The number of the users in RRC connected and inactive mode need to be monitored as it reflects the load of the radio network, the operators can use this information for dynamic frequency resource allocation or load balance purpose. Moreover, it is an important factor to be evaluated in the radio network capacity enhancement decision-making. Monitoring of UE Context Release To monitor the stability of the network and detect the service/connection interruption caused by NG-RAN, monitoring the UE Context Release Request initiated by gNB-DU and UE Context Release Command initiated by gNBCU is an effective method. Collecting the measurement information of the message and analysing the releasing cause conveyed in the message, operators could detect the stability of NG-RAN, and could decide a specific means to improve the NG-RAN performance.



Monitoring of UE Throughput in NG-RAN Keeping track of UL and DL UE throughput in the NG-RAN is essential, to ensure end-user satisfaction and well-functioning and well-configured cells and scheduling features. The restricted UE throughput per mapped 5QI will show the scheduling efficiency and QoS priority handling in the gNB and the ratio between unrestricted and restricted volume will show the gNB ability to handle small data transfers efficiently. To be able to monitor the spread of throughput within the cell, and estimate the ratio of satisfied users, the throughput distribution measurement can be used. When network slicing is supported by the NG-RAN, multiple NSIs may be supported. The UL and DL UE throughput for each NSI is then of importance to the operator to pinpoint a specific performance problem. Monitoring of Unrestricted volume in NG-RAN Measuring the share of unrestricted user data volume in the NG-RAN is important, to show the gNB ability to handle small data transfers efficiently and to see how large a share of the volume is part of the UE throughput measurement. It is not meant to measure throughput for data transfers so small that they fit in one single slot but it is still important to know how much such transfers can be handled by the gNB. When network slicing is supported by the NG-RAN, multiple NSIs may be supported. The share of unrestricted volume for each NSI is then of importance to the operator to pinpoint a specific performance problem. N3 data volume-related measurements N3 related measurements are used to measure data volume on the N3 interface including incoming and outgoing GTP data packets without counting the mandatory part of the GTP-U header It is useful to analyse transport bandwidth usage of the N3 interface. If the transport bandwidth usage is too high, more bandwidth should be deployed, or load balance should be considered according to core network dimension if there are multiple UPFs connected to multiple gNodeBs. So it is necessary to define N3 related measurements. N6 related measurements N6 related measurements are used to measure data volume on the N6 interface including incoming and outgoing IP data packets. It is useful to analyse transport bandwidth usage of the N6 interface. If the transport bandwidth usage is too high, more bandwidth should be deployed. So it is necessary to define N6 related measurements. Registration related measurements A UE needs to register with the 5GS to get authorization to receive services, to enable mobility tracking and to enable reachability. The following registration types are defined:

- Initial Registration to the 5GS; - Mobility Registration Update (upon changing to a new Tracking Area (TA) outside the UE's

Registration Area in both CM-CONNECTED and CM-IDLE state, or when the UE needs to update its capabilities or protocol parameters that are negotiated in Registration procedure with or without changing to a new TA);

- Periodic Registration Update (due to a predefined period of inactivity); and - Emergency Registration (i.e. the UE is in a limited-service state).

The performance of registration for each registration type needs to be monitored by the operator since it is relevant to whether the end-user can use the service of 5GS or a specific network slice. PDU session establishment related measurements The PDU session establishment is one of the essential procedures for the 5G network. The performance of PDU session establishment directly impacts the QoS of the network and the QoE of the end-users. Therefore, the performance measurements are needed to reflect the performance of the PDU session establishment. The number and success rate of PDU session creations, the number of PDU sessions running on the SMF are some of the basic performance measurements to monitor the performance of the PDU session establishment. And the performance measurements of failed PDU session creations are helpful to solve the network issues in case the performance is below the expectation. Policy association related measurements To ensure the UE properly use the services provided by 5GS, the UE needs to be associated with a policy. The policies are categorized into AM policy and SM policy and are executed by AMF and SMF



respectively. Both kinds of policies are provisioned by PCF. The AM policy association needs to be established in case the UE initially registers to the network or the UE needs the AMF re-allocation. The SM policy association needs to be established when the UE requests a PDU Session Establishment. The policy association establishment is the essential steps allowing the UE to be served by the 5GS under the designed policies, therefore it needs to be monitored.

12.4.2 Network Slicing in LTE Testbed

A demonstration of network slicing in LTE network using NETAŞ VIO Video Conferencing Application and ULAK MAYA SDN Controller had been made. Figure 36 describes the network nodes in the network topology in Figure 37.

Network Node Role

VIO CLIENT 1 Mobile phone using VIO

VIO CLIENT 2 Mobile phone using VIO

eNodeB ULAK 4G Base Station

PHYSICAL SWITCH The physical device represents the transport network and contains several OvS bridges.

EPC (Evolved Packet Core) Evolved Packet Core, containing SGW and PGW

SPGW Serving Gateway (SGW) and Packet Data Network Gateway (PGW) deployed apart from EPC.

NETAŞ VIO SERVER Video Conferencing Server

ULAK MAYA SDN Controller

Figure 36 Network Nodes in LTE Slicing Demo

Figure 37 Network Diagram for LTE Slicing Demo

In this setting VIO Client 1, is routed towards EPC and VIO Client 2, is routed towards SPGW. In this way, the two clients reach the data network. This means that each client is served in separate slices and with different QoS parameters.



DSCP tag, source IP and destination IP addresses can be used to classify flows into slices such that different QoS parameters can be provided for each slice. In this study, considering that each slice has its unique SGW and PGW table, the destination IP address is used for classification. In this manner, 172.19.0.10 is the destination IP address for the slice containing VIO CLIENT 1 and 172.19.0.60 is the destination IP address for the slice containing VIO CLIENT 2. QoS rules are defined with these criteria. Queue definitions have been made in OvS switches to apply the QoS parameters of network slices. Each queue can be defined with parameters like maximum bandwidth, minimum bandwidth and weight. Later, the network flows are directed towards these queues such that the flows are transmitted with the QoS limits in the selected queue. In this study, two queues are defined OvS switches as in Figure 38.

Queue Code Maximum Bandwidth

Q0 (Default) No Limit

Q1 100 Kbps

Figure 38 OvS Queue Configuration After queue definitions, using ULAK MAYA SDN Controller CLI one of the rules shown in Figure 39 are selected for the slice that is used by VIO CLIENT 1. In the experiments, when the first rule in the table is chosen, VIO CLIENT 1 traffic is transmitted only with the limitations of the available resources. However, when the second rule is chosen VIO CLIENT 1 traffic is transmitted with 100 Kbps bandwidth. On the other slice, VIO CLIENT 2 traffic is transmitted only with the limitations of the available resources again due to no rules to limit the second slice.

Target QoS Parameter Switch Destination IP Action

No Bandwidth Limitation S1 172.19.0.10 Q0, Normal

Maximum 100 Kbps Bandwidth S1 172.19.0.10 Q1, Normal

Figure 39 QoS rules defined with ULAK MAYA NBI

In the demonstration of LTE Network Slicing the following scenario is executed:

1. VIO CLIENT 1, VIO CLIENT 2 and a PC are connected to the same VIO conference room (Figure 40).

2. In the beginning, both slices use the default QoS parameter. Therefore all video captures are transmitted with normal speed and no abnormalities are observed.

3. The second rule in Figure 39 is defined for the slice containing VIO CLIENT 1 traffic. In this case, the video motion of VIO CLIENT 1 is transmitted with 100 Kbps speed to VIO server and other clients. However, video motion of VIO CLIENT 2 and PC are transmitted and observed with normal speed.

4. The first rule in Figure 39 is again defined for the slice containing VIO CLIENT 1. In this case, the video motion of VIO CLIENT 1 turned back to normal speed in the video conference room. Other user’s video is not affected and maintained normal speed. As shown in Figure 41, Figure 42 and Figure 43 this step is taken after 12:26.



Figure 40 Screenshot from NETAŞ VIO Application

The video motion of VIO CLIENT 1 changes from 10 FPS to around 25 FPS as seen in Figure 41. Figure 42 shows the bandwidth observed which increases from below 100 Kbps up to around 600 Kbps. Figure 43 shows the number of received packets which increases from around 10 to nearly 100 packets.

Figure 41 FPS received from VIO CLIENT 1 Figure 42 Bits received from VIO CLIENT 1 per second

Figure 43 Packets received from VIO CLIENT 1 per second

The demonstration described above in LTE network shows the network slicing realized in the forwarding plane using SDN capabilities and performance isolation of slices was also highlighted.



13 Conclusion

The Celtic-Plus RELIANCE project extends the 5G network architecture with new functionalities needed for multi-service management, with the related abstractions, interfaces, and mechanisms to support needed resilience, security, and scalability. A key innovation of the project is the highly scalable and robust slice choreography plane that enables the offering of vertical-tailored slices “as-a-Service”, ensuring the required scalability, security, and resilience features. It facilitates the mapping of service requirements from the verticals with their predicted traffic demands into dedicated networking and cloud resources through automated multiparty negotiation. Scalable slicing is complemented with protocols for deployment and runtime adaptation of resilience and security mechanisms over heterogeneous infrastructures, so to enable a unified reliability framework.

This deliverable is linked to Task 2.2. The main goal of Task 2.2 is to define a high-level architecture addressing the existing limitations for the provision of slicing capabilities across multiple domains concerning the management, control, access, leasing, federation and interconnection of slates for a set of well-defined and business-driven use cases. The D2.5 deliverable is the final version RELIANCE reference architecture specification. In this deliverable, we have presented the final version of the RELIANCE architecture. We have in particular (i) provided more detail about the 3GPP 5G Core and (ii) explained the Turkcell 5G SA Core testbed.



References

[1] IBM Emerging Technologies, “IBM Emerging Technologies,” [Online]. Available:

https://emerging-technology.co.uk/.

[2] Open JS Foundation, “Open JS Foundation,” The Linux Foundation Projects, [Online].

Available: https://openjsf.org/.

[3] NodeRed.org, “https://nodered.org/,” https://nodered.org/, [Online]. Available:

https://nodered.org/.

[4] J. P. R. Morrison, “Flow-Based Programming: A New Approach to Application Development,”

New York: Van Nostrand Reinhold, 1994.

[5] J. P. Morrison, “Flow-based Programming,” https://jpaulm.github.io/, [Online]. Available:

https://jpaulm.github.io/fbp/concepts.html.

[6] https://nodered.org/, “Running on Raspberry Pi,” https://nodered.org/, [Online]. Available:

https://nodered.org/docs/getting-started/raspberrypi.

[7] nodered.org, “Running on BeagleBone Boards,” nodered.org, [Online]. Available:

https://nodered.org/docs/getting-started/beaglebone.

[8] nodered.org, “Interacting with Arduino,” nodered.org, [Online]. Available:

https://nodered.org/docs/faq/interacting-with-arduino.

[9] nodered.org, “Running on Android,” nodered.org, [Online]. Available:

https://nodered.org/docs/getting-started/android.

[10] Canonical, “Juju as a Service,” Canonical, [Online]. Available: https://jaas.ai/jaas.

[11] E. F. A. Z. C. V. Panagiotis Gouvasa, “A Context Model and Policies Management Framework

for Reconfigurable-by-design Distributed Applications,” Procedia Computer Science, vol. 97,

p. 122 – 125, 2019.

[12] Ubitech, “The first press release of the ARCADIA H2020 project, wherein UBITECH has the

scientific and technical lead, is published,” Ubitech, [Online]. Available:

https://www.ubitech.eu/the-first-press-release-of-the-arcadia-h2020-project-wherein-ubitech-

has-the-scientific-and-technical-lead-is-published/.

[13] P. Gouvas, E. Fotopoulou, A. Zafeiropoulos, C. Vassilakis, “A Context Model and Policies

Management Framework for Reconfigurable-by-design Distributed Applications,” in Cloud

Forward, Madrid, Spain, 2016.

[14] Docker.com, “What is a Container,” Docker.com, [Online]. Available:

https://www.docker.com/resources/what-container.

[15] Docker.com, “Docker Windows Containers,” Docker.com, [Online]. Available:

https://www.docker.com/products/windows-containers.

[16] https://yaml.org/, “YAML Yamil Ain't Markup Language,” https://yaml.org/, [Online]. Available:

https://yaml.org/.

[17] Open Container Initiative, “About,” OCI, [Online]. Available:

https://www.opencontainers.org/about.



[18] Cloud Native Computing Foundation, “Cloud Native Computing Foundation,” Cloud Native

Computing Foundation, [Online]. Available: https://www.cncf.io/.

[19] kubernetes.io, “What is Kubernetes,” kubernetes.io, [Online]. Available:

https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/.

[20] RedHat, “Red Hat Enterprise Linux,” ResHat, [Online]. Available:

https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux.

[21] RedHat, “CoreOS,” RedHat, [Online]. Available: https://www.openshift.com/learn/coreos/.

[22] RedHat, “OpenShift Online,” RedHat, [Online]. Available:

https://www.openshift.com/products/online/.

[23] puppet.com, [Online]. Available: https://forge.puppet.com/.

[24] puppet.com, “Open Source Puppet,” puppet.com, [Online]. Available: https://puppet.com/open-

source/#osp.

[25] OASIS, “OASIS Topology and Orchestration Specification for Cloud Applications (TOSCA)

TC,” OASIS, [Online]. Available: https://www.oasis-

open.org/committees/tc_home.php?wg_abbrev=tosca.

[26] Data Modelling Language Working Group (NETMOD), “Network Modeling (netmod),”

NETMOD, [Online]. Available: https://datatracker.ietf.org/wg/netmod/about/.

[27] paessler.com, “SNMP, MIBs and OIDs – an overview,” paessler.com, [Online]. Available:

https://www.paessler.com/info/snmp_mibs_and_oids_an_overview.

[28] netconfcentral.org, “YANG Data Modeling Language,” netconfcentral.org, [Online]. Available:

http://www.netconfcentral.org/yang_docs.

[29] ETSI TS 138 521-1 V15.3.0 (2019-07), Technical Specification of 5G; NR; User Equipment

(UE) conformance specification; Radio transmission and reception; Part 1: Range 1

standalone, [Online]. Available: https://www.etsi.org/.

[30] ETSI TS 132 426 V16.0.0 (2020-08), Technical Specification of LTE; Telecommunication

management; Performance Management (PM); Performance measurements Evolved Packet

Core (EPC) network, [Online]. Available: https://www.etsi.org/.