GSTP Element 1 “Develop” Compendium 2019: Artificial ...emits.sso.esa.int › emits-doc › ESTEC › News › ... · Artificial Intelligence activities of strategic importance

Prepared by TEC-T Reference ESA-TECT-PL-015679 Issue 1 Revision 0 Date of Issue 28/10/2019 Status Document Type

Distribution

ESTEC

European Space Research and Technology Centre

Keplerlaan 1 2201 AZ Noordwijk

The Netherlands

T +31 (0)71 565 6565 F +31 (0)71 565 6040

www.esa.int

GSTP Element 1 “Develop” Compendium 2019: Artificial Intelligence

Page 2/28


Date 28/10/2019 Issue 1 Rev 0

Title GSTP Element 1 “Develop” Compendium 2019: Artificial Intelligence

Issue 1 Revision 0

Author TEC-T Date 28/10/2019

Approved by

Date

Reason for change Issue Revision Date

Issue 1 Revision 1

Reason for change Date Pages Paragraph(s)

Page 3/28



Table of contents:

1. INTRODUCTION ......................................................................................................................... 42. LIST OF ACTIVITIES .................................................................................................................. 63. DESCRIPTION ............................................................................................................................ 7

CD3 - Avionic Architecture / DHS / Onboard SW / (FDIR) / GNC + AOCS / TT&C (E2E) .............................. 7 CD9 - Digital Engineering for Space Missions ................................................................................................... 23

Page 4/28



1. INTRODUCTION

The GSTP E1 “Develop” Compendium 2019: Artificial Intelligence, is a list of candidate activities for the GSTP E1 “Develop” Work Plan. The aim of the GSTP E1 “Develop” Compendium 2019: Artificial Intelligence, is to provide to industry and Delegations a consolidated overview by Competence Domain of the priorities in the development of Artificial Intelligence technologies within the GSTP Programme. This document follows the previous GSTP Element 1 Compendia of Potential Activities (2013-2017) and complements the GSTP E1 Compendium 2019 for Generic Technologies (ESA-TECT-PL-015884). This compendium is issued to Delegations of GSTP Participating States and their industries for comments. Such comments will be considered in the following updates of the work plan for this GSTP Element 1 “Develop”. The objective is to have a good indication of the developments the participants intend to support in order to present updates of the GSTP E1 “Develop” Work Plan with consolidated set of activities to the IPC for approval. Artificial Intelligence activities of strategic importance for the future development of space system have been identified and form the basis of the compendium. A Call for Ideas from external parties, particularly Industry and Academia, who are interested in technology focussed on Artificial Intelligence was launched during the 2nd quarter of 2019. The most innovative ideas submitted to the Open Space Innovation Platform (OSIP), with the widest potential impact in space applications, have been used to formulate detailed activity descriptions and included in this compendium. AI have been sometime wrongly interpreted as a new software language or technique, instead AI is a new logic that will substitute the traditional way of programming, substituting the traditional software with algorithms that mimic human brain processes. This new approach can be applied in all domains for which at the moment traditional software automation is applied. In the space application AI could be a game-changer that is gathering momentum in space activities. It can be an enabling technology for future missions and an added value to increase scientific return and efficiency of current missions. Definitely AI has a key role for enabling future operations concepts and mission scenarios, for enhancing the degree of autonomy, both on-board and on-ground, to synergize the processes of spacecraft design, test and operations. In the automation operation area AI techniques, including machine learning, and automated planning, can definitely become an enhancer and enabler for future missions – e.g. adopting higher degree of command and control autonomy - or be a game changer in operating complex missions, such as earth observations constellations (both ESA and commercial), communication commercial constellations, challenging deep space scenarios such as the Space Gateway infrastructure. Implementation AI algorithms into space project can have significantly influence on attitude to:

0perational risks mitigation, operations automation and autonomy, operations cost reduction, increased (science) mission return,

AI can support and optimize operations in current and future missions, providing planning and scheduling advanced tools for mission control and GS management, anomaly detection, predictions, intelligent visualization tool in support of S/C engineers. Many potential applications can benefit from AI capabilities at different levels:

Smart payload data Our satellites are filled with payloads continuously producing large amount of data. Be it for remote sensing, for space environment or for astronomy this enormous flux of information contains essential information on our Planet, our Solar System and our Universe. Machine learning techniques are a perfect fit to improve the use we make of these data, to create new

Page 5/28



products or simply to help select on-board what observations are actually worth transmitting back to us.

AI in Data Exploitation AI can contribute to better exploit the scientific data coming from spacecraft, either being related to earth observation, space science, navigation science, telecommunication. Processes like extraction, identification and classification of information and knowledge from the received data will have enormous boost in terms of processed volume of data and new features

AI in Operations AI can be of great help to support and optimize operations, help in analysing trade-off alternative scenarios, their feasibility and limitation in terms of resources, constraints, performance. For each operations application domain AI will provide its added value, either in novel features, functions, in cost reduction, in risk mitigation.

Guidance Navigation and Control (GNC): Artificial intelligence scientists have proposed, in the last decade, numerous ideas on how to make Earth robotics systems smarter. From perception (navigation) to cognition (guidance) to action (control) AI systems based on deep networks, on reinforcement learning principles and several other smart techniques have proven their worth in several cases by now.

Edge/On board AI: Upcoming space missions are requiring a higher degree of on-board autonomy operations to increase quality science return, to minimize close-loop space-ground decision making, and to enable new operational scenarios. Artificial Intelligence technologies like Machine Learning and Automated Planning are becoming more and more popular as they can support data analytics conducted directly on-board as input for the on-board decision making system that generates plans or updates them while being executed.

The spin in of this technology will eventually lead to a breakthrough in data processing thus relieving requirements from other platform systems (as it needs to be assessed). It will potentially allow concentration of the data acquired over multiple sensors reducing the required number of electronic units, saving mass, power, volume and reducing harness complexity. Technology is by no way restricted in one domain, as it is applicable to both platform and payload sides allowing to provide, except for its main purpose, useful by-products.

Page 6/28



2. LIST OF ACTIVITIES

GEN - Generic Technologies – Artificial Intelligence

CD3 - Avionic Architecture / DHS / On-board SW / (FDIR) / GNC + AOCS / TT&C (E2E)

Programme Reference

Activity Title Budget (k€)

Edge/On board AI

GT1I-301ED Machine Learning-based processing for star trackers 600

GT1I-302ED Machine learning application benchmarking on COTS inference processors. 600

GT1I-303ED Complexity reduction for optimized lightweight on-board AI inference 600

GT1I-304ED Machine Learning-based on board autonomy, failure prognostics and detection.

800

GT1I-305ED AI for non mission critical on board data processing 1,000

GT1I-306ED Robust machine learning systems for dependable space applications 600

Guidance Navigation and Control (GNC)

GT1I-307SA Training datasets generation for machine learning: application to vision-based navigation

400

GT1I-308SA Development of distributed autonomous trajectory control 400

GT1I-309SA 3D shape reconstruction assisted by machine learning techniques 200

Total 5,200

CD9 - Digital Engineering for Space Missions

Programme Reference

Activity Title Budget (k€)

AI in Operations

GT1I-310OS Machine learning prediction for safer spacecraft operations and increased science return

350

GT1I-311OS AI learning services for space systems operations 700

GT1I-312OS Online deep learning for anomaly detection & isolation: from V&V to final operations

500

GT1I-313OS Autonomous AI-based satellite command & control for large number of cooperating spacecraft

750

GT1I-314OS Artificial Intelligence for large fleet network management 750

Total 3,050

Page 7/28



3. DESCRIPTION

3.1 GEN - Generic Technologies – Artificial Intelligence

CD3 - Avionic Architecture / DHS / Onboard SW / (FDIR) / GNC + AOCS / TT&C (E2E)

Domain Artificial Intelligence CD3 - Avionic Architecture / DHS / Onboard SW / (FDIR) / GNC + AOCS / TT&C (E2E)

Ref. Number: GT1I-301ED Budget (k€): 600

Title: Machine Learning-based processing for star trackers

Objectives: The objective of this activity is to develop a Machine Learning (ML) based star tracker processing prototype for precise spacecraft attitude determination. Existing off the shelf (OTS) camera head units will be coupled with a demonstrator electronics and integrated control hardware and software. This activity will increase the know-how in building dependable platform instruments based on ML.

Description: The aim of the activity is to demonstrate the ability of ML-based processing systems to achieve high dependability in platform applications. In this activity, star pattern recognition through machine learning techniques applied to current model-generated images and space star-tracker sensors will be use case. The machine learning star pattern match algorithm utilized will employ a standard star catalogue and get real-time comparisons of the star tracker observed motion with the rotational motion of the Earth. This new technique of star pattern encoding will remove the star magnitude dependency, increase the number of tracked stars (from the current ~10 to more than 100, without changing sensors/optics). The machine learning recognition will potentially be able to perform data fusion on multiple heads and treat the merged image as a set of triangles, thus automatically recognizing constellations and removing moving objects (radiation noise, planets, glares, etc). Machine learning image recognition techniques applied to star tracking do not depend on star magnitudes and allow a varying number of stars to be identified and used in calculating the attitude quaternion. This technique combined with feed-forward neural network pattern identification creates a robust and fast technique for greatly increasing STR performances without changes in STR optics. As indeed robustness is paramount for the application, machine learning techniques will be applied at different object levels, and the correlation between the features extracted at those diverse levels will eliminate the false indicators that might diminish the accuracy of the STR tasks. The proposed parameters to achieve that are: increased reference stars detection (one order of magnitude more efficient that current STRs), group of stars detection (like the human eye does with the classic constellations), correlation between the positions of the detected objects, faster and more efficient star-catalogue comparison. All-in-all, it is tested to have an order of magnitude increase in performances in all STR key characteristics as acquisition from lost in space, maximum rotation rate (rates up to 20 deg/s might be possible), with quaternion precision up to 0.1 arc/sec. This activity will help in developing techniques for formally proving that machine learning models are specification-consistent. There is a need for algorithms that can verify that the model predictions are provably consistent with a specification of interest for all possible inputs. While the field of formal verification has studied

Page 8/28



such algorithms for several decades, these approaches do not easily scale to modern deep learning systems despite impressive progress. The tasks performed in this activity will be:

Analysis of multi-level object recognition techniques (within the deep learning process), through maximizing recall on stars, constellations and their inter-relation (for increased reliability);

Final implementation of the system in a representative on-board data processing platform.

Demonstration of system dependability with comparison with classical implementation

Deliverables: Breadboard

Current TRL: 2 Target TRL: 5 Duration (months):

12

Target Application/ Timeframe:

All mission, but particularly interesting for missions with very variable environment like LEO to GEO (EOR) and planetary exploration.

Applicable THAG Roadmap: Not relevant to a Harmonisation topic.

Page 9/28





Title: Machine learning application benchmarking on COTS inference processors.

Objectives: The objective of this activity is to design and develop of benchmarks to assess the performance of AI-related algorithms, computer vision and machine learning, using platforms that have a good TRL with a possibility of almost immediate space use.

Description: The performance of the algorithms assessed during this activity will enable practical usage of the selected processors on board of satellites for Earth Observation (EO) data processing and Vision Based Navigation (VBN) tasks, as well as typical generic platform control applications. The activity target is to build fair and useful benchmarks for measuring training and inference performance of ML hardware, software, and services. A widely accepted benchmark suite will benefit the entire community, including researchers, developers, hardware manufacturers, builders of machine learning frameworks, cloud service providers, application providers, and end users. Among targeted platforms there are:

COTS FPGA from XILINX, using machine learning HLS tools GPUs and Tensor processors (NVIDIA, Intel) Intel Myriad2 and X processors. Space grade processors and FPGA (Xilinx, BRAVE, GR740)

These algorithms are selected to be representative for tasks that are needed for future space missions (e.g. EO image processing, VBN, Super-resolution, NN-based image classification for payload application, but also non-linear control and time series analysis for platform applications). Our goals include:

Accelerate progress in ML via fair and useful measurement; Serve both the commercial and research communities; Enable fair comparison of competing systems yet encourage innovation to

improve the state-of-the-art of ML; Enforce replicability to ensure reliable results; Keep benchmarking effort affordable so all can participate.

The technical objectives to be reached at the end of the GSTP; Set of CV algorithms leveraging target processors capabilities to be used in

EO images processing and VBN; Efficient and reliable method of CV/ML tasks management on the target

board; Algorithms that can be developed/assessed:

Change detection in time series of Earth Observation images, various resolutions;

Template matching (scale and rotation invariant) in Earth Observation -type image (e.g. from Sentinel-2);

Supervised NN Image Classification of Multi-Spectral Images Based on Statistical Machine Learning (TBD if learning speed should be measured by benchmark as well);

Page 10/28



Cloud detection algorithm (Fmask or Sen2Cor, however the whole Sen2Cor is quite big, maybe some essential parts of it)

Increase resolution of all Sentinel-2 bands to 10m/pixel; Reconstruction involving multiple Images alignment using SURF

equivalent, like BRISK or ORB (SURF is patented) and RANSAC; Super-resolution (increase resolution using series of images) through

compressive sensing methods, like overdetermined equations; Image compression (jpeg/CCSDS), (preferably Earth Observation - like

picture). The following tasks are planned for this activity:

Develop and optimize a set of the most useful Computer Vision (CV)/ML algorithms to be used in EO image processing and VBN tasks for a set of target boards/platforms;

Develop an efficient method of dispatching CV tasks to target processors through (for example) interface FPGA and results verification.

Deliverables: Benchmark Application


12


All missions, in particular satellites for EO data processing and visual based navigation tasks as well as generic platform control applications.


Page 11/28





Title: Complexity reduction for optimized lightweight on-board AI inference

Objectives: The objective of this activity is to define and develop a set of tools and a workflow in order to enable the inference of Artificial Neural Networks (on future space) missions.

Description: The main goal is to achieve sufficient compression ratios (20-50:1 and even 1000:1 in some cases) in order to obtain lightweight enough models compatible with space qualified hardware, in particular BRAVE, Xilinx and Microsemi FPGAs (but also CPUs or SoCs). In the last years Artificial Neural Networks have been proven to outperform traditional algorithms in many different processing tasks, in particular image processing, classification and feature extraction. However, to achieve state-of-the-art performances very complex models are usually required. This is a fundamental enabling technology in order to ease the full operational adoption of AI in future space missions, in this way High Level Synthesis FPGA tools can be used to quickly create optimized FPGA/ASIC HW for a Neural Network Inference Solution, using radiation tolerant components and these models can be updated on operational missions in a much more agile way. Unfortunately, is technically impossible to implement or even upload such complex networks (millions of computations and hundreds of MB in memory footprint) in space qualified hardware, while meeting the requirements (notably in terms of throughput) of most of our space missions. Existing Deep Learning frameworks do not tackle deep enough the optimization of the model complexity, and only provide integer quantization capabilities (or at must some limited pruning) in order to ease the hardware implementations. Given the limited resources of the space-qualified processing units, it is crucial to take advantage of all the existing techniques for the complexity reduction of the models (quantization, pruning, model compression, approximate computing...). The combined used of those techniques can achieve compression ratios and inference speed-ups in the range of 20-50 without introducing noticeable impacts in the quality of the results (in some cases, compression ratios up to 1000:1 can be obtained). On the other hand, those light-weighted less-redundant models can be more severely affected in the presence of faults. Thus, the tool will also need to take care of the dependability of the network and provide the necessary capabilities to trade-off the complexity reduction vs. the robustness of the network. The designed workflow would also need to take into account the characteristics of the target device in terms of architecture, type of resources (FPGA) and/or instruction set (CPU,...) as well as the integration with existing (or future) HLS frameworks for VHDL synthesis (including current end-to-end definition-to-inference tools like nvDLA, xDNN Xilinx, N2D2 CEA-List,…). The principal tasks that will be performed during the activity can be summarized as follows:

Analysis of the current state of the art regarding Deep NeuDNN implementation on FPGA and processors;

Preliminary requirements definition for the framework being developed; Identification of representative cases to be considered;

Page 12/28



Implementation and use of a demonstrator of the framework; Tests execution and preliminary results assessment; Definition of the way forward.

Deliverables: Software (VHDL design), Report


12


Enabling operational AI on future missions, in particular for Earth Observation and Space Science and Earth Observation constellations.


Page 13/28





Title: Machine Learning-based on board autonomy, failure prognostics and detection

Objectives: The objective of this activity is to design, test and prototype a generic and reusable deep learning approach for both anomaly detection & isolation as well as failure prognostics.

Description: A critical function on board all spacecraft is the Failure Detection, Isolation and Recovery (FDIR) subsystem, which is vital in ensuring the safety, autonomy and availability of the system. Modern satellites complexity is increasing, and these highly complex systems require bespoke FDIR solutions with complicated architectures and difficult testability. Furthermore, traditional FDIR techniques are generally good at detecting 'single' failures but limited in isolation capabilities, and struggling when multiple faults combine in non foreseen behaviours. The resulting functional unavailability and operational costs can be prohibitive – for example – for large fleets or constellation. The solutions developed in this activity offer the benefit of moving from a constant data downstream of all parameters (data-pull) with threshold-based alerting to a predictive alerting system that can handle seasonal effects, system noise and data correlations (information-push) scalable to very large fleets of satellites. These functionalities can be migrated from ground on-board the spacecraft, as the algorithms are very lean, require only little computational power and are highly energy-efficient. This activity aims at demonstrating that a highly reusable, generic FDIR solution can perform satisfactorily in flight, streamlining the current FDIR development process in both time and cost whilst increasing the autonomy and availability of the next generation of spacecraft. Furthermore, it offers serious contributions towards raising the TRL of neural network implementations on space-qualified HW to flight SW standards, setting a precedent for future innovative machine learning-based applications in on-board functional avionics. By implementing embedded-AI-based analysis of on board TM time series, the failure detection results can be applied not only to trigger alarms when a fault occurs, but also for preventive maintenance. This applies both to stand alone electronics (eg high performance COTS FPGA, uC) to generic units, assemblies, systems. The key feature of the proposed activity is the real-time condition monitoring of all spacecraft parameters. The algorithms learn from the machine itself by training on-ground during simulation/testing (e.g. hardware-in-the-loop) of the spacecraft to identify patterns in the data that characterize nominal operations. The observed patterns are used as a reference when compared to instantaneous data. In case unusual patterns are detected, the operator is informed about identified deviations, significant statistics and correlated patterns for further analysis. A deviation manifests earlier than the fault itself. Thus, appropriate actions can be taken not only to recover from faults, but also to prevent faults from occurring or to reduce their impact. Main tasks:

Selection of target spacecraft(s) + use case elaboration; Development of Machine learning based FDIR algorithms with both

unit/system test and qualification data and in flight data; Verification of the algorithms performances on suitable on-board processors

that will enable running deep learning applications on the edge;

Page 14/28



Dry run on 'ground data' (coming from unit/system level tests) and experimentation on in-flight data on past missions;

Preparation for suitable In Orbit Demonstration for upcoming missions.

Deliverables: Software, Report


18


All missions, which require bespoke FDIR solutions with complicated architectures and difficult testability.


Page 15/28




Ref. Number: GT1I-305ED Budget (k€): 1,000

Title: AI for non mission critical on board data processing

Objectives: The objective of this activity is investigating the possibility of using AI-based algorithms to improve the autonomy and the efficiency of the on-board management of payload data.

Description: Spacecraft’s produce an ever-increasing amount of data for increased scientific output and better monitoring and operations capabilities. Due to limited on-board computational power, most of the processing and analysis is performed on-ground, requiring all raw data to be downlinked first. This requires significant bandwidth, which in many cases could have been saved, for example, by pre-selecting a sub-set of raw data to downlink or by extracting higher-level information on-board and sending a smaller but equally relevant set of data. When it comes to housekeeping, some missions have tens of thousands of parameters and this process becomes challenging. That is why, traditionally, data processing has only been performed on-ground. Over the last decade, data analytics techniques have been developed to support operators: they automatically detect new behaviour, find recurring anomalous situations or identify dependencies among parameters. This is time-consuming and prevents immediate reaction to identified events, whether on the payload or on the platform itself. However, the latest generations of on-board computers open the door to extended on-board data processing. On-board data processing, e.g. using machine learning (ML), could allow defining autonomous actions triggered by the results of the processing. For example, if data is processed on-board, interesting phenomena can be detected much quicker and the spacecraft can react dynamically, for example by retargeting itself for further observations. There are many possible applications of ML and parallel computing on-board satellites, including image processing, complex on-board scheduling, autonomous decisions, etc. This will require the selection of algorithms particularly adapted to the severe resource constraints of on-board software. We propose to research both software and hardware aspects of this concept and build a solution compatible with these constraints that could be introduced for in-flight demonstration. The proposed activity aims at investigating the possibility of using AI-based algorithms to improve the autonomy and the efficiency of the on-board management of payload data. The idea is to use raw payload data and ground-post processed data from current and past missions to realize a framework to allow the use of such data to train and test different types of neural networks with different objectives, among which:

Generative networks for the identification of unnecessary patterns for final knowledge extractions, to allow compression of transmitted data;

Convolutional networks for segmentation/classification (e.g. clouds, shadows) segmentation adaptive to contents through to allow dynamic reduction of transmitted data;

Recurrent deep networks for the prediction and/or detection of specific events, to allow autonomous response (e.g. instrument recalibration, operative mode selection, etc.);

Page 16/28



Sensor performance analysis for anomaly detection and timely management, and to verify the results against those obtained with current methodologies.

Besides the more commonly used network topologies and associated training algorithms, consideration will be given also to the use of evolutionary programming for both structural and parametric definition of the networks. The development of this space-based AI application paves the way for more advanced applications, such as (e.g.) exploiting space-based video in real time for vessel tracking or sensor data fusion. The principal tasks that will be performed during the activity can be summarized as follows:

Analysis of the current state of the art regarding on-board payload data management;

Preliminary requirements definition for the framework being developed; Identification of representative cases to be considered; Identification of available data sources and computational environments; Implementation and use of a demonstrator of the framework; Tests execution and preliminary results assessment;

Deliverables: Software Suite Demonstrated on a relevant environment


24


Earth observation and science missions.


Page 17/28





Title: Robust machine learning systems for dependable space applications

Objectives: Objective of this activity is to develop methods for detection and rejection of adversarial examples in security sensitive and safety-critical systems using deep Convolutional Neural Networks (CNNs). This is an essential activity to enable use of Artificial Intelligence (AI) and Machine Learning (ML) on the edge on mission critical applications. The methods developed in this activity, although targeting mostly Machine Learning inference in FPGA, are also applicable to more generic ML implementations.

Description: Recent developments in AI and Machine learning research have proven that many different models of inference system have potential use in next generation space missions for on-board data analysis; autonomous decision making; sensor diagnostic and prognostic; etc. platforms, although these methods are highly flexible computational tools supported by energy efficient hardware accelerators, it is still an unsolved problem to explain the relationships between their inputs and their final outputs, especially on AI systems based on deep learning techniques. The inability to backtrack the inference process in a meaningful way hinders the adoption of AI models for safety-critical systems - such as satellites - which instead require the inference process to be manifest and non-ambiguous, possibly supported by well defined mathematical models. Machine learning systems are not robust by default. Even systems that outperform humans in a particular domain can fail at solving simple problems if subtle differences are introduced. For example, consider the problem of image perturbations: a neural network that can classify images better than a human can be easily fooled into believing that sloth is a race car if a small amount of carefully calculated noise is added to the input image. For dependable systems we need a solution that evaluates the AI/ML network as a black box algorithm already trained to perform a specific task, and evaluates the 'corner cases' in (for example) image analysis. The idea is to develop a modular framework that can be adopted for multiple analyses. We need to supplement next AI inference systems with a set of operational conditions and ranges of tolerance for user-defined input distortions that may serve as a first step in defining the robustness of the system to errors and faults, possibly bridging the gap between safety-critical systems and AI systems towards the adoption of AI engines in future space applications. Furthermore, the SRAM FPGAs or COTS processors targeted in many cases as the preferred HW inference engines are themselves vulnerable to radiation-induced Single Event Effects (SEEs). Thus, low-power and radiation hardening design techniques should be developed to enable their integration in on board data modules. This project aims to propose design techniques that will improve the performance-energy-reliability trade off of SRAM FPGA ANNs and processor implementations. The core this project is on the set-up of a full end-to-end tool-chain and neural-network inference framework that can be trained on-ground (e.g. cloud based) and then be deployed and executed on a typical flight system, with particular care on fault tolerant techniques and verification and validation problems, impacting use of AI/ML in all relevant space missions.

Page 18/28



Tasks: Definition of ML use cases in mission critical applications (example rover

guidance, star tracking, FDIR management, autonomous collision avoidance manoeuvres);

Definition of target hardware (Accelerators, FPGA, processors) and of suitable FT techniques to be adopted;

Development of an Engineering Model, using representative space graded HW.

Verification and validation of the developed EM.

Deliverables: Engineering Model


18


All missions.


Page 19/28



Domain Generic Technologies - CD3 - Avionic Architecture / DHS / Onboard SW / (FDIR) / GNC + AOCS / TT&C (E2E)

Ref. Number: GT1I-307SA Budget (k€): 400

Title: Training datasets generation for machine learning: application to vision-based navigation

Objectives: The main objective is to design a process for generation of training datasets in support to the deployment of machine learning in vision-based navigation applications.

Description: One of the limiting factors for adoption of machine learning techniques in the areas of vision-based navigation and control (e.g. multi-body control applications) is the lack of datasets. Availability of space images is limited, or even scarce (e.g. in the case of solar system exploration). This holds for both celestial objects and, particularly, for man-made objects. This activity entails a systematic analysis of data availability and data generation for training machine learning-based algorithms in vision-based navigation applications like pose estimation or feature tracking for rendezvous or planetary landing. Different processes for image generation shall be investigated and compared:

The generation of training data by computer graphics and simulations. Realistic synthetic images shall be rendered to provide a database for training.

Use of mock-ups to generate realistic image data. Use of deep-learning methods to generate training data from generative

neural networks (e.g. generative adversarial networks). Main tasks of the activity are the following:

Define and agree with the Agency a number of mission scenarios for application of machine learning in vision-based navigation (e.g. small-body proximity operations, in-orbit servicing, etc.) and for which real images are available (e.g. Rosetta NavCam images, LIRIS experiment, etc.);

Generate a range of synthetic images (covering several celestial bodies and man-made objects at several levels of resolution and texture fidelity) for the scenarios defined in the previous task;

Create mock-ups of the same celestial bodies and man-made objects as in the previous task, and, using a camera representative of a navigation camera, generate a database of images under different poses and illumination conditions;

Design and implement a generative neural network capable of taking in input the synthetic image of a target spacecraft with a certain pose configuration and output a high fidelity image of the target spacecraft with the same pose;

Feed the synthetic images and databases of mock-up images to the generative neural network to train the image generator at increased levels of fidelity;

Use the real imagery as a benchmark to assess the quality of the images generated by the generative neural network by running the same image-processing algorithm on the real and generated images.

Deliverables: Report, Software


15

Page 20/28




Missions requiring visual navigation (e.g. rendezvous missions, asteroid missions, planetary landing)


Page 21/28





Title: Development of distributed autonomous trajectory

Objectives: Objective of the activity is to develop a machine learning-based guidance algorithm to enable autonomous formation reconfigurations in response to high-level goals at large number of cooperating spacecraft.

Description: Current techniques for station-keeping and trajectory control in general are heavily relying on on-ground model-based optimization techniques whose computational complexity scales badly with the number of spacecraft. As a consequence, the rise of constellations calls for a paradigm shift from traditional ground-based trajectory control to more flexible architectures providing autonomous on-board large number of cooperative S/C management capabilities. In the context of constellations, enhancing the system responsiveness and flexibility requires that high-level goals at constellation level are propagated and allocated at spacecraft level such that each spacecraft can collaboratively optimise its trajectory on-board, while complying with mission constraints and resource availability. Modern machine learning techniques have the potential to accomplish such computationally intensive task, due to their capability to handle complex guidance problems using a relatively small number of parameters. This activity entails the development of AI-based on-board guidance optimisation algorithms which can be implemented in a distributed fashion among multiple satellites: machine learning elements will be integrate in an on-board guidance optimiser so that such optimiser can be trained to ensure that the spacecraft collectively accomplish a formation reconfiguration objective (e.g., maximization of the coverage of a given area) by optimizing resource usage (e.g. fuel consumption) and performance (e.g. time minimisation), while ensuring safe trajectories (e.g. collision avoidance) and tolerance to failures (e.g. loss of satellites). The main tasks of this activity include:

A literature review of machine-learning techniques for distributed agents and on-board guidance optimisation techniques for spacecraft;

The architectural design of a machine-learning-based guidance algorithm, based on a functional analysis of the distributed guidance problem and a functional allocation of tasks at constellation and spacecraft level;

The detailed design and implementation of the machine learning-based guidance algorithm;

The profiling of such algorithm in order to assess implementability on space-qualified processors;

The development of benchmark scenarios and corresponding implementation in a simulation environment where autonomous large number of cooperative S/C management can be tested;

The verification and validation of the algorithm implementation in the simulation environment.

Deliverables: Report, Software


12


Missions involving large constellations or large number of cooperating spacecraft


Page 22/28





Title: 3D shape reconstruction assisted by machine learning techniques

Objectives: The objective of this activity is to design a machine learning techniques for 3D shape reconstruction, with specific application to vision-based navigation with respect to non-cooperative unknown targets.

Description: Rendezvous with non-cooperative targets with uncertain or unknown properties (e.g. space debris) poses significant navigation challenges, in that common model-matching techniques cannot be relied upon since a sufficiently accurate 3D model of the target is not available. Such problem can be circumvented by generating the 3D model on the basis of images taken by cameras on-board a chaser spacecraft under different views. Multiview 3D reconstruction is capable to infer the geometrical structure of a scene captured by a collection of images: by using multiple images, 3D information can be recovered by solving a pixel-wise correspondence problem Previous studies have used standard computer vision techniques to solve this pixel-wise correspondence problem, however the potential of machine learning for 3D shape reconstruction has not yet fully explored: indeed, machine learning techniques such as convolutional neural networks are very suitable to infer correlations among images that are essential in solving the pixel-wise correspondence problem. This activity entails the following tasks:

Review the existing literature on 3D model reconstruction, for both space and non-space applications and select the best candidate techniques for

Design the 3D shape reconstruction algorithms for these techniques and implement the corresponding SW code;

Generate a set of synthetic images of man-made space objects to train the machine learning-based 3D shape reconstruction algorithm until 3D models are generated which are sufficiently accurate to be used in a model-matching algorithm;

Use an existing set of space images of man-made objects (e.g. from LIRIS experiment or PRISMA experiment) as benchmark to test the performance of the 3D shape reconstruction algorithm;

Provide a way-forward for future developments.

Deliverables: Software, Report


9


Asteroid and debris removal missions


Page 23/28



CD9 - Digital Engineering for Space Missions

Domain Artificial Intelligence CD9 - Digital Engineering for Space Missions

Ref. Number: GT1I-310OS Budget (k€): 350

Title: Machine learning prediction for safer spacecraft operations and increased science return

Objectives: The objective of the activity is to develop a procedure that will make use of Machine Learning techniques in order to reduce operational risks and increase science return.

Description: One of the major factors affecting the risk of a mission and the ability to perform scientific experiments is uncertainty. The classic way to cope with uncertainty is to add operational margins on expected outcomes. Operational margins often have a negative impact on the science return and, when the expected outcomes are also uncertain, they may not even be enough to keep the spacecraft safe. Machine Learning (ML) has proven useful to reduce uncertainty, provide accurate predictions and even detect anomalies in the way to happen. To capitalize on the capabilities of Artificial Intelligence (AI) and Machine Learning (ML), this activity will explore additional potential areas of application of Machine Learning in Space Operations. In particular, this activity will focus on reducing operations risk, increasing science return or, ideally, both simultaneously. The proposed activity is generic by nature and its results shall be applicable to domains such as space exploration, earth observation and astronomy missions. The identification and selection of the use cases is part of the activity and it is recommended to cover more than one domain, with a common goal of exploiting AI for reducing uncertainties, optimize operational decision making and have a mixture of increased science return and operational risk reduction. For each selected use case an AI ML application prototype will be implemented, using the most appropriate algorithm, refined iteratively. The ML prototype will be then validated and evaluated. The following tasks will be part of this activity:

Use Cases definition and selection: considering needs / opportunities and available data

Define evaluation criteria to measure the performance of the ML approach (and be able to compare it with non-ML approaches, if any) – per identified use case

Data Preparation (e.g. selection, cleaning, handling missing values, resampling, etc.)

Feature Engineering. Develop a prototype with the identified ML techniques for the selected use

cases. Perform verification and validation activities using numerical simulations. Provide a way-forward for future developments.

Deliverables: Prototype, Report


18


All missions.


Page 24/28





Title: AI learning services for space systems operations

Objectives: The objective of this activity is to develop an AI design algorithm design process in order to increase robustness of the AI and to develop a production process to allow industries to provide quality AI products.

Description: Current AI software design and deployment processes in ESA are highly dependent of the information available where models were designed, like which experience the end-users participating to the process exposed to the experts that designed the models. In theory, AI quality software design process should be instead independent from the data or the specific application being designed. This dependency makes still the process of generating quality AI applications a highly specialized, not easily reproducible activity, putting a significant barrier against the adoption of AI in the private sector. In fact, while AI is demonstrated effective in the space sector for several operation application (satellite health monitoring, GNC, mission planning and so on), industrialization of each of those capabilities is still a challenge. Emerging methodologies, such as DevOps mindset, agile with scrum, iterative data science project management are transforming the process of design, development and deployment of AI-powered applications. We propose to explore a secured way to break classical TRL maturity ramp-up and allow delivery of TRL4-6 directly within operations to raise maturity directly aside operators. We consider such a solution will:

Enable AI delivery to operations; Reduce the recurring cost of industrializing separately each AI capability.

Allowing maturity increment within operations and control AI autonomy increment within operations, it will enable AI effective use in operations.

The following tasks will be part of this activity:

Identification of criteria/KPI for benchmarking the quality of AI-based application development methods for space systems operation

Analysis, classification and trade-off of emerging software development and project management methods of AI-powered applications, based on identified criteria/KPI

Identification of two AI-application use-cases Selection of two suites of software methodology Development of AI-applications prototypes related for the use-cases and

based on the selected suites of methodologies Deployment of prototypes in operations and fine-tuning of the AI

applications following the two different suites of methodologies. Assessment of the results and evaluation of the KPI of the two suites of

methodologies. Provide a way-forward for future developments.

Deliverables: Prototypes, Report


18


Mission operations.


Page 25/28





Title: Online deep learning for anomaly detection & isolation: from V&V to final operations

Objectives: The objective of this activity is to prototype and validate an online AI-based service of anomaly detection in support of spacecraft final validation, integration, testing and operations phases

Description: During final validation, integration, testing and ultimately nominal spacecraft operations vast amounts of data are generated and made available to engineers and operators. Effective mining of this data is essential to identify anomalies and anticipate or diagnose failures efficiently. In the context of nominal operations, several machine learning techniques for anomaly detection have been employed for decades now (e.g. Novelty Detection, PCA or SVMs or Neural Nets). However, little effort has been done towards adapting those techniques for validation, integration and testing phases where detecting anomalies, inconsistencies or equipment failure can also significantly impact budget and schedule. This proposal is focused on an on-line deep learning solution servicing the spacecraft from validation up to final operations. A suitable technique or suite of techniques need to be identified and validated for processing the stream of incoming data, during the mentioned phases, starting at validation, and progressively refine the techniques and their capabilities to detect and isolate anomalies and failures as early as possible. This technology has the potential of reducing costs and schedule during final validation, integration and testing increasing the competitiveness of the European spacecraft industry, and during in-orbit operations, reducing cost/improving performance of operating a mission. Furthermore, it is deemed highly beneficial for large-scale constellations where affordable operations are a must. The activity shall also look at possibility to use the validated techniques also on-board of constellation spacecraft, aiming to future solutions for more autonomous missions. The following tasks will be part of this activity:

Use Cases definition and selection: considering needs / opportunities and available data

Define evaluation criteria to measure the performance of the AI based approach (e.g. false alarm rate, anomaly detection rate, etc.)

Data Preparation (e.g. selection, cleaning, handling missing values, resampling, etc.)

Feature Engineering. Develop a prototype with the identified AI/ML techniques for the selected

use cases. Perform verification and validation activities using numerical simulations. Provide a way-forward for future development.



18


All missions.


Page 26/28





Title: Autonomous AI-based satellite command & control for large number of cooperating spacecraft

Objectives: The objective of this activity is to develop an autonomous AI based system for commanding and control of spacecraft constellation.

Description: Reducing the costs of operations of a very large constellations (hundreds or thousands of satellites) is really challenging. Automation is a necessity and can be partially done using either simple automation means or using more advanced data analytics/machine learning techniques for improving situation awareness and triggering predefined recovery procedures. However, the limit is brought by the fact that defining in advance all the possible anomalies and how to react to them is not really easy and thus, actual autonomy of the command and control is limited. AI could help to reach a drastically higher level of satellite C&C autonomy (still using ground control at this stage) using inspiration from what is done for driving assistance. As a matter of fact, we face the same issues: the difficulty to collect a large number of cases - and especially non-nominal situations - to use supervised learning. It is proposed to investigate the use of a combination of satellite simulators and Generative Adversarial Networks (GAN) to generate situation “images” (i.e. satellite telemetries sets) and Deep reinforcement Learning to implement satellite control (TC or FOP generation). This could be developed on two use cases, typically one ESOC controlled satellite on one side and at a commercial satellite premises on the other side. The activity will also leverage on the experience already gained in other domains, such as autonomous car driving. The results will be a foundation for further developing and implementing autonomy-driven operational AI-solutions for complex missions, such as constellations, where world-wide competition is crucial for European space market The proposed tasks are:

Analyse what is done in driving assistance field or similar domains; Analyse and implement satellites simulators adaptations; Build TM “images” generator to generate a large number of situations with

anomalies for example using GANs; Learn how to react to these generated situations (TC/FOP automated

generation) via deep learning (typically using deep reinforcement learning); Analyse how to validate the generated actions e.g. through simulation or ad-

hoc means to be defined during the project.



20


Large number of cooperating spacecraft missions


Page 27/28





Title: Artificial Intelligence for large fleet network management

Objectives: The objective of this activity is to develop a prototype of advanced management of complex network of large number of cooperative satellites, using AI techniques.

Description: Everyday, both observation and telecommunication end-to-end systems are getting more and more complex with:

more and different assets (satellites, HAPS, drones,…); more interconnected layers (satellites with Inter Satellite Links (ISL),

hybrid system (HydRON), connected HAPS or drones, etc.); more exigent clients demanding more flexibility, more speed, faster data

delivery, etc. Operate this kind of complex systems and achieve the expected performance is very challenging due to a high combinatorial level. Therefore, new advanced and specific resource and network management means need to be developed for each project. Network planning can be solved as load balancing and routing problem: sharing traffic from one end to the other through multiple interconnected network layers (satellites, HAPS, drones and/or ground networks). The network manager will also have the capacity to reconfigure in near real-time and adapt the system in case of failure (for instance, a satellite failure) by finding alternative paths that ensure a similar performance. This study will be focus on the benefits of AI algorithms (e.g. constraint satisfaction programming, multi-agent deep reinforcement learning, ant colony optimisation, etc.) for the network planning problem of the resource manager. Based on a given satellite capacity (from telecom users or video flux), it will consider the satellite network (satellites connected through ISL) and ground network together to find the best routing path considering the system constraints (such as maximum ISL capacity, maximum data rate from satellite to ground, performance and cost of ground networks, etc.). The global performance will depend on the resource manager algorithm. The expected benefits include generic resource and network management tools with reduction of the product development time; High performance and cost reduction due to increased network capacity gives added value for the customers with robust, near real-time, evolutionary and optimized system planning. The proposed tasks include:

analysis of communication networks of large number of cooperative satellites and selection of use case

requirements definition and operational scenario definition of the AI-based network management and planning prototype

technical analysis and trade-off of AI-based techniques for network management and planning

design and implementation of the prototype validation and performance assessment



18

Page 28/28




Large number of cooperating spacecraft missions


Documents

GSTP Element 1 “Develop” Compendium 2019: Artificial ...emits.sso.esa.int › emits-doc › ESTEC › News › ... · Artificial Intelligence activities of strategic importance