From AGL Supervisor to platform global data collection

Sébastien DouheretTechnical Director at IoT.bzh<sebastien@iot.bzh>

Automotive Linux SubmitJuly 17 - 19, 2019

Tokyo, Japan

July 2019From AGL Supervisor to Platform global data collection

Who are we ?

● 25 Engineers dedicated to AGL● Location: Lorient/Brittany/France/Europe● Team : strong visibility in Open Source Community● People background : Telecom, Automotive or Marine

domains, Embedded developments (Intel, WindRiver, ...)

https://iot.bzh/en/ http://github.com/iotbzh

LORIENT

IoT.bzh New Office in Lorient

Large space (700m2) including Hacking Car Lab

Working on ...● Main contributions on AGL :

– Cybersecurity (Secure Application Framework)

– Development tools for AGL

(SDK, X(cross) Development System)

– Integration and board support (Renesas)

– Community Support & Documentation

– 1st AGL technical contributor since 2016

● Connected Boat project

– Consortium with 4 other partners

– Targeting work boats

– Goals : increase Security, predictive maintenance and automation of administrative documents filling

● Conferences / White papers (https://iot.bzh/en/publications)

References: Walt Miner presentationsAGL AMM 2017 https://sched.co/9KB6AGL AMM 2018 https://sched.co/DRAXAGL AMM 2019 https://sched.co/L8W3

Agenda

XDS : Apps and Bindings development (Recap XDS - Code / X-Build / Run / Debug)

Monitor a single binding

XDS Supervision (Needs, Architecture and a Demo)

Challenges & Roadmap

Agenda

Develop

Deploy Debug

MonitorXDS

XDS – Quick OverviewX(cross) Development System

● Dedicated to Applications Developers

– enables development without Yocto skills

● Cross-platform build using AGL SDK

– leverages AGL/Yocto toolchain

● Deploy on development boards orQemu

● Remote debugging from IDE

● Documentation on AGL docs website:https://docs.automotivelinux.org/docs/en/master/devguides/#xcross-development-system-user's-guide

XDS – Reducing complexity

● Easy target access ● console, SSH, …

● Developer uses common IDEs● VS Code, Netbeans, Eclipse...

● Dashboard Web App to manage configurations and trigger actions

● XDS API for CI workflows & specific environments● Command line tool also available (xds-cli)

XDS – Key Features● Multi-platform : no dependencies on developer host

(Linux / Windows / MacOS)

● Easy to setupNear-zero install, no admin privileges required

● Application sources remain localCompatibility with existing IT policies (e.g. corporate backup, git, ...)

● Cross toolchain & tools embedded in a container :

● Local : run locally (local subsystem, virtual machine, docker container ...)

● On-premises : run on a local build server● Cloud : SaaS

● Leverage specific OS capabilities where applicable(e.g. Linux for Windows Subsystem, docker)

1) Install xds-server on a build server machine

2) Install xds-agent and start-it on your machine

3) Browse XDS dashboard: http://localhost:8800

4) Add a new project

5) Cross build/compile your project using either :

1) Dashboard build page

2) IDE using xds-cli

6) Deploy app

7) Source debug from IDE (xds-gdb)

XDS – Operations

XDS – Web Dashboard and CLI

Web dashboardavailable on

http://localhost:8800

Command Line Interface

XDS – Dashboard (cont.)

Agenda

Monitor : a simple built in tool

● Monitoring API is built in the binder

● Monitor API is enabled by AppFW in any binder(--monitoring option)

● Monitor only ONE binding● UI is available easily:

http://<host>:<port>/monitoring/

NOTE THE TRAILING SLASH !

Build-in Monitor – UI Overview

Monitor – outcome● Trace verbs calls

– API calls with args

– Results with data● Trace events content● View internal log calls (AFB_NOTICE et al.)● Filtering available to reduce “noise”:

– Filter per API

– Filter per trace type

Agenda

AGL Micro-Services architecture

Bindings topology agl-demo-platform HH

How to detect abnormal binding exchanges ?

Topology alre

complex

for a si

mple demo !

Why Supervision is needed ?

Debugging (bug analysing) and Tunning

Telematic : Black / Save-box logger

2 Main Needs

Systems based on AGL are becoming more & more complex

Simple or traditionnal debug methods no longer adapted

Constructors will need to push some metrics to cloud

1 SolutionXDS Supervision

Supervision & Vehicle to Cloud● Data collection for debugging

● Collect system data (cpu, memory usage, …) and AGL binding exchanges (supervision / monitoring)

● Only in development mode

● Tuning● Car testing campaign (offline storage)

● Telematics● Based on acquisition CAN, vehicle signals, GPS, ... ● Externalized vehicle data analysis (failure detection)● Black box, Save box● Push useful data to the could (aggregation, retention policies)● Development and Production mode

Store caching: data saved on-board or in a cloud Time-Serie database

Supervision – ArchitectureAPPLICATION

BINDERafb-daemon

BINDING

http ws

AGL App 1

BINDERafb-daemon

BINDING

AGL Service 2

APPLICATION

BINDERafb-daemon

BINDING

http ws

AGL App 2

afs-supervisor

BINDERafb-daemon

BINDING

AGL Service 1BINDERafb-daemon

BINDING

AGL Service 3

BINDERafb-daemon

XDS Supervision

BINDERafb-daemon

HARVESTER

Harvester Service

TimeSeries DB

BINDERafb-daemon

collectd

Low Collector

TARGET HOST

Supervision – Target side● afs-supervisor daemon and xds-supervision

service

xds-supervision control afs-supervisor daemon to gather AGL data (events, resquests, ...)

● Low-collector service

collect system wide metrics (CPU, memory load, temperature, …)

● Harvester service

Received and format metrics to storethem in a local or remote Time Serie Database (TSDB)

Supervision – Host side

● Time Serie Database POC tested with influxDB

● Graph composer and query processing tools

Grafana draws reports based on influxdb records

● XDS Supervision dashboard : control panel (config / start / stop)and virtualization (Grafana dashboards integration)

Low Collector - collectdDeveloped in parternship with a University

● Developed by Marc-Antoine Riou (intership)

collectd integration as an AGL service● reuse collectd source code plugins (≈ 130 plugins available)● run plungins in the Low Collector binding context

Architecture :

Low-Collector - collectd : rendering

Supervision – Visualize in XDS

What about logs? Loki + promtail

● What is promtail ?● responsible for gathering logs and sending them to Loki

● What is Loki ?● Scalable & highly-available log aggregator system

(≈ Prometheus)● main server, responsible for storing logs and processing

queries

● Compatible with Grafana for rendering logs

Demo scenario● Example: 2 apps simulate requests to 2 low

level bindings and receive responses and events

Sources : https://github.com/iotbzh/agl-services-mockup

Demo...

https://iot.bzh/en/publications/xds-supervision-als-2019.htmlhttps://vimeo.com/iotbzh

Agenda

Technical challenges

● Loki, promtail powerfull and promising tools but quite recent

● Promtail written in go (go support almost done in IoT.bzh recipe)

● Time Serie Database not initially designed for embedded world (only a challenge for offline mode)

● Local processing in Car / on board (NAND writes)● Collecting too many data = intrusiveness● Tag data (ownership, priority, lifetime)● Security privileges (API V3 and multi-users)

Roadmap● Done : XDS Supervision POC (running on HH)

● Beta version (not production ready) need more time and resources to fully integrate supervision – Increase integration/validation of collectd plugins (today only 6

plugins)– reduce again intrusiveness– Improve smooth integration with AGL security– Log analyser specifically for Journalctl (systemd logs)

more investigations about Loki & promtail integration

● Other improvements: – on-board TSDB with edge load balancing algorithm– offline data collecting for Key Performance Indicator analysis

3 Things to remember

● AGL System is becoming more & more complexSupervision and log analysis mandatory to debug problem on those systems

● Don’t re-invent the wheel Integration of existing tools and reuse data format to benefit of existing UI viewer or log analyser

● Need more time and ressources Integration, Optimization, make tool robust are time consuming

Visiting us in Lorient

Gulf of Morbihan, south of Brittany, France

References● Proof of concept sources

● https://git.automotivelinux.org/apps/agl-service-harvester● https://git.automotivelinux.org/apps/agl-service-xds-monitoring● https://github.com/iotbzh/agl-service-low-collector*● https://git.automotivelinux.org/src/xds/xds-agent

● TSDB and Logger and Rendering Tools● https://fosdem.org/2019/schedule/event/loki_prometheus_for_logs/

● https://www.influxdata.com/● https://grafana.com/● https://d3js.org/

* Not latest version, need Univesity acceptance

More References...

● IoT.bzh:

– Website: https://iot.bzh/

– Publications: https://iot.bzh/en/publications

– Github: https://github.com/iotbzh● AGL:

– Website: https://www.automotivelinux.org/

– Documentation: http://docs.automotivelinux.org/

– Sources: https://git.automotivelinux.org/

From AGL Supervisor to platform global data collection · Build Develop Deploy Debug Monitor XDS...

Documents

20160713_ALS AGL-JTA CIAT tool for AGL

XDS – a practical view€¦ · 2 Outline I. XDS – the program package II. XDS – how to obtain good data III. Serial crystallography – some aspects From tomorrow – tutorial

debug eap through debug hw-module subslot periodic...debug eap through debug hw-module subslot periodic • debugeapthroughdebughw-modulesubslotperiodic, page 1 debug eap through debug

Lettuce Agl

Web Marketing | Ignite XDS

Personnel agl

Table of ContentsInstalling XDS client tools xds-agent is a client tool that must run on your machine (user / developer host) to be able to use XDS. Installation of other XDS client

XDS (Cross-Enterprise Document Sharing)

Agl Evidence

Integrating the Healthcare Enterprise · XDS : CMPD actors are based on XDS Document Source, Document Consumer, Registry and Repository Actors and use XDS transactions. Required to

wiki AGL/AGL-repo Jan-Simon Möller Add manifest for Funky ... · 6.0.0 Seite 3 fbe6821 AGL/meta-agl Jan-Simon Möller Bump gpu memory on rpi3 a792671 AGL/meta-agl Jan-Simon Möller

NIRS XDS RapidContent Analyzernirs-instruments.com/pdf/Manual for NIRS XDS RapidContent Analyzer… · 6 1 Introduction to XDS™ Rapid Content Analyzer . Thank you for selecting

Guía Técnica de Metadatos XDS

AGL Enercv Limited ABN: 74 115 061375 agl · agl Essential Services Commission Submitted electronically: retailenergyreview@esc.vic.gov.au 5 October 2018 AGL Enercv Limited ABN: 74

EIGHTY20/XDS REPORT

XDS/XDR/XCA Testing

XDS-1078 flyer Retail IQ · Title: XDS-1078_flyer_Retail_IQ Created Date: 10/16/2017 2:24:27 PM

XDS Serisi Dijital Osiloskop - tekolab.com XDS SERİSİ.pdfDMM digital m ultimeter TOU touch s creen (Interpolation capacitor-type) ＋Optional Module / Function XDS Serisi Dijital

AGL GL3000_2700

XDS Zimbabwe Agricultural Conference