Testing Overview · 2019-10-31 · Introduction Automated Testing Summit Lyon, 2019 3 Civil Infrastructure Platform Establishing an open source base layer of industrial grade software

Testing Overview

Chris Paterson, Renesas Michael Adler, Siemens

CIP Testing Working GroupAutomated Testing Summit Lyon, 31/10/19

Introduction

• Michael Adler• Linux Consultant, Siemens AG• Working on Linux container solutions and firmware updates for

embedded devices• Interested in functional programming and modern programming

languages• Passionate NixOS user

• Chris Paterson• Project lead in the Linux team at Renesas Electronics Europe• Heading the “testing” working group for the CIP project• Newly found love for CI

Automated Testing Summit Lyon, 2019 2

Introduction


Civil Infrastructure Platform

Establishing an open source base layer of industrial grade software to enable the use and implementation of software building blocks for civil infrastructure

• Super Long-Term Support(ed) Linux Kernels – 10+ years

• Real Time Linux

• CIP Core – Reference filesystem

• Security – IEC 62443 certification

The backbone of CIP are the member companies

Developers, maintainers

€¥

$£Budget

Open Source Projects

Contribution & usage / integration Optional: funding of

selected projects

Automated Testing Summit Lyon, 2019

Testing Goals

• Provide an environment to test CIP software projects on the CIP reference hardware

• SLTS Kernel• SLTS RT Kernel• CIP Core

• Tiny profile (Deby)• Generic profile (ISAR)

• SW update• […]

• Open source

• Collaboration


CIP Reference Hardware

• The CIP project is currently supporting the following reference hardware:

• Currently testing ~30 Kernel configurations (including some RT configs) across the two SLTS versions.


Device SLTS v4.4 SLTS v4.19

AM335x Beaglebone Black (Armv7) ✔ ✔

QEMU x86_64 ✔ ✔

RZ/G1M iWave Qseven Dev Kit (Armv7) ✔ ✔

RZ/G2M HopeRun HiHope (Armv8) ✔

SIMATIC IPC227E (x86-64) ✔ ✔

OpenBlocks IoT VX2 (x86-64) ✔

Architecture Overview


LAVA Worker

LAVA MasterArtifact Storage

GitLab runner @ k8s master

LAVA Worker

Built Artifacts

CIP Reference Hardware

Source CIP Kernel CIP Core

Build

k8s pod (build)

Test

LTS-rc

k8s pod (build) k8s pod (test)k8s pod (test) GitLab.com

AWS EC2

AWS EC2

on-demand

Local

Location key

SWUpdate

Tests (early days!)

• Currently CIP are running the following tests:• Boot test

• uname -a

• Spectre/Meltdown checker• https://github.com/Linaro/test-definitions/tree/master/automated/linux/spectre-meltdown-

checker-test

• In progress:• LTP

• ltp-cve-tests, ltp-dio-tests, ltp-fs-tests, ltp-ipc-tests, ltp-math-tests, ltp-open-posix-tests, ltp-sched-tests, ltp-syscalls-tests and ltp-timers-tests

• https://github.com/Linaro/test-definitions/tree/master/automated/linux/ltp

• https://github.com/Linaro/test-definitions/tree/master/automated/linux/ltp-open-posix


https://github.com/Linaro/test-definitions/tree/master/automated/linux/spectre-meltdown-checker-test

https://github.com/Linaro/test-definitions/tree/master/automated/linux/ltp

https://github.com/Linaro/test-definitions/tree/master/automated/linux/ltp-open-posix

Next Steps

• Improve job reporting• Results in GitLab/LAVA/email are easily lost• We plan to submit all test results to the KernelCI project

• Increase test coverage• CIP Core (reference filesystem)• kselftests, Jitterdebugger, Linaro test definitions, Benchmarks, Hardware testing (CAN/PCIe/USB etc.)

• Add more boards/LAVA workers• Speed up testing• Improve board availability

• Collaboration with the Automated Testing Community


Tooling

gitlab-cloud-ciContainer-based CI-infrastructure from Scratch

siemens.comUnrestricted © Siemens AG 2019

Unrestricted © Siemens AG 2019

31.10.2019Page 12 Corporate Technology

gitlab-cloud-ci

Motivation

• ISAR: Integration System for Automated Root filesystem generation

• Similar to Yocto, but use upstream deb-packages as much as possible

• => No need to maintain your own Linux distribution anymore

• Perform cross-architecture builds

• Very versatile and flexible, but also easy to make mistakes => CI solution desired

• Used by Civil Infrastructure Project, Siemens and others

• Problem: Standard (Gitlab) CI runners do not work with ISAR-based builds in general

• Reasons:

• binfmt_misc is not namespace-aware (yet!), but is required for cross-architecture builds

• debootstrap: some (minor) issues with CAP_MKNOD

https://github.com/ilbers/isar



gitlab-cloud-ci

Possible Solutions

• “Hard” Way

• Make binfmt_misc user namespace aware and merge it upstream

• Fix all issues with debootstrap (mostly working around MKNOD issue)

• Not that hard, but would take some years until it is merged upstream and available in distros

• Easy Way

• Just use privileged containers (CAP_SYS_ADM)

• … or use a good old-fashioned VM after all

Which way to go?

Both ways! But start with the easy one and wait for the hard one ☺



gitlab-cloud-ci

Requirements Engineering

• Following the “Easy“ Way

• Let’s make a wishlist! It should be:

• Fast: CI feedback time must be short (fast CPU, fast SSD, fast network)

• Scalable: Perform many CI jobs in parallel

• Secure: Can we always trust our payload? Isolation would be nice!

• Cheap: opex, capex ☺

• Possibly Reliable: SA or HA, reproducible setup (automated)

• Must be compatible/usable with Gitlab

• Avoid vendor lock-in

• Should work on-premise and in the cloud



gitlab-cloud-ci

Enter gitlab-cloud-ci

• gitlab.com/cip-project/cip-testing/gitlab-cloud-ci

• Developed internally at Siemens and published under Apache-2.0 license

• Written in Python 3

• Yet another Kubernetes bootstrapping tool? Like Kops or Kubespray?

• Or is it more a “general tool” like Terraform?

• Neither! gitlab-cloud-ci is a thin wrapper (“glue code”) around existing battle-proven tools

• Features

• Create and bootstrap Kubernetes cluster (SA/HA) from scratch

• Choose between AWS and on-premise setup

• Heavy-lifting is done by kops/kubeadm

• Deploy dashboard, cluster-autoscaler, Gitlab runners, binfmt_misc “hack” and more

• Everything that can be created can be destroyed as well

• Basic lifecycle management functionality

https://gitlab.com/cip-project/cip-testing/gitlab-cloud-ci



gitlab-cloud-ci

Real-World Usage

• CIP: Has been permanently in-use for almost 6 months:

• 100% uptime for master node (thanks AWS)

• Occasionally minor hick-ups with cluster-autoscaler (or AWS)

• AWS + Kubernetes autoscaling is rather slow (a few minutes) and sometimes results in timeouts

• Might improve once we upgrade to latest Kubernetes + cluster-autoscaler add-on

• Started small (m5d.xlarge 4 vCPUs, 16GB mem, 1xNVMe SSD),

but recently went up to m5d.4xlarge (16vCPUs, 64GB mem, 2xNVMe SSD)

• Dynamically scaling between 0 and 40 slave nodes

• Siemens: Used for internal Gitlab instance (on-premise setup)



gitlab-cloud-ci

Outlook & Contributing

• Contributions are welcome!

• … bug reports too ☺

• Planned Features:

• Add cluster monitoring (Issue #4)

• Mixed EC2 instance sizes (configurable on a per-job basis) (Issue #6)

• More configuration options (Issue #3)

• Profiles

• Kata Containers

• Ideas:

• GCE integration (cheaper than AWS because master node is free)

linux-cip-ci

• Simple ‘tool’ that manages the build and test process for the CIP Linux SLTS Kernels.

• Consists of two Docker containers:• ‘Build’

• Contains: Linux Kernel build scripts & dependencies, CIP Kernel configs repo• Appropriate cross-toolchain downloaded on the fly• Builds the Kernel for the specified configuration• Stores build artifacts in GitLab’s artifact storage

• ‘Test’• Contains: LAVA test job generation scripts, LAVA and AWS CLI tools• Uploads build artifacts to AWS S3 storage (so LAVA workers can access them)• Creates LAVA job definitions for the specified platforms and tests• Submits LAVA jobs and waits for the results

• https://gitlab.com/cip-project/cip-testing/linux-cip-ci


https://gitlab.com/cip-project/cip-testing/linux-cip-ci

linux-cip-ci: Example .gitlab-ci.yml


Questions?

Thank you!


Contact#cip (freenode) [email protected] [email protected] [email protected]

mailto:[email protected]



Links

• CIP testing wiki page• https://wiki.linuxfoundation.org/civilinfrastructureplatform/ciptesting/centalisedtesting

• CIP reference hardware• https://wiki.linuxfoundation.org/civilinfrastructureplatform/ciptesting/cipreferencehardware

• CIP Kernel Configurations• https://gitlab.com/cip-project/cip-kernel/cip-kernel-config

• CIP LAVA master• https://lava.ciplatform.org/

• CIP lava-docker• https://gitlab.com/cip-project/cip-testing/lava-docker

• CIP Kubernetes k8s pod manager• https://gitlab.com/cip-project/cip-testing/gitlab-cloud-ci

• CIP Kernel CI build tool• https://gitlab.com/cip-project/cip-testing/linux-cip-ci


https://wiki.linuxfoundation.org/civilinfrastructureplatform/ciptesting/centalisedtesting

https://wiki.linuxfoundation.org/civilinfrastructureplatform/ciptesting/cipreferencehardware

https://gitlab.com/cip-project/cip-kernel/cip-kernel-config

https://lava.ciplatform.org/

https://gitlab.com/cip-project/cip-testing/lava-docker



Documents

Testing Overview · 2019-10-31 · Introduction Automated Testing Summit Lyon, 2019 3 Civil Infrastructure Platform Establishing an open source base layer of industrial grade software