Building Open Container Initiative images based on ......Building Open Container Initiative images...

Building Open Container Initiative images based on Freedesktop SDK

Valentin David

Codethink Ltd.

foss-north 2020March 30th

Valentin David (Codethink) Freedesktop SDK and OCI foss-north 2020 1 / 28

Table of Contents

1 Introduction

2 Very quick introduction to BuildStream

3 OCI images

4 Design your image with BuildStream

5 Conclusion

What is Freedesktop SDK?

Runtime of Flatpak applications.Basic runtime and SDK to build containers for desktop applications.

Flatpak is not related to OCI.

On top of it, two Flatpak runtimes are built: KDE SDK and GNOME SDK.

Freedesktop SDK features

Release every year

Bug and security updates for 2 years

ABI stability

Automatic scan for CVEs

Bootstrapped

Architectures

x86-64 and i686

aarch64 and armv7

powerpc64le (experimental)

What does Freedesktop SDK contain?

Basic glibc, bash, coreutils, util-linux, findutils, diffutils, gawk...Archive tar, cpio, zlib, bzip2, xz, zip...Security openssl, gnutls, gnupg, nss...

Graphics X.org (x11 and xcb), Wayland, Cairo, GTK+3, SDL2...Acceleration OpenGL, Vulkan, OpenCL dispatchers, Mesa drivers

Sound Pulseaudio, Alsa w/ pulse pluginMedia gstreamer, mpg123, ffmpeg, vorbis, theora, giflib, libpng...

Programming GCC, LLVM, gperf, flex, bison, ccache...Build make, autotools, meson, ninja, cmake...

Interpreters Perl, Python, RubyDocumentation gtk-doc, asciidoc, docbook, man-db...

Fonts DejaVu, Liberation, GNU Free, EmojiOne....Font rendering Pango, Fontconfig, HarfBuzz

Debugging Strace, GDBSpelling hunspell, aspell, LibreOffice dictionaries

Web curl, libsoupValentin David (Codethink) Freedesktop SDK and OCI foss-north 2020 5 / 28

What does Freedesktop SDK contain?

Basic glibc, bash, coreutils, util-linux, findutils, diffutils, gawk...Archive tar, cpio, zlib, bzip2, xz, zip...Security openssl, gnutls, gnupg, nss...

Graphics X.org (x11 and xcb), Wayland, Cairo, GTK+3, SDL2...Acceleration OpenGL, Vulkan, OpenCL dispatchers, Mesa drivers

Sound Pulseaudio, Alsa w/ pulse pluginMedia gstreamer, mpg123, ffmpeg, vorbis, theora, giflib, libpng...

Programming GCC, LLVM, gperf, flex, bison, ccache...Build make, autotools, meson, ninja, cmake...

Interpreters Perl, Python, RubyDocumentation gtk-doc, asciidoc, docbook, man-db...

Fonts DejaVu, Liberation, GNU Free, EmojiOne....Font rendering Pango, Fontconfig, HarfBuzz

Debugging Strace, GDBSpelling hunspell, aspell, LibreOffice dictionaries

Web curl, libsoup

The choice of technologies used by Freedesktop SDK is notan endorsement by the Freedesktop organisation.

BuildStream

Freedesktop SDK is built with BuildStream.

Build and integrate artifacts.

Separate sandbox per element

Reproducible build environment

Cached

Parallel builds

Why building OCI images?

For our own infrastructure.

Helping existing continuous integration of applications building for Freedesktop SDK.

Some applications may have daemon and desktop frontend components.

Freedesktop SDK is the main project using entirely built with BuildStream.

Table of Contents

1 Introduction

3 OCI images

5 Conclusion

Example: GNU Hello

project.conf

elements

freedesktop-sdk.bst

components

hello.bst

Example: GNU Hello

project.conf

elements

freedesktop-sdk.bst

components

hello.bst

This is a junction element. Itrefers to an upstream Build-Stream project.

Example: GNU Hello

project.conf

elements

freedesktop-sdk.bst

components

hello.bst

This is the main build element.

Anatomy of a BuildStream element

kind: a u t o t o o l sbuild -depends:

- freedesktop -sdk.bst:public -stacks/buildsystem -autotools.bst

- freedesktop -sdk.bst:components/texinfo.bst

- freedesktop -sdk.bst:components/help2man.bst

depends:

- freedesktop -sdk.bst: boo t s t r ap−impor t . b s tvariables:

autogen: |./bootstrap --no-git \

--gnulib -srcdir=gnulib \

--skip -po

sources:

- kind: git_tag

url: https://git.savannah.gnu.org/git/hello.git

track: master

ref: v2.10-0-gdc7dc56a00e48fe6f231a58f6537139fe2908fb9

The “kind” selects the plugin

depends:

--skip -po

sources:

- kind: git_tag

track: master

List of build and runtime dependencies

depends:

--skip -po

sources:

- kind: git_tag

track: master

Some customization for the plugin

depends:

--skip -po

sources:

- kind: git_tag

track: master

Description of sources

What to do from there?

$ b s t b u i l d components / h e l l o . b s t...

$ b s t s h e l l components / h e l l o . b s t / u s r / b i n / h e l l o...

H e l l o , w o r l d !$ b s t c h e c k o u t components / h e l l o . b s t h e l l o −r o o t f s

Table of Contents

1 Introduction

3 OCI images

5 Conclusion

Typical build of OCI images

A base image probably from a distribution

A Dockerfile

Eventually use package manager to add dependenciesSequentially build some other dependenciesBuild your main projectOptionally, extract runtime files to an new image to remove development files (multi-staged)Configure

Hello container image w/ Dockerfile

FROM debian AS build

RUN apt−get updateRUN apt−get install −y git autoconf automake autopoint \gcc make texinfo help2man

RUN mkdir /buildWORKDIR /buildRUN g i t c l o n e h t t p s : // g i t . savannah . gnu . org / g i t / h e l l o . g i tWORKDIR /build/helloRUN git checkout dc7dc56a00e48fe6f231a58f6537139fe2908fb9

RUN . / b oo t s t r a p −−sk i p−poRUN . / configure −−prefix=/usr −−disable−dependency−trackingRUN make −j16RUN mkdir /installRUN make −j1 install DESTDIR=/install

FROM debian

COPY −−from=build /install /ENTRYPOINT /usr/bin/hello

Hello container image w/ Dockerfile

FROM debian AS build

RUN apt−get updateRUN apt−get install −y git autoconf automake autopoint \gcc make texinfo help2man

RUN mkdir /buildWORKDIR /buildRUN g i t c l o n e h t t p s : // g i t . savannah . gnu . org / g i t / h e l l o . g i tWORKDIR /build/helloRUN git checkout dc7dc56a00e48fe6f231a58f6537139fe2908fb9

RUN . / b oo t s t r a p −−sk i p−poRUN . / configure −−prefix=/usr −−disable−dependency−trackingRUN make −j16RUN mkdir /installRUN make −j1 install DESTDIR=/install

FROM debian

COPY −−from=build /install /ENTRYPOINT /usr/bin/hello

Some commands may download from ex-ternal sources. They break reproducibility.

Dependencies and Dockerfile

What if you require libraries not shipped in distribution?

Either make a package for the used distribution

Package manager deal with more complex situations: upgrade, uninstall, services,configuration files, user and permission managementsSandbox is optionalRebuild is not automatic

Build directly as Dockerfile

No support for common build systemsNo build dependencies between Dockerfile sCached: if one command is modified, all following commands have to be run againBuild sandbox has network by default, no reproducibility

OCI layering

OCI images are based on Dockefiles.

1 Dockerfile command = 1 image layer.

Download and storage can reuse common layers.

Table of Contents

1 Introduction

3 OCI images

5 Conclusion

Graph to layers - Non solution

Translate every element to a layer.

Topologically sort all layers.

Each image is a subsequence.

Issues:

OCI implementations index layers by stack hash (ChainID) rather then layer hash(DiffID), so no subsequence.

Some implementations or filesystem backends might not scale with hundreds of layers.

Our approach

Developer decides of sensible layers.

One BuildStream element per layer.

Each layer makes also an image.

Elements use dependencies to copy layers from other OCI images

Freedesktop SDK full dependency graph

Let’s simplify this graph a bit...

OCI image layering in Freedesktop SDK

Dependency graph platform.bst

sdk.bst

bootstrap.bst

platform-oci.bst

sdk-oci.bst

bootstrap-oci.bst

flatpak.bst flatpak-oci.bst

debug-oci.bst

bootstrap

platformbootstrap

sdkplatformbootstrap

sdk.bst

bootstrap.bst

platform-oci.bst

sdk-oci.bst

bootstrap-oci.bst

debug-oci.bst

Each OCI element builds an image.

The dependencies between OCI elementsform the layers.

bootstrap

platformbootstrap

sdk.bst

bootstrap.bst

platform-oci.bst

sdk-oci.bst

bootstrap-oci.bst

debug-oci.bst

Each OCI element builds an image.

The dependencies between OCI elementsform the layers.

bootstrap

platformbootstrap

The BuildStream OCI plugin

OCI or Docker 1.2 (with legacy compatibility)

Enable/disable layer compression

Configuration, annotations, history comments.

Multi-image

Let’s go back to our example

project.conf

elements

freedesktop-sdk.bst

components

hello.bst

Layer element

kind: oci

build -depends:

- freedesktop -sdk.bst:oci/bootstrap -oci.bst

- components/hello.bst

config:

mode: o c iimages:

- os: linux

architecture: amd64parent:

element: o c i / boo t s t r ap−o c i . b s tlayer:

- components/ h e l l o . b s tcomment: "Import GNU hello"

config:

Entrypoint: [ "/usr/bin/hello" ]

Select the OCI plugin

Layer element

kind: oci

build -depends:

config:

mode: o c iimages:

- os: linux

config:

We need the base image one which webuild the layer.And the elements to build the currentlayer.

Layer element

kind: oci

build -depends:

config:

mode: o c iimages:

- os: linux

config:

Use OCI specifications

Layer element

kind: oci

build -depends:

config:

mode: o c iimages:

- os: linux

config:

Architecture description

Layer element

kind: oci

build -depends:

config:

mode: o c iimages:

- os: linux

config:

The base image which we build the layer on

Layer element

kind: oci

build -depends:

config:

mode: o c iimages:

- os: linux

config:

The elements included in the layer

Layer element

kind: oci

build -depends:

config:

mode: o c iimages:

- os: linux

config:

Comment for the history

Layer element

kind: oci

build -depends:

config:

mode: o c iimages:

- os: linux

config:

Configuration of the image

Building the image

$ b s t b u i l d o c i / h e l l o . b s t...

$ b s t c h e c k o u t o c i / h e l l o . b s t −−t a r h e l l o . t a r...

$ podman l o a d − i h e l l o . t a r

Table of Contents

1 Introduction

3 OCI images

5 Conclusion

Conclusion

Fully build OCI images with one tool

Cached, reproducible, parallel

Customizable layers to optimize storage and network

Freedesktop SDK provides a basic SDK with the most common system dependencies

Pointers

Freedesktop SDK https://gitlab.com/freedesktop-sdk/freedesktop-sdk

BuildStream https://buildstream.build/

OCI plugin doc https://buildstream.gitlab.io/bst-external/elements/oci.html

Docker images https://hub.docker.com/u/freedesktopsdk

This work was sponsored by Codethink.

Building Open Container Initiative images based on ......Building Open Container Initiative images...

Documents

RECOMMENDED MINIMUM SAFETY …RECOMMENDED MINIMUM SAFETY SPECIFICATIONS FOR QUAY CONTAINER CRANES A joint initiative from TT Club, ICHCA International and Port Equipment Manufacturers

Student Sustainability Initiative · Web viewAssess dining hall compost system Evaluate opportunities to expand Eco-to-go container program Assess biofuel production opportunities

Singamas Container Holdings · PDF fileSingamas Container Holdings Limited (“ Singamas” or “the Company”) is the world’s ... Platform Container Open-side Container Container

Container 2...- Container image repositories - Spreadsheet scheduling OS OS Machine Infrastructure Machine Machine Container Runtime Container Runtime Container Runtime Service Service

S P TUDY ON RIVATE-INITIATIVE … ON S PRIVATE-INITIATIVE INFRASTRUCTURE PROJECTS IN DEVELOPING COUNTRIES IN FY2011 STUDY ON THE NEW CEBU CONTAINER PORT AND THE REDEVELOPMENT OF THE

Red Hat OpenShift Container Platform - Meetupfiles.meetup.com/18882572/OpenShift Container Platform.pdf · Red Hat OpenShift Container Platform The container runtime & development

A Comparative Taxonomy and Survey of Public Cloud ... · Container Service AWS EC2 Container Service Azure Container Service Container Engine Container Service Serverless Computing

Optimisation of Container Process at Multimodal Container Terminals …eprints.qut.edu.au/16626/1/Andy_Wong_Thesis.pdf · Optimisation of Container Process at Multimodal Container

Container Security Initiative - DTICThe Container Security Initiative is a revolutionary program to extend our zone of security by pre-screening containers posing a potential security

Codethink elce 2017_maintaining_a_linux_kernel_for_13_years_you_must_be_kidding_me_we_need_at_least_30_abenito_bhutchings

Ben Dooks © 2017 Codethink Ltd. · Hardware Physical scanner closed design – No longer needs liquid helium Aquisition to 600 channels, 24bit data, 72kHz max – About 3400KB/sec

ASIA - CONTAINER BROCHURE 2019 48 UPDATED · Container Dimensions METRIC STANDARD EXTERNAL CONTAINER DIMENSIONS STANDARD INTERNAL CONTAINER DIMENSIONS Container Length Container Width

The Open Container Initiative

You Must be Kidding Me. We Need at Least 30! Ben Hutchings / Agustín Benito Bethencourt ELCE, Prague, 24th October 2017 The speakers: Ben Hutchings Kernel developer at Codethink

CONTAINER PORT ELEMENT - Tacomacms.cityoftacoma.org/Planning/Container Port Element/Container Port...Adopted 7/22/14, Amd. Ord. No. 28229 Container Port Element CP-3 A container port

OpenShift Container Platform 4 - access.redhat.com · OpenShift Container Platform 4.3 Container-native virtualization Container-native virtualization installation, usage, and release

OpenShift Container Platform 4 - access.redhat.com · OpenShift Container Platform 4.2 Container-native virtualization Container-native virtualization installation, usage, and release

Container Security: A Proposal for a Comprehensive Code of ... 9 Container Security.pdfthe Container Security Initiative (CSI) and the Proliferation Security Initiative (PSI). These

Panel: The Open Container Initiative at 12 months with Chris Aniszczyk, Rob Dolin, Jeff Borek, Mrunal Patel and Michael Crosby

The Mythical App Container - Open Source Conference · 15 Thus we need... • A standard that ‒ declares the content in a way that is vendor neutral ‒ The Container Spec initiative