TUT 8118

SUSE® Studio Onsite in the Datacenter

Andreas ThomasDesignated Support Engineer

athomas@novell.com

Ralf DannertSystems Engineer

rdannert@suse.com

2

Agenda

• Overview SUSE Studio

• SUSE Studio and SUSE Manager Integration

• SUSE Studio POCs in Datacenter environments‒ Challenges in Build Environment

‒ Challenges in Deployment

‒ More Troubleshooting

• Q&A

Overview SUSE Studio

4

SUSE Studio Onsite: Key Features

• Supported Linux in Minutes

• Integrated Testing(testdrive)

• Multiple Platforms‒ Live CD/DVD, preload ISO/USB, HDD, PXE, Xen, vmdk, OVF,

Amazon EC2)

• Supportability Analyzer

• Automated Dependency Discovery

• Multi-Host Staged Delivery

• SUSE Gallery

5

SUSE Studio Onsite (I/II)

‒ Behind the firewall, on-premise, installable and fully supported version

‒ similar core functionalities as SUSE Studio Online

‒ Delivered as software appliance, requires bare metal server to install

‒ Shipped as raw disk image and a bootable CD containing the raw disk image

‒ SUSE Studio is proprietary software and delivered based on a software licensing model

6

SUSE Studio Onsite (II/II)

• SUSE Studio needs access to the following types of repositories used to build the appliance:

‒ Installation repositories: Pool repositories

‒ Update repositories:

‒ directly from NCC/SCC

‒ use Subscription Management Tool (SMT) for SLE 11 SP3 to mirror the update repositories from NCC

• create appliances using the following base Operating Systems:

‒ SLE{DS} 10 SP4 x86 and x86_64

‒ SLE{DS} 11 SP1,SP2,SP3 x86 and x86_64

‒ SLE{DS} 12 with Maintenance Update after SLE 12 Release

7

Challenges Addressed by SUSE Studio Onsite

• Reduced Image footprint – JeOS

• Standardization and Documentation

• Fast iterative development process of a custom distribution

‒ Import changed files from testdrive

‒ Overlay files

• Use KIWI for additional functionality later

9

SUSE Studio Architecture

SUSE Studio and SUSE Manager Integration

11

SUSE Manager

✔ Optimize ✔ Control ✔ Innovate

12

Background: SUSE Manager Staging

‒ SUSE Manager: move channels errata into the next stage/environment for ongoing phased testing/rollout

‒ natural progression from DEV to QA to PROD

‒ Formerly: spacewalk-clone-by-date

‒ now: spacewalk-channel-patch-lifecycle

‒ spacewalk-manage-channel-lifecycle --promote --phases=DEV,QA,PROD -c sles11-sp3-pool-x86_64

13

SUSE Manager IntegrationBuild appliances from stages Dev/Test/Prod/

‒ cobbler used as installation source for a distribution

‒ Prerequisite: distribution exists and child channels available

‒ SUSE Manager:

‒ Systems → Autoinstallation → Distributions → lable

‒ parent channel of child channel must be assigned to the distribution

‒ SUSE Studio: use these channels as repositories

‒ http://<susemanager>/ks/dist/child/<childchannel-label>/<dist-label>

‒ Admin User →Advanced → Add Repository

‒ Usecases:

‒ Easy (de)provisioning of repositories possible

‒ Snapshotted, reproducible builds

14

SUSE Manager IntegrationTroubleshoot

‒ URLs to integrate with SUSE Manager are not “normal URLs”

‒ "grep /ks *" in /etc/apache2/conf.d reveals that /ks/dist gets re-routed to "/rhn/common/DownloadFile.do"

‒ zz-spacewalk-www.conf:RewriteRule ^/ks/dist(.*)$

‒ /rhn/common/DownloadFile.do?url=/ks/dist$

‒ every URL starting with /rhn is passed to tomcat as of this rule:

‒ zz-spacewalk-www.conf:RewriteRule ^/rhn(.*) ajp://localhost:8009/rhn$1 [P]

Challenges in Build Environment

16

SUSE Studio and Datacenter

• Differences between datacenter and “old” purpose of studio usage

• Benefits of image deployment vs autoyast ‒ speed up of deploment (less reboots / no hardware probing)

‒ Faster development process (e.g. test drives, overlay files)

‒ Ideally: base image for deployment and customizing via software management stack

17

Using Your Own Repository(I)

18

Using Your Own Repository(II)

19

Using Your Own Repository(III) Declaring Pattern

20

Using Your Own Repository(IV)Updating Repository Data

Now we've got a reposory providing “our” RPMs and “our” software patterns

21

Include and Use it in SUSE Studio

Challenges in Deployment

23

Challenges in Deployment

• Disk-less Servers (boot from SAN, discovery of disks)

• Different Network Adapters / IP Address advertising

• Struggeling “unknown” Networks (blade center)

• Multipathing / host-based mirror requirements

• Use of “own” Patterns and custom RPMs

• Kiwi version in Studio / Containment‒ Upstream kiwi with fixes / features not yet in Studio

‒ Awareness of Product Management to update kiwi version in products

‒ There's always more than one way to do it (pxe, initrd, kiwi)

24

Disk-less Server / Boot from SAN

• Only plain SCSI disks are being detected / supported out of the box

• Need support for /dev/disk/by-*/scsi-XXXX

• Solution:‒ Current Kiwi version

‒ Upcoming Maintenance Update

25

Solving Software Dependencies

‒ Solving dependencies in Studio is based on zypper mechanics

‒ Repositories based on zypper

‒ Refresh of repositories

‒ Priorities of repositories

‒ Custom change in Studio (use old version of RPM as default)

‒ Self created repositories (using createrepo)

‒ Custom templates as FATE

26

Choose Non-default rpm Versions

‒ choose a specific software version from another repository manually

‒ visible in Build → Configuration → Selected software

‒ if version attached to name → have been manually selected

‒ lower priority of repository in /srv/studio/options.yml

‒ add repos_with_lower_priority: according to

‒ Changing Repository Order from the SUSE Studio Onsite Deployment And Administration Guide

‒ example: Add OBS repository

‒ Admin User → Repositories →Add repository

‒ Name: python

‒ URL: http://download.opensuse.org/repositories/devel:/languages:/python/SLE_11_SP3/

27

Software Management

• Adjustments in post build script (strip down)

• firstboot_script

• Adding repositories during bootup/firstboot

• Limitations of 3rd party rpms‒ Scripts in RPMs (e.g. add users/permissions/acls)

‒ Boot-related rpm (missing bootloader file)

‒ Custom templates as FATE

28

RPMs Break Build Process

WHY?

• Scripts in 3rd party RPMs may use acls ‒ Fixes in the meantime

• Scripts in RPMs might require boot-related files‒ /boot/grub/menu.lst, /etc/fstab etc.

Solutions:

‒ Fix RPM script (if possible)

‒ Install after deployment (firstboot script)

‒ Using “wrapper RPMs” if not network accessible (e.g. test drive)

‒ add a repository using “zypper ar” during firstboot and install afterwards

29

No Network... No PXE Deployment

• Root cause: Stripping of unnecessary packages after build

• Solution: Include kernel-firmware package in bootrequired, specify dedicated

30

...And Strange Behaviours

• First DHCP request is taken for Network configuration

• BUT: Some blade centers run their own DHCP server

• 169.X.X.X is not routed to “our” TFTP for image rollouts

• Possible Solutions: ‒ 1. specify MAC/NIC assingments as append parameter

‒ 2. build initrd having a recent KIWI build environment

‒ 3. Edit initrd (see custom initrd) to limit NICs used for discovery

31

...Just Specify to Your Demands

DEFAULT KIWI-Boot

LABEL KIWI-Boot

kernel boot/kernel_new

append initrd=boot/initrd_new vga=0x314 kiwiserver=4.239.87.130 PXE_IFACE=eth0 lang=de_DE insmod=bnx2 netwait=90 netretry=5 prefer_iface=eth0 BOOTIF=eth0

IPAPPEND 1

32

Building a Custom Image

• How to include‒ Required firmware (e.g. include kernel-firmware package)

• Troubleshooting tools (less, vim, util-linux, sshd)

• troubeshooting initrd vs roll-out initrd

• Use of hooks to extend initrd

• Business as usual, integration in process is the key

33

Export – Adapt – Build – Deploy (0)

34

Building Appliances Locally with KIWI

‒ Build tab, scroll down and select Export your appliance's KIWI configuration

‒ have latest version from the Open Build Service repository Virtualization: Appliances

‒ sudo ./create_appliance.sh

‒ specify repository URL for internal (non-public) repositories

‒ <repository type='rpm-md'>

‒ <source path='{SLES 11 SP3 Updates i386}'/>

‒ </repository>

‒ README for Kiwi source from SUSE Studio

35

Export – Adapt – Build – Deploy (I)

36

Export – Adapt – Build – Deploy (II)Kiwi Hooks

37

Export – Adapt – Build – Deploy (III)config.xml for PXE

38

Export – Adapt – Build – Deploy (IV)Pattern in Kiwi

39

Export – Adapt – Build – Deploy (V)toolchain in bootincludes

40

Export – Adapt – Build – Deploy (VI)Adapt Repositories

41

Export – Adapt – Build – Deploy (VII)Kiwi Hooks

‒ RAID 1 (mirror) supported out of the box

‒ Multipath and DM support in recent kiwi

‒ Need to use hook functions for enablement

42

Export – Adapt – Build – Deploy (VIII)

● Copy output to tftp

43

Export – Adapt – Build – Deploy (IX)tftp

44

Export – Adapt – Build – Deploy (X)PXE

45

Export – Adapt – Build – Deploy (XI)

46

Export – Adapt – Build – Deploy (13)

More Troubleshooting

48

Some Hints

• Use kiwidebug=1 parameter in PXE configuration‒ Emergency shell

‒ Detailed log in /var/log/boot.kiwi

• Build a debug initrd with “your” tools

• Use KIWI_FORBID_HOOKS=1 to eleminate “your bugs”

• env output helps you to verify what went wrong

• /include helps you to understand systems behaviour

49

How to Debug a Containment(I)

• What is a containment

• %description‒ Containment appliance to build studio images secured by a

VM layer

‒ tar -cjf $RPM_SOURCE_DIR/$NAME-$VERSION-$RELEASE-vmx.tar.bz2 $SOURCE metadata

‒ rpmbuild -ba $FILES_DIR/image.spec

‒ https://github.com/openSUSE/containment-rpm

50

How to Debug a Containment(II)

‒ minor debugging session in containment:

‒ in kiwi-job/lib/containment.rb, set self.debug to 1

‒ start the build. roughly after 'downloading packages', you'll see something like:

‒ Containment running in debug mode. SSH to 172.16.X.2 on the runner

‒ /root inside the guest is mounted as /home/containX on the host

‒ rm /etc/studio/setup_done; rm /etc/studio/debug

‒ /bin/run-contained:

‒ Delete # setup done

‒ touch "$SETUP_DONE"

‒ [ $DEBUG = 1 ] && touch "$DEBUG_MODE"

‒ to keep logfiles after build delete also the last halt -fp

‒ start the build with: /bin/run-contained

51

References

• SUSE Studio‒ https://www.suse.com/products/susestudio/

‒ https://www.suse.com/documentation/suse_studio/

‒ https://susestudio.com/

‒ Best Practice document(email rdannert at suse.com)

• Kiwi‒ git://github.com/openSUSE/kiwi

‒ https://en.opensuse.org/Portal:KIWI

‒ irc: #opensuse-kiwi

52

Unpublished Work of SUSE LLC. All Rights Reserved.This work is an unpublished work and contains confidential, proprietary and trade secret information of SUSE LLC. Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.

General DisclaimerThis document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for SUSE products remains at the sole discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.