Theory and Practice of Linux System Administration

SystemV vs systemd

All Things Open

October, 2015
This presentation is taken from my class
Theory and Practice of Linux System Administration

Legal Stuff

Linux is a registered trademark of Linus Torvalds

Red Hat is a registered trademark of Red Hat, Inc.

Unix is a registered trademark of AT&T

Other marks are the trademarks of their respective owners

Copyright 2015 David P. BothContent licensed under the terms of the Creative Commons Attribution-ShareAlike 4.0 International license ("CC-BY-SA")

David Both
dboth@millennium-technology.com
David.Both@OpenSource.com

RHCE, SCSA

Introducing systemd

Developed byLennart Poettering

Kay Sievers

ObjectivesFix outdated SystemV startup

Start lessOnly daemons that are actually needed

Start more in parallel

Improve support for hotplug hardware

Maintain compatibility with SystemV start scripts

systemd Functions

A system and service manager (manages both the system and its services)

A software platform (serves as a basis for developing other software)

The glue between applications and the kernel (provides various interfaces that expose functionalities provided by the kernel)

Strategy

At least 2 ways to ensure that a service is available when it is neededKeep track of all other services which may need it and be sure to start things in the right order

Wait until some task tries to connect to the service and start it on demand

systemd takes the second approach

Similar to MacOS launchd

Implementation

Create sockets for servicesSerialization during startup was due to waiting for dependent services sockets to be created

When a connection request arrives on a specific socket, the associated daemon is started

Detailed systemd informationhttp://0pointer.de/blog/projects/systemd.html

http://www.freedesktop.org/wiki/Software/systemd

systemd by Release

Started showing up in about Fedora 14Available as alternative

Fedora 15First release with systemd as default for many services

Conversion essentially complete by Fedora 17

CentOS 7

Benefits

Speed startupStart as much in parallel as possible

Only start what needs to be started

Manage dynamic hardware hotplugStatic hardware managed like dynamic

systemd Pros and Cons

Not portableOnly works in Linux

Opportunity to do things better for Linux

More binary executablesFaster

Fewer shell scriptsLess discoverable

Open Source code

ASCII Text configuration filesStill provides openness

Linux Boot Sequence

BIOS POST

Checks basic operability of hardware

INT 13H Locates Boot Sector

Boot Loader Stage 1

GRUB

LILO

GRUB

Stage 1 Loads GRUB Stage 1.5

File system specific

Loads Stage 2

/boot/grub/grub.conf

GRUB

GRUB locates the kernel image and loads it

Located in /bootStandard EXT3/4 partition

Cannot be LVM

Located at beginning of disk

Usually loads an initrd image

Select from multiple kernels

grub.conf

Defines grub menu options

Allows selection of boot kernel

Edit kernel boot parameters

Specifies kernel boot parameters

Boot hard drive

GRUB2

Introduced in Fedora 16

Command based pre-OS environment

GRUB2 Locates the Kernel image and loads it

Located in /boot/grub2

Standard ext3/4 partition

Cannot be LVM

Located at beginning of disk

May also load initrd image

Multiple kernel

Kernel

Uncompresses itself into RAM

Loads device drivers

Mounts /

Launches init or systemd

Kernel Boot Messages

Ring buffer

Console

dmesg command

/var/log/messagesgrep kernel:

/var/log/dmesgDiscontinued with systemd

Never used by many distros

Timestamps in square bracesRecent releases of Fedora

Sample DMESG Log

[ 0.000000] NR_IRQS:2304[ 0.000000] CPU 0 irqstacks, hard=f3c18000 soft=f3c1a000[ 0.000000] Console: colour VGA+ 80x25[ 0.000000] console [tty0] enabled[ 0.000000] Fast TSC calibration failed[ 0.000000] TSC: Unable to calibrate against PIT[ 0.000000] TSC: HPET/PMTIMER calibration failed.[ 0.000000] Marking TSC unstable due to could not calculate TSC khz[ 0.016998] Calibrating delay loop... 2027.52 BogoMIPS (lpj=1013760)[ 0.038997] pid_max: default: 32768 minimum: 301[ 0.043996] Security Framework initialized[ 0.043996] SELinux: Initializing.[ 0.044996] SELinux: Starting in permissive mode[ 0.049996] Mount-cache hash table entries: 512[ 0.076993] Initializing cgroup subsys ns[ 0.077993] ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.[ 0.077993] Initializing cgroup subsys cpuacct[ 0.079993] Initializing cgroup subsys memory[ 0.081992] Initializing cgroup subsys devices[ 0.081992] Initializing cgroup subsys freezer[ 0.082992] Initializing cgroup subsys net_cls[ 0.082992] Initializing cgroup subsys blkio[ 0.086992] mce: CPU supports 0 MCE banks

Devices

Device files located in /dev

Previously all possible device files created at installationTens of thousands

Most unused

NOT device drivers

Device file major/minor codesAssigned by Linux assigned names and numbers authority (LANANA)

Current device list can be found on web site

Devices

Enter udevCreates only as required mostly

Treats all devices as plug'n'pray

Reduces /dev/chaos

udev rules

DBUS

Inter-Process Communications (IPC)Daemon for kernel

User space for applications

New devices plugged inTriggers udev to create a new device file

Application coordinationManage communicationsMultiple instances of one application

Multiple different applications

DBUS rules

Linux Startup

Boot vs Startup

Boot ends with loading the kernel and init or systemd

Startup begins when the init or systemd process takes control

initThe entire startup process is handled by scripts

systemdBinary executables

Configuration files

Compatible with SystemV init scripts

Upstart

Was to have replaced traditional start scripts and /etc/inittab

/etc/init

Changeover started in some distros

Fedora

CentOS

Ubuntu

Quickly superseded by systemd

Traditional Startup

Using init and SystemV Start Scripts

/sbin/init

ELF Binary

Controls Linux initialization and runlevel startup

Mother of all processes

PID #1

Startup Files

Configuration files/etc/inittab

/etc/fstab

Log Files/var/log/dmesg

/var/log/boot.log

/etc/inittab

Defines runlevels 0 through 6

Specifies default runlevel

gettys for defined runlevels

Ctrl-Alt-Del action

UPS power failure actionsRarely used

inittab 1

## inittab This file describes how the INIT process should set up# the system in a certain run-level.## Author: Miquel van Smoorenburg, # Modified for RHS Linux by Marc Ewing and Donnie Barnes#

# Default runlevel. The runlevels used by RHS are:# 0 - halt (Do NOT set initdefault to this)# 1 - Single user mode# 2 - Multiuser, without NFS (The same as 3, if you do not have networking)# 3 - Full multiuser mode# 4 - unused# 5 - X11# 6 - reboot (Do NOT set initdefault to this)

inittab 2

#id:5:initdefault:

# System initialization.si::sysinit:/etc/rc.d/rc.sysinit

l0:0:wait:/etc/rc.d/rc 0l1:1:wait:/etc/rc.d/rc 1l2:2:wait:/etc/rc.d/rc 2l3:3:wait:/etc/rc.d/rc 3l4:4:wait:/etc/rc.d/rc 4l5:5:wait:/etc/rc.d/rc 5l6:6:wait:/etc/rc.d/rc 6

# Trap CTRL-ALT-DELETEca::ctrlaltdel:/sbin/shutdown -t3 -r now

inittab 3

# When our UPS tells us power has failed, assume we have a few minutes# of power left. Schedule a shutdown for 2 minutes from now.# This does, of course, assume you have powerd installed and your# UPS connected and working correctly.pf::powerfail:/sbin/shutdown -f -h +2 "Power Failure; System Shutting Down"

# If power was restored before the shutdown kicked in, cancel it.pr:12345:powerokwait:/sbin/shutdown -c "Power Restored; Shutdown Cancelled"

# Run gettys in standard runlevels1:2345:respawn:/sbin/mingetty tty12:2345:respawn:/sbin/mingetty tty23:2345:respawn:/sbin/mingetty tty34:2345:respawn:/sbin/mingetty tty45:2345:respawn:/sbin/mingetty tty56:2345:respawn:/sbin/mingetty tty6

# Run xdm in runlevel 5x:5:respawn:/etc/X11/prefdm -nodaemon

/etc/fstab

Common to all startup systems

Specifies disk partitions to mount

Some directories must be part of / (root)

/etc

/dev

/bin

/sbin

/lib

/usrFor only a couple releases of Fedora

Typical fstab

## /etc/fstab

# Created by anaconda on Thu Jun 9 15:15:28 2011

#

# Accessible filesystems, by reference, are maintained under '/dev/disk'

# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info

#

/dev/mapper/vg_instructor-root/ext4defaults1 1

UUID=d1a58f39-5002-4150-933b-131fecb106c5/bootext4defaults 1 2

/dev/mapper/vg_instructor-home/homeext4defaults1 2

/dev/mapper/vg_instructor-tmp/tmpbtrfsdefaults1 2

/dev/mapper/vg_instructor-usr/usrext4defaults1 2

/dev/mapper/vg_instructor-var/varext4defaults1 2

/dev/mapper/vg_instructor-swapswapswapdefaults0 0

tmpfs/dev/shmtmpfsdefaults0 0

devpts/dev/ptsdevptsgid=5,mode=6200 0

sysfs/syssysfsdefaults0 0

proc/procprocdefaults0 0

Another fstab

/dev/mapper/vg_david1-root/ ext4 defaults 1 1/dev/mapper/vg_david2-Virtual/Virtual ext4 defaults 1 2#UUID=d352f5a0-f399-4d9c-b46f-02dd81dca3f8/boot ext4 defaults 1 2#/dev/sda1 /boot ext4 defaults 1 2LABEL=/boot/boot ext4 defaults 1 2/dev/mapper/vg_david2-home/home ext4 defaults 1 2/dev/mapper/vg_david2-stuff/stuff ext4 defaults 1 2/dev/mapper/vg_david1-tmp/tmp ext4 defaults 1 2/dev/mapper/vg_david1-usr/usr ext4 defaults 1 2/dev/mapper/vg_david1-usrlocal/usr/local ext4 defaults 1 2/dev/mapper/vg_david2-var/var ext4 defaults 1 2/dev/mapper/vg_david1-swapswap swap defaults 0 0# Various USB HDDs# Using labels makes it possible to mount at the same location consistentlyLABEL=WD-500GB-USB/media/WD-500GB-USB ext4 auto,owner,user,defaults 0 0LABEL=Iomega-1TB-USB/media/Iomega-1TB-USB ext4 auto,owner,user,defaults 0 0LABEL=Iomega-160GB-USB/media/Iomega-160GB-USB ext4 auto,owner,user,defaults 0 0tmpfs/dev/shm tmpfs defaults 0 0devpts/dev/pts devpts gid=5,mode=620 0 0sysfs/sys sysfs defaults 0 0proc/proc proc defaults 0 0

/etc/rc.d/rc.sysinit

Bash script started by init

Manages other startup processes

Set hostname

Start SELINUX

Set the clock

Load modular kernel device driver modules

/etc/modprobe.conf

Remount / as rw

Mount other file systems

Sets quotas

etc...

/etc/rc.d/rc

Bash script

Started by init after rc.sysinit ends

Changes runlevels

rc scripts

Start Scripts

/etc/rc.d/init.d

/etc/rcX.dLinks

K = Kill

S = Start

service commandservice start|stop| restart|status|reload ...

SystemV Runlevels

0Poweroff1,S,sSingle User2Multi-user, no NFS

3Normal, multi-user with NFS

4Unused but basically same as 3

5GUI

6Reboot

Runlevels

/etc/rc.d/rc

Script to change runlevels

Kill scripts first

Then Start scripts

rc scripts in /etc/init.d

K or S links in /etc/rc.d/rcX.d

Links to directories in /etc/init.d

Runlevel Management
SystemV

init or telinit

Change runlevel now

chkconfig

For persistent configuration

The service command

Turn on/off now

Default runlevel/etc/inittab

StartupUsing systemd

systemd Units

Service

Most obvious kind of unitDaemons that can be started, stopped, restarted, reloaded

For compatibility can also read classic SysV init scriptsIn particular parse the LSB header, if it exists

/etc/init.d is just another source of configuration

systemd Units

Socket

Encapsulates a socket in the file-system or on the InternetCurrently support AF_INET, AF_INET6, AF_UNIX sockets of the types stream, datagram, and sequential packet

Classic FIFOs as transport

systemd Units

Each socket unit has a matching service unitService started when the first connection comes in on the socket or FIFO

Example: nscd.socket starts nscd.service on an incoming connection

Sockets can be viewed with netstat command

systemd Units

Device

Encapsulates a device in the Linux device tree

If a device is marked for this via udev rules, it will be exposed as a device unit in systemd

Properties set with udev can be used as configuration source to set dependencies for device units

systemd Units

Mount

Encapsulates a mount point in the file system hierarchysystemd monitors all mount points, how they come and go, and can also be used to mount or unmount mount-points

/etc/fstab used as an additional configuration source for mount pointssimilar to how SysV init scripts can be used as additional configuration source for service units.

systemd Units

Automount

Encapsulates an automount point in the file system hierarchy

Each automount unit has a matching mount unit, which is started (i.e. mounted) as soon as the automount directory is accessed

systemd Units

Target

Logical grouping of unitsInstead of actually doing anything by itself it simply references other units, which are controlled together

Examples for this are: multi-user.target equivalent to run-level 3 on classic SysV system

bluetooth.target pulls in bluetooth related services that otherwise would not need to be started: bluetoothd and obexd, etc.

graphical.target depends upon multi-user.target

systemd Units

Snapshot

Similar to target units Snapshots do not actually do anything themselves

Purpose is to reference other units

Save/rollback the state of services and units of the init system

Two intended use casesAllow the user to temporarily enter a specific state such as "Emergency Shell"

Ease support for system suspending

Control Groups

cgroups define groups of processes

Process aggregationControl of cgroups

Resource allocation

Runaway process spawn

/cgroup directoryLet me know if you find anything here

/sys/fs/cgroup/systemd/

/proc//cgroup

systemd

Configuration/etc/systemdModify here

/lib/systemdDon't change this

Managementsystemctl

init command is softlink to systemd

The service command still works for many servicesRedirected through systemd

systemctl Command

List running unitssystemctl

List all unitssystemctl -a list-units

Start and stopsystemctl start|stop

Display device statussystemctl status udisks2.service

Disable startup of a servicesystemctl disable telnet.service

systemctl

systemctl -asystemctl stop|start cups.service

cgroups

Cgroup treesystemd-cgls

System slice

User slice

PS with Cgroupsps xawf -eo pid,user,cgroup,args

systemd Targets

DefaultSymlink to the desired run target

/etc/systemd/system/default.target

multiuser multi-user.target

graphicalgraphical.target

Still has target runlevels 0-6/lib/systemd/system

Runlevels 2,3,4 all point to multi-user.target

systemd Targets

Single UserRecovery mode

Somewhat more functionality than SystemV runlevel 1 or Single User mode

Target related commands

List all targetssystemctl list-units type=target

Set default targetsystemctl set-default .target

Display current default targetsystemctl get-default

Changing the Default Target

systemctl set-default .target

ln -sf /lib/systemd/system/runlevel5.target /etc/systemd/system/default.target

ln -sf /lib/systemd/system/multi-user.target /etc/systemd/system/default.target

ln -sf /lib/systemd/system/graphical.target /etc/systemd/system/default.target

Thank you

SystemV vs systemd

Slide of

SystemV vs systemd

Slide of