Transactional Systems: Examples

Core OS / RedHat:

Various:

SUSE:

Common Properties of Transactional Systems

Transactional / atomic updates• Read-only root file system• Focus on large deployments (clouds)

or embedded systems• Minimal base system• Automatic updates / reboots• Integrity protection

What Is a Transactional Update?

A Transactional update is an update that● Is atomic

– Either fully applied or not applied at all– Update does not influence the running system

● Can be rolled back– A failed or incompatible update can be quickly discarded to

restore the previous system condition–➔ This definition is valid for all atomic distributions➔ Atomicity guaranteed by rebooting the system➔ Apart from that, implementations vary widely

Transactional Updates in the SUSE World

SUSE Tools

● Zypper for package management

● Snapper for snapshot creation

● Btrfs as default file system

Updates with snapper

current/

Backup/ (pre)

Backup/ (post)

Active system

1. Create “pre” snapshot2. Update the current system3. Create “post” snapshot

• Update is modifying the currently active file system

• Restarts services immediately

Updates with snapper

A Transactional Update is an update that● Is atomic

– Either fully applied, or not applied at all– Update does not influence the running system

● Can be rolled back– A failed or incompatible update can be quickly discarded to

restore the previous system condition

Updates with transactional-update

current/ (ro)

new/

1. Create new target snapshot2. Update system, make snapshot

read-only and set as default for next boot

● Current root file system is not modified

● No backup snapshots required, due to ro system

● Will not do anything if there are no updates

● Discard snapshot in case of errors

Active system

Updates with snapper

A Transactional Update is an update that● Is atomic

– Either fully applied, or not applied at all– Update does not influence the running system

● Can be rolled back– A failed or incompatible update can be quickly discarded to

restore the previous system condition

➢ Reboot will provide atomicity

Helper Applications: health-checker

● Add your own checker scripts to check for system consistency● Automatic rollback if checks fail on first boot● GRUB2: Select old kernel if system wasn’t able to complete initrd

current/ (ro)

new/

Comparison & Usage

Comparison to Other Read-Only Systems

● No separate tooling or infrastructure needed● Works with any standards-compliant RPM package● General-purpose tool: especially useful for servers and clusters● Fast snapshot switching● Sane /etc and /var handling● Only works with BTRFS root file systems

Live Demo

Cheat SheetTransactional Updates

List repositorieszypper lr -dzypper lr -d

Refresh repositorieszypper refzypper ref

Update installed packagestransactional-update uptransactional-update up

Perform a distribution updatetransactional-update duptransactional-update dup

Install package(s)transactional-update pkg in <name>transactional-update pkg in <name>

Update package(s)transactional-update pkg up <name>transactional-update pkg up <name>

Remove package(s)transactional-update pkg rm <name>transactional-update pkg rm <name>

List snapshotssnapper listsnapper list

Mark snapshots for removal by snappertransactional-update cleanuptransactional-update cleanup

View default subvolumebtrfs subvolume get-default /btrfs subvolume get-default /

Open shelltransactional-update shelltransactional-update shell

Request reboottransactional-update reboottransactional-update reboot

System rollbacktransactional-update rollback [number]transactional-update rollback [number]

A Deeper Look

Handling of Special Directories

Writable directories on an otherwise read-only system:●/var●/etc

Snapshots are only taken of the root file system by default!

/var handling/var handling

● /var is a special directory because it contains variable data● Has to have read-write permissions● Cannot be rolled back● A rollback would usually delete production data (e.g., databases or

Docker images)● Typically stored on a separate subvolume or partition● /var will not be mounted into the update snapshot (i.e. packages can

not modify it)● Special handling: directories will be created on next boot● Files can be created with systemd-tmpfiles; for more complex

initialization or migration tasks, use systemd services on first boot

/etc handling

● On read-only systems, /etc has to be writable● Mounted as an overlay file system● Overlays stored in /var/lib/overlays● Each snapshot has one corresponding /etc overlay

● Overlay will be mounted during the update● Before: Only one overlay for all snapshots

● /etc overlays are stacked● Changes accumulated before the next boot will be visible both in the

current and the next snapshot (if not changed in both overlays)● On rollback, only the layers up to that snapshot will be mounted

/etc handling

● To prevent the overlayfs stack from growing infinitely, the /etc state of the oldest available snapshot is synchronized into the new snapshot

● The /etc directory in a snapshot (/.snapshots/<num>/snapshot/etc) alone does not represent a consistent system state!

● Do not mess with the files in /var/lib/overlays (although, you can use them as a backup source if you want)

overlay /etc overlay defaults,upperdir=/sysroot/var/lib/overlay/18/etc,lowerdir=/sysroofor=/var/lib/overlay,x-system d.requires-m ounts-for=/sysroot/var,x-system d.requires-m ounts-for=/sysroot/var/lib/overlayoverlay /etc overlay defaults,upperdir=/sysroot/var/lib/overlay/18/etc,lowerdir=/sysroofor=/var/lib/overlay,x-system d.requires-m ounts-for=/sysroot/var,x-system d.requires-m ounts-for=/sysroot/var/lib/overlay

Other Directories

● /opt, /var/log and /boot/grub2 will be bind mounted into the update snapshot

● Everything else, including /srv, won’t!

➔ Packages have to follow the FHS and packaging guidelines

Mount Summary

/ (ro)

/

Active system

Visible duringupdate

/var /etclayer x

/etclayer x+1

/opt

/opt

Bind mounts:

/var/log

/var/log

/bogrubarc

/bogrubarc

/srv/home

...

Utilities

Helper Applications: rebootmgr

● transactional-update.timer triggers daily update including reboot● rebootmgr manages reboot (e.g. in maintenance windows or

synchronized via etcd)● SUSE CaaS Platform users: not used there; use Velum to trigger

cluster-wide reboots instead

Other

● Configuration file: /etc/transactional-update.conf● Snapper will clean up old snapshots● transactional-update is the only way to update a read-only system● Zypper integration is planned

Pitfalls

● Snapshots will be branched from the current system→ snapshots will not contains the previous snapshot’s contents if the system hasn’t been rebooted!

● When using transactional-update on a read-write system→ Don’t forget to reboot your system before making any changes to the root file system!

What’s Next?

Availability

SUSE Linux Enterprise Server Service Pack 1:● Technology Preview● Compatible with most

packages (~30 packages left out of 10,000)

● See documentation for a list of unsupported packages

Availability

Future Development

● Integration into SUSE Linux Enterprise Server● IMA / EVM support for system verification / integrity● Integrate transactional-update as zypper plugin● Fix remaining RPM packages with scripts modifying /var and /srv