Transactional Systems: Examples
Core OS / RedHat:
Various:
SUSE:
Common Properties of Transactional Systems
Transactional / atomic updates• Read-only root file system• Focus on large deployments (clouds)
or embedded systems• Minimal base system• Automatic updates / reboots• Integrity protection
What Is a Transactional Update?
A Transactional update is an update that● Is atomic
– Either fully applied or not applied at all– Update does not influence the running system
● Can be rolled back– A failed or incompatible update can be quickly discarded to
restore the previous system condition–➔ This definition is valid for all atomic distributions➔ Atomicity guaranteed by rebooting the system➔ Apart from that, implementations vary widely
Transactional Updates in the SUSE World
SUSE Tools
● Zypper for package management
● Snapper for snapshot creation
● Btrfs as default file system
Updates with snapper
current/
Backup/ (pre)
Backup/ (post)
Active system
1. Create “pre” snapshot2. Update the current system3. Create “post” snapshot
• Update is modifying the currently active file system
• Restarts services immediately
Updates with snapper
A Transactional Update is an update that● Is atomic
– Either fully applied, or not applied at all– Update does not influence the running system
● Can be rolled back– A failed or incompatible update can be quickly discarded to
restore the previous system condition
Updates with transactional-update
current/ (ro)
new/
1. Create new target snapshot2. Update system, make snapshot
read-only and set as default for next boot
● Current root file system is not modified
● No backup snapshots required, due to ro system
● Will not do anything if there are no updates
● Discard snapshot in case of errors
Active system
Updates with snapper
A Transactional Update is an update that● Is atomic
– Either fully applied, or not applied at all– Update does not influence the running system
● Can be rolled back– A failed or incompatible update can be quickly discarded to
restore the previous system condition
➢ Reboot will provide atomicity
Helper Applications: health-checker
● Add your own checker scripts to check for system consistency● Automatic rollback if checks fail on first boot● GRUB2: Select old kernel if system wasn’t able to complete initrd
current/ (ro)
new/
Comparison & Usage
Comparison to Other Read-Only Systems
● No separate tooling or infrastructure needed● Works with any standards-compliant RPM package● General-purpose tool: especially useful for servers and clusters● Fast snapshot switching● Sane /etc and /var handling● Only works with BTRFS root file systems
Live Demo
Cheat SheetTransactional Updates
List repositorieszypper lr -dzypper lr -d
Refresh repositorieszypper refzypper ref
Update installed packagestransactional-update uptransactional-update up
Perform a distribution updatetransactional-update duptransactional-update dup
Install package(s)transactional-update pkg in <name>transactional-update pkg in <name>
Update package(s)transactional-update pkg up <name>transactional-update pkg up <name>
Remove package(s)transactional-update pkg rm <name>transactional-update pkg rm <name>
List snapshotssnapper listsnapper list
Mark snapshots for removal by snappertransactional-update cleanuptransactional-update cleanup
View default subvolumebtrfs subvolume get-default /btrfs subvolume get-default /
Open shelltransactional-update shelltransactional-update shell
Request reboottransactional-update reboottransactional-update reboot
System rollbacktransactional-update rollback [number]transactional-update rollback [number]
A Deeper Look
Handling of Special Directories
Writable directories on an otherwise read-only system:●/var●/etc
Snapshots are only taken of the root file system by default!
/var handling/var handling
● /var is a special directory because it contains variable data● Has to have read-write permissions● Cannot be rolled back● A rollback would usually delete production data (e.g., databases or
Docker images)● Typically stored on a separate subvolume or partition● /var will not be mounted into the update snapshot (i.e. packages can
not modify it)● Special handling: directories will be created on next boot● Files can be created with systemd-tmpfiles; for more complex
initialization or migration tasks, use systemd services on first boot
/etc handling
● On read-only systems, /etc has to be writable● Mounted as an overlay file system● Overlays stored in /var/lib/overlays● Each snapshot has one corresponding /etc overlay
● Overlay will be mounted during the update● Before: Only one overlay for all snapshots
● /etc overlays are stacked● Changes accumulated before the next boot will be visible both in the
current and the next snapshot (if not changed in both overlays)● On rollback, only the layers up to that snapshot will be mounted
/etc handling
● To prevent the overlayfs stack from growing infinitely, the /etc state of the oldest available snapshot is synchronized into the new snapshot
● The /etc directory in a snapshot (/.snapshots/<num>/snapshot/etc) alone does not represent a consistent system state!
● Do not mess with the files in /var/lib/overlays (although, you can use them as a backup source if you want)
overlay /etc overlay defaults,upperdir=/sysroot/var/lib/overlay/18/etc,lowerdir=/sysroofor=/var/lib/overlay,x-system d.requires-m ounts-for=/sysroot/var,x-system d.requires-m ounts-for=/sysroot/var/lib/overlayoverlay /etc overlay defaults,upperdir=/sysroot/var/lib/overlay/18/etc,lowerdir=/sysroofor=/var/lib/overlay,x-system d.requires-m ounts-for=/sysroot/var,x-system d.requires-m ounts-for=/sysroot/var/lib/overlay
Other Directories
● /opt, /var/log and /boot/grub2 will be bind mounted into the update snapshot
● Everything else, including /srv, won’t!
➔ Packages have to follow the FHS and packaging guidelines
Mount Summary
/ (ro)
/
Active system
Visible duringupdate
/var /etclayer x
/etclayer x+1
/opt
/opt
Bind mounts:
/var/log
/var/log
/bogrubarc
/bogrubarc
/srv/home
...
Utilities
Helper Applications: rebootmgr
● transactional-update.timer triggers daily update including reboot● rebootmgr manages reboot (e.g. in maintenance windows or
synchronized via etcd)● SUSE CaaS Platform users: not used there; use Velum to trigger
cluster-wide reboots instead
Other
● Configuration file: /etc/transactional-update.conf● Snapper will clean up old snapshots● transactional-update is the only way to update a read-only system● Zypper integration is planned
Pitfalls
● Snapshots will be branched from the current system→ snapshots will not contains the previous snapshot’s contents if the system hasn’t been rebooted!
● When using transactional-update on a read-write system→ Don’t forget to reboot your system before making any changes to the root file system!
What’s Next?
Availability
SUSE Linux Enterprise Server Service Pack 1:● Technology Preview● Compatible with most
packages (~30 packages left out of 10,000)
● See documentation for a list of unsupported packages
Availability
Future Development
● Integration into SUSE Linux Enterprise Server● IMA / EVM support for system verification / integrity● Integrate transactional-update as zypper plugin● Fix remaining RPM packages with scripts modifying /var and /srv