View
2.730
Download
0
Category
Preview:
Citation preview
Ottimizzare i tempi di boot di Linux
Agenda
Overview
Case study: Raspberry Pi
Kernel optimizations
rootfs optimizations
Q/A
Why a faster boot?
Consumer electronics products require very fast boot times (digital camera, TV, mobile phone, etc.)
Fast recovery upon system failures (crashes)
You can show to your friends your new awesome board booting in a couple of seconds ;-)
Overview of boot phases
Firmware (boot-loader)Hardware probing
Hardware initialization
Kernel load and decompression
Kernel executionCore init (start_kernel)
Driver init (initcalls)
User-space init/sbin/init
RC scripts
Application start (first user impression)
Generic concepts
Identify boot time functionalities
Measure boot time of each functionality
Remove unnecessary functionalities
Optimize required functionsDefer loading of less-important features (modularization - LKM)
Re-order initialization / parallelizationAsynchronous initialization
Generic concepts
Identify boot time functionalities
Measure boot time of each functionality
Remove unnecessary functionalities
Optimize required functionsDefer loading of less-important features (modularization - LKM)
Re-order initialization / parallelizationAsynchronous initialization
Premature optimization is the root of all evil. -Donald Knuth
Case study
Raspberry Pi
Boot time functionalityReponsive ssh login
Tools and techniques used to improve boot time
Focusing on cold-boot optimizations
Boot steps
RPi boot steps:1st-stage boot loader from SoC BCM2835 ROM
bootcode.bin: 2nd-stage boot loader from external SD card, starts the GPU
start.elf: GPU firmware, starts the CPU
kernel.img: Linux
rootfs: external SD card
application: ssh (dropbear)
Measure boot-time functionality (kernel)
CONFIG_PRINTK_TIMEConfigure it in Kernel hacking section
Adds timing informations to kernel messages (dmesg)
# dmesg...[ 0.022030] initcall bcm_mbox_init+0x0/0x38 returned 0 after 0 usecs[ 0.022054] calling bcm_power_init+0x0/0xa4 @ 1[ 0.022065] bcm_power: Broadcom power driver[ 0.022080] bcm_power_open() -> 0[ 0.022090] bcm_power_request(0, 8)[ 0.522766] bcm_mailbox_read -> 00000080, 0[ 0.522783] bcm_power_request -> 0[ 0.522812] initcall bcm_power_init+0x0/0xa4 returned 0 after 488281 usecs
Kernel initcall tracer
Introduced in Linux 2.6.28
Add initcall_debug to the kernel boot options
Allows to record the timings of each initcall in dmesg
# dmesg | grep " initcall " | sed "s/ */ /g" | sort -n -t' ' -k8 | tail -5[ 0.701759] initcall bcm2708_fb_init+0x0/0xc returned 0 after 32518 usecs[ 0.794306] initcall pty_init+0x0/0x3e0 returned 0 after 90313 usecs[ 0.660366] initcall init_kprobes+0x0/0x108 returned 0 after 94523 usecs[ 1.224203] initcall dwc_otg_driver_init+0x0/0xb8 returned 0 after 402667 usecs[ 0.518454] initcall bcm_power_init+0x0/0xac returned 0 after 488281 usecs
Kernel initcall tracer (example)
dmesg > dmesg.log
cat dmesg.log | perl scripts/bootgraph.pl > /boot/dmesg.svg
Raspbian
http://www.raspbian.org
General-purpose distro based on Debian
Optimized for the Raspberry Picompilation settings adjusted to produce hard-float code
2013-05-25-wheezy-raspbian.img
Raspbian: boot time
# dmesg...[ 3.107163] smsc95xx 1-1.1:1.0: eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:a9:d6:24[ 5.664780] EXT4-fs (mmcblk0p2): recovery complete[ 5.677326] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)[ 5.689427] VFS: Mounted root (ext4 filesystem) on device 179:2.[ 5.700523] devtmpfs: mounted[ 5.706028] Freeing init memory: 128K[ 6.257687] init start[ 7.337703] udevd[154]: starting version 175[ 8.511002] Registered led device: led0[ 16.041293] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)[ 16.522927] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)[ 25.553157] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1[ 28.672178] Adding 102396k swap on /var/swap. Priority:-1 extents:2 across:507900k SS[ 30.436046] init done
- Shell after ~6.3sec
- SSH after ~30.5sec
sysv-rc-conf
Run-level configuration for SysV like init scriptsdisable npt
disable plymouth
disable rsync
disable x11-common and lightdm
disable alsa-utils
disable swap (dphys-swapfile)
disable checkfs
disable ntp
Static IP vs DHCP
Assign a static IP to the board (save the time to get an IP via DHCP)
Disable CONFIG_IP_PNP in the kernel .configUsed to mount rootfs via NFS
Raspbian: boot time after simple optimizations
# dmesg...[ 3.132731] smsc95xx 1-1.1:1.0: eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:a9:d6:24[ 5.730131] EXT4-fs (mmcblk0p2): recovery complete[ 5.742783] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)[ 5.754881] VFS: Mounted root (ext4 filesystem) on device 179:2.[ 5.765958] devtmpfs: mounted[ 5.771460] Freeing init memory: 128K[ 6.310549] init start[ 7.389321] udevd[154]: starting version 175[ 8.536932] calling leds_init+0x0/0x60 [led_class] @ 229[ 8.537065] initcall leds_init+0x0/0x60 [led_class] returned 0 after 82 usecs[ 8.543802] calling gpio_led_driver_init+0x0/0xc [leds_gpio] @ 229[ 8.544067] Registered led device: led0[ 8.545242] initcall gpio_led_driver_init+0x0/0xc [leds_gpio] returned 0 after 1344 usecs[ 14.724169] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)[ 15.202445] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)[ 24.493346] smsc95xx 1-1.1:1.0: eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1[ 24.991030] init done
About 25 sec to login via ssh, instead of 30 sec (16% faster)
Build a minimal distro from scratch: requirements
build toolchain (gcc, glibc, binutils)git://github.com/raspberrypi/tools.git
HINT: use hard-float toolchain (arm-bcm2708hardfp-linux-gnueabi)
Linux (kernel)git://github.com/raspberrypi/linux.git
busybox (rootfs)git://busybox.net/busybox.git
dropbear (ssh server)https://matt.ucc.asn.au/dropbear/
Kernel initcalls (before optimization)
# dmesg | grep " initcall " | sed "s/ */ /g" | sort -n -t' ' -k8...[ 0.810927] initcall iscsi_transport_init+0x0/0x154 returned 0 after 446 usecs[ 1.225990] initcall bcm2835_thermal_driver_init+0x0/0xc returned 0 after 451 usecs[ 1.232472] initcall pm_qos_power_init+0x0/0xac returned 0 after 713 usecs[ 1.230686] initcall sysctl_ipv4_init+0x0/0x98 returned 0 after 755 usecs[ 0.810406] initcall vchiq_init+0x0/0x1dc returned 0 after 1565 usecs[ 1.228797] initcall sdhci_drv_init+0x0/0xc returned 0 after 1829 usecs[ 0.560116] initcall inet_init+0x0/0x260 returned 0 after 2739 usecs[ 0.808747] initcall loop_init+0x0/0x11c returned 0 after 4852 usecs[ 0.803722] initcall brd_init+0x0/0x1c0 returned 0 after 8230 usecs[ 0.540586] initcall genhd_device_init+0x0/0x84 returned 0 after 9765 usecs[ 0.556774] initcall chr_dev_init+0x0/0xd8 returned 0 after 12234 usecs[ 0.538982] initcall param_sysfs_init+0x0/0x1d4 returned 0 after 19531 usecs[ 0.701759] initcall bcm2708_fb_init+0x0/0xc returned 0 after 32518 usecs[ 0.794306] initcall pty_init+0x0/0x3e0 returned 0 after 90313 usecs[ 0.660366] initcall init_kprobes+0x0/0x108 returned 0 after 94523 usecs[ 1.224203] initcall dwc_otg_driver_init+0x0/0xb8 returned 0 after 402667 usecs[ 0.518454] initcall bcm_power_init+0x0/0xac returned 0 after 488281 usecs
Kernel optimizations
(initcalls)
disable kprobes (CONFIG_KPROBES): ~95ms
reduce the number of PTYs: ~90msCONFIG_LEGACY_PTY_COUNT=256 => CONFIG_LEGACY_PTY_COUNT=1
disable framebuffer: ~33ms
remove loop device: ~5ms
Kernel initcalls (after optimization)
# dmesg | grep " initcall " | sed "s/ */ /g" | sort -n -t' ' -k8...[ 0.570870] initcall tun_init+0x0/0x8c returned 0 after 259 usecs[ 0.568035] initcall des_generic_mod_init+0x0/0x10 returned 0 after 341 usecs[ 0.975476] initcall bcm2835_cpufreq_module_init+0x0/0xc returned 0 after 358 usecs[ 0.569678] initcall pty_init+0x0/0x194 returned 0 after 375 usecs[ 0.561542] initcall vc_mem_init+0x0/0x1b4 returned 0 after 393 usecs[ 0.974729] initcall bcm2835_thermal_driver_init+0x0/0xc returned 0 after 393 usecs[ 1.018751] initcall deferred_probe_initcall+0x0/0x6c returned 0 after 395 usecs[ 0.561078] initcall bcm2708_gpio_init+0x0/0xc returned 0 after 405 usecs[ 1.018028] initcall pm_qos_power_init+0x0/0x60 returned 0 after 506 usecs[ 0.559402] initcall inet_init+0x0/0x260 returned 0 after 1540 usecs[ 1.021006] initcall net_secret_init+0x0/0x1c returned 0 after 2145 usecs[ 1.024343] initcall initialize_hashrnd+0x0/0x1c returned 0 after 3052 usecs[ 0.557293] initcall chr_dev_init+0x0/0xd8 returned 0 after 10978 usecs[ 0.541305] initcall param_sysfs_init+0x0/0x19c returned 0 after 19531 usecs[ 1.016084] initcall sdhci_drv_init+0x0/0xc returned 0 after 39250 usecs[ 0.973739] initcall dwc_otg_driver_init+0x0/0xc4 returned 0 after 392536 usecs[ 0.522812] initcall bcm_power_init+0x0/0xa4 returned 0 after 488281 usecs
Kernel optimizations
(other optimizations)
preset loops_per_jiffyAt each boot the kernel calibrates a delay loop (used later by the udelay() function)
1 jiffy = time between 2 timer interrupts
CONFIG_HZ=100 => 250ms!!!
loop
disable console outputremove console=xxx and add quiet to the kernel boot parameters
use LZO kernel decompression (CONFIG_KERNEL_LZO)LZO is a compression algorithm that is much faster than gzip, at the cost of a slightly degrade compression ratio (+10%)
reduce kernel size...A smaller kernel is faster to load, less code also means smaller working set (good for caches)
Kernel image (before => after)
rootfs
Generated using busybox and dropbear
Hints about busybox (reduce fork()s):CONFIG_FEATURE_SH_STANDALONE: use applets instead of fork/exec/wait external binaries
CONFIG_FEATURE_SH_NOFORK: call _main() directly without spawning another task
Build everything with -Os
Use mklibs to strip system libraries (rootfs is mounted to /mnt)mklibs -v -D -d lib2 -L /mnt/lib --ldlib lib/ld-linux.so.3 --target=arm-bcm2708-linux-gnueabi /mnt/bin/* /mnt/sbin/* /mnt/usr/sbin/* /mnt/usr/bin/*; rm -rf lib; mv lib2 lib
After all these stops total rootfs size is ~3.2MB
filesystem optimizations
Split the filesystem into read-only portion and read/write portionRead-only filesystem mounts faster
Use Squashfs + tmpfs (or aufs)
root filesystem: boot time
Kernel image (before => after)
1.5GB
3.2MB
Demo: custom kernel+rootfs boot time
...[ 1.144025] Freeing init memory: 92K[ 1.228550] init start[ 1.249067] waiting eth0 to come up[ 1.345882] usb 1-1: new high-speed USB device number 2 using dwc_otg[ 1.346059] Indeed it is in host mode hprt0 = 00001101[ 1.556942] hub 1-1:1.0: USB hub found[ 1.557074] hub 1-1:1.0: 3 ports detected[ 1.836029] usb 1-1.1: new high-speed USB device number 3 using dwc_otg[ 1.940066] smsc95xx v1.0.4[ 1.951858] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-bcm2708_usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:a9:d6:24[ 2.086695] init done[ 3.406004] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
- Shell after ~1.2 sec
- SSH after ~3.4 sec
References
https://github.com/arighi/rpi-mini-distro
LPC: Booting Linux in 5 Seconds - Arjan van de Ven's talk: http://lwn.net/Articles/299483/
elinux wiki - Tim's fastboot tools:
http://elinux.org/Tims_Fastboot_Tools
elinux wiki- RPi Kernel Compilation: http://elinux.org/RPi_Kernel_Compilation
Update on boot time reduction techniques - Michael Opdenacker
Linux: http://www.kernel.org
Busybox: http://busybox.net
Dropbear: https://matt.ucc.asn.au/dropbear/dropbear.html
Raspberry Pi: http://www.raspberrypi.org
http://it.emcelettronica.com/embedded-gnulinux-partendo-da-zero-test-sulla-raspberry-pi
Q/A
You're very welcome!
Twitter@arighi
#bem2013
Andrea Righi - andrea@betterlinux.com
Recommended