Presented by

Date

Event

BKK16-312: Integrating and controlling embedded

devices in LAVA

Bill FletcherLinaro Field Engineering

Linaro Connect BKK16

Bill Fletcher

BKK16-312 March 9, 2016

v2.0

Overview

● Practical desk-top LAVA configuration● Embedded 32 and 64-bit targets● Power control● Boot test automation example for real-world devices● Starting point to leverage the awesomeness of LAVA

More about this session

● The slide deck has a lot of detail● I don’t propose to go line-by-line through

configuration files here● Kind of a meta-tutorial in the session● It’s a big invitation to have a go with LAVA and

physical devices

Invitations to interrupt

● Any LAVA folks feel free to pitch in on new dispatcher, supported versions, ...

● Any 96Boards folks feel free to pitch in on U-Boot status on Hikey and other boards

● Questions & comments welcome

All the configs & images

● All material is at: http://people.linaro.org/~bill.fletcher/

● LAVA configuration files● Sample drivers and scripts for power control● Boot images for Hikey and BB

Preamble - Brief Intro To LAVA

● The Linaro Automated Validation Architecture (LAVA wiki link)

● An automation system for deploying kernel,dtb and rootfs onto physical and virtual hardware for running tests.

● A collection of participating components● Very scalable

LAVA is ...

YAML test definition

Basic Elements of LAVALAVA Dispatcher

● Executes the test job● Contains target platform

configuration files● Locally intelligent and tries to 'do

the right thing'● Manages communication to the

target device● Collects and evaluates the job

results

LAVA Server● Handles job submission& scheduling● Handles instances of target platforms● Invokes the dispatcher● Consolidates the results into bundle

stream

Device configuration.conf

JSON job definition

Dashboard/Admin● Accepts jobs, shows device status,

job queues etc● Manages object creation and

permissions● Reports results

Target Device

Pre-parsed results

scheduling

single instance

YAML test definition

Focus for this SessionLAVA Dispatcher

● Executes the test job● Contains target platform

configuration files● Locally intelligent and tries to 'do

the right thing'● Manages communication to the

target device● Collects and evaluates the job

results

LAVA Server● Handles job submission& scheduling● Handles instances of target platforms● Invokes the dispatcher● Consolidates the results into bundle

stream

Device configuration.conf

JSON job definition

Dashboard/Admin● Accepts jobs, shows device status,

job queues etc● Manages object creation and

permissions● Reports results

Target Device

Pre-parsed results

scheduling

single instance

i.e. interfacing the dispatcher to the target

Start now - be aware that LAVA is evolving ...

● The pipeline will be the new model for the dispatcher code:

● Submitted jobs converted to a pipeline of discrete actions

● The entire pipeline is validated before the job starts.● The model integrates concepts like fail-early, error

identification, avoid defaults, fail and diagnose later

COMING SOON

● LAVA is a Debian package● Work with supported versions (more info here):

● apt-get install lava○ Everything you need for a single instance

● Installs latest packaged release● Find it under /usr/lib/python2.7

Getting Started - Installing

1. Finish configuring the installation2. Configure a virtual test device (KVM, qemu)3. Acquire a bootable test image4. Write a basic json boot test job definition5. Submit a test jobSee these virtual target tutorials:https://youtu.be/T8jFzXRrFh8https://youtu.be/0FlfRMxwC00http://people.linaro.org/~bill.fletcher/SFO15-TR8_Getting_Started_With_LAVA_supporting_material/

After Installing

Adding Embedded Devices in LAVA (i) Mainly about the hardware

Embedded Devices

● 2 worked examples - Beaglebone and Hikey● Both boot with U-Boot● Each has a console, network, boots Linux● Note: a bootloader device is a specific type of target

in LAVA i.e. client_type = bootloader

Embedded devices in LAVA

Requirements:1. An uninterrupted serial console connection2. A way to put the system in a known state

and reboot3. TFTP assumed

i.e. serial, network and power control ...

Serial Console and Network for TFTPYou need console and network connections:● ser2net package is a dependency of lava-dispatcher, so will be

installed automatically. In /etc/ser2net.conf add something like: 2000:telnet:0:/dev/tty.beaglebone:115200 8DATABITS NONE 1STOPBIT banner

● The dispatcher relies on TFTP downloads to all be made from /var/lib/lava/dispatcher/tmp. After installing tftp-hpa, the configuration file for tftpd-hpa needs to be modified to use the LAVA directory instead of the default

Beaglebone

Comparison vs previously stated requirements:An uninterrupted serial connection - BB(W) has FTDI serial via USB peripheral connector that also supplies power to the boardA way to put the system in a known state and reboot the board - Removing USB power kills the FTDI serial connection - Removing 5V power - the device switches to using the USB power- :(A bootloader with tftp - Very mature U-Boot with TFTP via onboard RJ45

Where there’s a will there’s a way ...

(hard-wiring the reset line to a jack for relay control)

Hikey

Comparison vs previously stated requirements:An uninterrupted serial connection - FTDI serial (a couple of options)A way to put the system in a known state and reboot the board - Power cycle 12V inputA bootloader with tftp - Support in U-Boot upstream- TFTP Support using KY88772 USB ethernet dongles

Rebooting the board: An Introduction to PDUs

● PDU = Power Distribution Unit● “A way to put the system in a known state and reboot the board”● Power cycle the board under software control● PDUs tend to be shared between targets (potential access conflicts)● pdudaemon is a framework that LAVA provides to manage multiple

accesses to PDUs● LAVA device config file: hard_reset_command =

PDU Daemon - elements

pduclient - command line utility to send a command as a TCP request to the daemone.g. pduclient --daemon=localhost --hostname=127.0.0.1 --port=4 --command=reboot

pdusocket listener server - TCP request handler. Puts a request into the request database.pdurunner - one runner process for each PDU - loops looking for jobs in the database for a given pdu.pdu drivers - class-based drivers to interact with PDUs and now some other measurement/control hardware, mainly over ethernet. Many PDUs have an interface via Web, SNMP, and Telnet.

Reboot requestsLAVA Dispatcher

PDUclient

listener server

PDU queue

SQL adapter

PDUrunner

PDUdriverPDUdriver

PDUdrivers

TCPrequest handler

PDUprocess

insert request

PDU queuePDU

queues

read next job

PDUprocessPDUprocesses

reboot Requests database

PDUdriverPDUdriverPDU hardware

reboot

Trivial Desktop PDU Example

● Arduino with a relay shield for low-voltage or reset line switching

● USB control of up to 4 targets

● Access managed via pdudaemon localcmdline driver class

{

"daemon": {

"hostname": "0.0.0.0",

"port": 16421,

"dbhost": "127.0.0.1",

"dbuser": "pdudaemon",

"dbpass": "pdudaemon",

"dbname": "lavapdu",

"retries": 5,

"logging_level": "INFO"

},

"pdus": {

"127.0.0.1": {

"driver": "localcmdline"

},

"192.168.10.3": {

"driver": "apc7952",

"telnetport": 5023

}

}

Configuring pdudaemon

● /etc/lavapdu/lavapdu.conf ->● Example

○ Local daemon○ 2 local pdus○ one networked APC (via telnet)

and one localcmdline

Adding Embedded Devices in LAVA (ii) Mainly about the configuration

● Inform the dispatcher about the devices● What are the sequences of commands

that the bootloaders should execute

/etc/lava-dispatcher/devices/beaglebone_001.conf

Basic device information in the device configuration file:

device_type = beaglebone (predefined type)

hostname = beaglebone_001 (arbitrary)

connection_command = telnet localhost 2000 (see ser2net)

hard_reset_command = pduclient --daemon=localhost --hostname=127.0.0.1 --port=4 --command=reboot

(see previous pdu_daemon slide)

...

Boot stanzas● Boot command stanzas are

predefined Boot Commands which are included in the device configuration

● ‘boot_cmds_ramdisk’ is the default boot stanza for bootloader clients

● This default defined in /python2.7/dist-packages/lava_dispatcher/device/bootloader.py

● The Boot stanza is executed as if typed at the u-boot command line

● Leverages target’s command line functionality

● {Filenames} supplied from the job file

boot_cmds_ramdisk =

setenv autoload no,

setenv label "'Hikey LAVA U-Boot Tutorial'",

setenv fdt_high "'0xffffffffffffffff'",

setenv kernel_addr_r "'0x00000000'",

setenv initrd_addr_r "'0x02000000'",

setenv fdt_addr_r "'0x01f00000'",

setenv loadkernel "'tftp ${kernel_addr_r} {KERNEL}'",

setenv loadinitrd "'tftp ${initrd_addr_r} {RAMDISK};

setenv initrd_size ${filesize}'",

setenv loadfdt "'tftp ${fdt_addr_r} {DTB}'",

setenv bootargs "'console=ttyAMA3,115200 debug root=/dev/sda0 rootwait ip=none'",

setenv bootcmd "'usb start; setenv ipaddr 192.168.1.210; setenv serverip {SERVER_IP}; run loadkernel; run loadinitrd; run loadfdt; booti ${kernel_addr_r} ${initrd_addr_r} ${fdt_addr_r}'",

boot

/etc/lava-dispatcher/devices/beaglebone_001.confdevice_type = beaglebone (predefined type)

hostname = beaglebone_001 (arbitrary)

connection_command = telnet localhost 2000 (see ser2net)

hard_reset_command = pduclient --daemon=localhost --hostname=127.0.0.1 --port=4 --command=reboot

(see previous pdu_daemon slide)

boot_cmds_ramdisk = (boot stanza - see previous slide)

setenv autoload no,

setenv label "'Beaglebone LAVA U-Boot Tutorial'",

setenv fdt_high "'0xffffffff'",

setenv kernel_addr_r "'{KERNEL_ADDR}'",

setenv initrd_addr_r "'{RAMDISK_ADDR}'",

setenv fdt_addr_r "'{DTB_ADDR}'",

setenv loadkernel "'tftp ${kernel_addr_r} {KERNEL}'",

setenv loadinitrd "'tftp ${initrd_addr_r} {RAMDISK}; setenv initrd_size ${filesize}'",

setenv loadfdt "'tftp ${fdt_addr_r} {DTB}'",

setenv bootargs "'console=ttyO0,115200n8 root=/dev/ram0 ip=none'",

setenv bootcmd "'setenv ipaddr 192.168.1.200; setenv serverip {SERVER_IP}; run loadkernel; run loadinitrd; run loadfdt; {BOOTX}'",

boot

(DEVICE CONFIGURATION FILE)

/etc/lava-dispatcher/devices/hikey_002.confdevice_type = hikey

hostname = hikey_002

connection_command = telnet localhost 2099

hard_reset_command = pduclient --daemon=localhost --hostname=127.0.0.1 --port=3 --command=reboot

boot_cmds_ramdisk =

setenv autoload no,

setenv label "'Hikey LAVA U-Boot Tutorial'",

setenv fdt_high "'0xffffffffffffffff'",

setenv kernel_addr_r "'0x00000000'",

setenv initrd_addr_r "'0x02000000'",

setenv fdt_addr_r "'0x01f00000'",

setenv loadkernel "'tftp ${kernel_addr_r} {KERNEL}'",

setenv loadinitrd "'tftp ${initrd_addr_r} {RAMDISK}; setenv initrd_size ${filesize}'",

setenv loadfdt "'tftp ${fdt_addr_r} {DTB}'",

setenv bootargs "'console=ttyAMA3,115200 debug root=/dev/sda0 rootwait ip=none'",

setenv bootcmd "'usb start; setenv ipaddr 192.168.1.210; setenv serverip {SERVER_IP}; run loadkernel; run loadinitrd; run loadfdt; booti ${kernel_addr_r} ${initrd_addr_r} ${fdt_addr_r}'",

boot

(DEVICE CONFIGURATION FILE)

Adding Embedded Devices in LAVA (iii) Putting it all together

Submitting the job file - 2 ways1.Web Interface 2.lava-tool

● Rather than paste our json job files into the web interface, we can submit test jobs to LAVA via the command line using lava-tool.

● Generate an authentication token via the web interface. lava-tool auth-add http://bill@localhost

● (note that you need to generate the authentication token logged in to the local web interface as yourself)

● Submit: lava-tool submit-job http://bill@localhost /home/bill/development/lava/boot_bb.json

json job file - Beaglebone{

"actions": [

{

"command": "deploy_linaro_kernel",

"parameters":

{

"dtb": "file:/home/bill/development/lava/images/am335x-bone.dtb",

"kernel": "file:/home/bill/development/lava/images/uImage-bone",

"ramdisk": "file:/home/bill/development/lava/images/rootfs-bone.cpio.gz"

}

},

{

"command": "boot_linaro_image"

}],

"job_name": "boot-bb",

"target": "beaglebone_001",

"timeout": 18000

}

json job file - Hikey{

"actions": [

{

"command": "deploy_linaro_kernel",

"parameters":

{

"dtb": "file:/home/bill/development/lava/images/hi6220-hikey-0202B.dtb",

"kernel": "file:/home/bill/development/lava/images/Image-0202A",

"ramdisk": "file:/home/bill/development/lava/images/rootfs-aarch64-2up-2000000.uboot",

"login_prompt": "aarch64_test login:",

"username": "root"

}

},

{

"command": "boot_linaro_image"

}],

"job_name": "boot-hi-318-0202",

"target": "hikey_002",

"timeout": 18000

}

Adding Embedded Devices in LAVA (iv) A quick demo

● All the referenced configuration files are available at http://people.linaro.org/~bill.fletcher/bkk16_lava

● Support via mail and IRC. Details at https://validation.linaro.org/static/docs/support.html

● Have a go!

Wrap Up

Thanks for listening

Any questions?

Screenshot - PDU reboot

Screenshot - boot command (BB)

Screenshot - boot command (Hikey)

lava stores images in /var/lib/lava/dispatcher/tmppdudaemon logs stored in /var/log/lavapdu*

Under the hood

Dispatcher Device conf files

Inherent defaults for client_type = bootloader configuration in /usr/lib/python2.7/dist-packages/lava_dispatcher/default-config/lava-dispatcher/device-defaults.conf

Default configurations exists in: /usr/lib/python2.7/dist-packages/lava_dispatcher/default-config/lava-dispatcher/device-types/device.conf

Local device types can be defined in /etc/lava-dispatcher/device-types

Specific device configs can be created in /etc/lava-dispatcher/devices