LCE13: Test and Validation Summit: Evolution of Testing in Linaro (II)

Linaro Test and Validation Summit

Linaro Engineering Teams

LCE13 - Dublin, July 2013

How Do We Better Test our Engineering

PART 1: Linaro Platform

Overview

LAVACitius, Altius, Fortius"Faster, Higher, Stronger"

Builds & CIBuild Your Code When You are Not Looking

QA ServicesCover all bases

PART 2: Linaro Engineering

Kernel Developers& Maintainers

Landing Teams Linaro GroupsLEG / LNG

( Do they use CI ? Manual or Automated Testing? )* How do they validate/verify their output? * What/how do they develop?

& easier to use

Agenda (Tuesday, 9am-1pm

Time Topic Speaker

9:00 Introduction (Dev, Test, Loop) Alan Bennett

9:15 Overview of the CI loop (25 min) Fathi Boudra

9:40 QA services (20 min) Milosz Wasilewski

10:00 Recent LAVA updates (45 min) Antonio Terceiro

10:45 BREAK

11:00 LNG Mike Holmes

11:30 Landing Teams Scott Bambrough

12:00 KWG PM Kevin Hilman

12:30 LEG Grant Likely

Plat

form

Upd

ates

tech

nica

l det

ails

Preface

Why is the Quality of Linaro Engineering so important?

Preface

Why is the Quality of Linaro Engineering so important?

● applying continuous quality control

● frequent integration of small pieces of software

● rapid feedback

● Extreme programming (XP)○ minimize integration

problems

● Shared code repositories● daily commits● automated build systems● extensive unit tests● testing in cloned

production environments

Highlights of Continuous Integration

Preface

Continuous Integration

CI Loop

Chang

es M

ade Automated Build

Test Report /

FeedbackTes

t

Source Control System

Build

Testing

Development

Linaro Test and Validation Summit

Fathi BoudraBuilds and Baselines

LCE13 - Dublin, July 2013

How Do We Better Test our Engineering

● CI Present○ Anatomy of CI loop

● CI Future○ What is on the CI roadmap

Overview

● Get the source○ Source code is under SCM

■ Git (git.linaro.org)■ Bazaar (bazaar.launchpad.net)

● Build the code○ Use a build system

■ Jenkins (ci.linaro.org and android-build.linaro.org)

■ LAVA (yes, LAVA can be used!!!)● Publish the build results

○ Build artifacts are available (snapshots.linaro.org)

Anatomy of CI loop

● Submit the results for testing○ LAVA (validation.linaro.org)

● Get the tests results○ E-mail notifications with filters (validation.linaro.

org/lava-server/dashboard/filters)○ LAVA dashboard (validation.linaro.org/lava-

server/dashboard)

Anatomy of CI loop

● Different type of jobs○ Kernel CI○ Engineering builds○ Components

● Build triggers○ manual, periodically, URL trigger, post-commit

● Do the build○ shell script(s)

■ can be maintained under SCM (linux-preempt-rt)○ Groovy script(s)

● Publish○ to snapshots.linaro.org○ to package repositories (PPA, other)

Build jobs in depth

● Submit to LAVA○ Generate a LAVA job file (json)○ test definitions are pulled from SCM (git.linaro.

org/gitweb?p=qa/test-definitions.git)● Misc

○ Jenkins can run unit tests (e.g qemu-ltp job)■ junit■ xunit

○ CI helpers■ post-build-lava■ post-build-ppa■ Linaro CI build tools

Build jobs in detail

● LAVA CI Runtime○ LAVA as a build system

● LAVA Publishing API○ LAVA ability to publish artifacts on remote host

● Build time optimization○ persistent slaves○ mirrors and caching

● Better documentation

CI Future

Any Questions?

Q&A

Linaro Test and Validation Summit

Milosz WasilewskiQA Services

LCE13 - Dublin, July 2013

How Do We Better Test our Engineering

QA ServicesTasks:

● manual testing

● dashboard monitoring

● reporting

● porting tests to LAVA

Manual TestingCurrent approach:

● test results are not very detailed

● no connection between test case description and result sheet

● results stored in google spreadsheet

● bug linking done manually (makes it hard to extract the list of 'known issues')

Future:

● store test cases in some better suited place than wiki

● preserve test case change history

● store manual test results along automatic ones (in LAVA)

● have ability to link bugs from various tracking systems to failed cases (in LAVA)

● generate reports easily (known issues, fixed problems, etc.)○ might be done using LAVA if there is an easy way to extract testing

results (for example REST API)

Manual Testing

● Monitoring dashboard ○ adding bugs○ debugging failed runs

● Creating custom dashboards○ Dashboard from filter○ No need to edit python code to create/edit dashboard○ Private/public dashboards○ Dashboard email notification (falls in the concept of filter-as-dashboard

approach)

Dashboards

● Use only binaries that were already automatically tested

● Don't repeat automated tests in manual run (we have to be confident that automated results are reliable)

Release workflow

LAVA: Faster, Higher, Stronger (& easier to use)

Antonio TerceiroLAVA

LCE13 - Dublin, July 2013

Test and Validation Summit

● Improvements● New testing capabilities● Engineering Progress Overview● What are we missing?

○ Open Discussion○ We want to hear from you

Overview

● ~90 ARM devices● ~300 ARM CPUs● ~150 jobs submitted per day● ~99% reliability

Context (0): the size of LAVA, today

● LAVA started as an in-house solution● Open source since day 1● Other organizations (incl. Linaro members)

interested in running their own LAVA lab

We need to go from an in-house service to a solid product

Context (1)

● No bootloader testing● Tests only involve single devices

We need to provide features to support new demands in test and validation

Context (2)

Improvements

● Queue size monitored with munin● Nagios monitoring all sorts of things (e.g.

temperature on Calxeda highbank nodes)● Health check failures

Monitoring

Easing LAVA installation

● Effort on proper upstream packaging so that packages for any (reasonable) OS can be easily made

● WIP on Debian and Fedora packaging

$ apt-get install lava-server$ yum install lava-server

Packaging enhancements

Easing LAVA learning

● Documentation is○ scattered○ outdated○ confusing

Documentation overhaul is in the LAVA roadmap.

Documentation overhaul

Easing LAVA usage

ATM a lava-test-shell job requires● 1 JSON job file● 1 YAML test definition file● + the test code itself

$ sudo apt-get install lava-tool$ lava script submit mytestscript.sh$ lava job list

LAVA test suite helper tool

Getting more out of LAVA data

More information out of LAVA data

● Improvements in test results visualization in the LAVA dashboard

LAVA is too hard to develop

● Too many separate components○ Also a mess for bug/project management

● Requires almost a full deployment for development

● Consolidated client components (3 to 1)● Will consolidate server components (3+ to 1)

Developer-friendliness

New testing capabilities

● LAVA Multi-purpose Probe● 1 base design, 5 boards now● USB serial connection(s) to the host● management of other connections to/from

devices under test

LMP

● prototype sets manufactured and under test● Use cases: ethernet hotplug, SATA hotplug,

HDMI hotplug and EDID faking, USB OTG testing, USB mux (sort of), lsgpio, audio hotplug, SD-Mux for bootloader testing

LMP (2)

LMP (3) - how it works (e.g. SD-MUX)

DUT

SDC1

Host

LMP

USB serialUSB MSD

Multi-node testing (1)

● Schedule jobs across multiple target devices○ Client-server, peer-to-peer and other scenarios

● Combine multiple results into a single result● LAVA will provide a generic interface, test

writers can program any tests they need.○ (special hardware setups possible but need to be

handled case-by-case)

Other sessions:● LAVA multi-node testing on Thursday● LNG multi-node use-cases on Friday

Multi-node testing (2)

● Logistics challenge!● We might end up needing 20 of every device

type in the lab● Need to manage the needed growth in the

lab in a sensible way

Other projects

● Lightweight interface for kernel developers● Boot from test UEFI on all Versatile Express

boards● Support for new member boards

Overview of Engineering Progress

In Progress

● LAVA LMP● Multi-node testing● Helper tool● Test result visualization

improvements● Lightweigth interface for

kernel devs● UEFI on V. Express● Support for new member

boards

In Progress X Planned

Planned (for soon)

● Server components consolidation

● QA improvements● Doc overhaul

Open Discussion

● What is your experience getting started with LAVA?

● What would have made your experience easier?

● Any suggestions to the LAVA team? Let us know!

● Feedback about the image reports revamp?

Seed Questions