QA with Canonical - Ubuntuodm.ubuntu.com/uhs/2011/OEM QA with Canonical.pdf · 2011-12-12 · 3...

QA with CanonicalDecember 8th 2011

Ethan Chang

QA Engineer, PES

2 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

Agenda

● Overview

● Process Flow

● Tools Overview

3 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

QA Team Overview

Ubuntu QA

● Driven by a core of QA Professionals in Canonical

● Community supported with a network of thousands

● Beta user testing begins as soon as the repos are created

PES QA

● Designed around more formal SQA common practices

● Globally distributed with members in USA, Spain, France, Taiwan, and

Mainland China to maximize “up time”

Tools

● All tools used by Canonical QA and Community QA are open source

and all test cases are public

● Automated testing combined with manual repeatable tests and Beta

usability testing

4 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

Quality Advantages

Four QA passes

● Upstream QA

● Ubuntu QA

● PES QA

● Hardware Certification

Many “Eyes” on Ubuntu

● Over 100 bugs a day submitted against Ubuntu

● Over 1300 unique Ubuntu mailing list posts per day

● Over 300,000 bugs logged to date

● Over 100,000 bugs fixed released

5 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

Actual Ubuntu Testing Workflow

Spec Dev QA IntegrateBetaTest QA

Upstream

No

Yes

Ubuntu

Yes

No

CustomizeHardware

Enable ReleaseRelease Release

No

PES QA

Bug Fixes

Bug Fixes

6 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

PES QA Workflow

RequirementGartering

QA Test PlanTest CaseCreation

Test CaseExecution

Final TestReport

Review

Project WikiCertification

Bug Stats

Bug CreationBug Triage

7 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

PES Documentation

PES QA has produced a series of documents that provide more detailed

information on our process

● Test PlanningThe test plan will be used to verify and ensure the system meets its design specifications and other requirements

● Test ReportingThe test report will record all the test results and provide online review and office CSV file

● Test CasesPES QA uses Checkbox-editor to create test cases and save test cases on Launchpad via the Bzr tool

● Testing SOPAccording to Customer's requirement, we will provide SOP document for ODM/OEM QA to perform test and Canonical Tools

8 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

Customer QA

Customer QA

Customer QA

TL/PM

TL/PM

NewIncomplete Invalid

TL/PM

CQA

Confirmed

Fix Released

Fix Committed

Dev

In Progress

Won't Fix

Triaged

CQACQA

TL/PM

TL/PM

CQA

CQA

Tag: CQA verified Tag: Verified

CQA

Incomplete

Invalid

Bug Workflow

9 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

Tools Overview

Design Organize

Checkbox Editor GTK application for creating, editing and managing test cases for Checkbox framework. Developed and maintained by PES QA

Checkbox Testing framework for executing manual or automated tests and reporting their results. PES QA maintains its own branch

Lazybones installerApplet designed to make installation of OEM test frameworks easierCombines public and private PPAs and their dependent packagesDeveloped and maintained by PES QA

Install Execute

10 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

Tools Overview

Log Review Analyse

OEM ReportsCollection of advanced reports designed from customer requirementsGenerated automatically from OEM test frameworksDeveloped and maintained by PES QA

Bug Time TrackerApplication which collects defect information from LaunchpadGenerates trending reports based on collected dataDeveloped and maintained by PES QA

Report

OEM BugGTK applet for reporting defects to LaunchpadIncludes automatically collected data on the system under test Developed and maintained by PES QA

11 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

Checkbox Editor

Developed by PES QA GTK front endCreating, Edit, Organize test cases

Enforce standard templatesProduce customer facing documents BZR integration

Design

12 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

Checkbox Editor

All generated files are tracked by BZR Easily export test run configurations Tracks test dependencies and statistics

Executes tests, provides debug info More information on the QA WikiOrganize

13 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

PES QA Case Structure

Req Based Device Spec Customer

0

200

400

600

800

1000

1200

# of cases

Creation cases based on customer requirements

Creation of device specific cases

Integration of cases provided by customer

14 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

Lazybones

Installs all required packages Assistant interface guides the user Configures public and private PPAs Auto accepts PPA invitations Package list can be configured

Install

15 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

Checkbox

● Guided test runs● Automated test support● Manual test support● Pluggable Framework● Config file version control● Launchpad Lib support● GTK, CLI, and Urwid interface● HTML and Excel reports● Localized cases in Mandarin

Execute

16 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

OEM Bug

Log

● Uses apport.hookutils for automatic collection of system information● Uses assistant interface to guide the user through a decision tree● Developer feedback determines additional information to be collected● Automatically populates hardware information● Integrated into Checkbox and Stand-alone application

17 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

OEM Reports

Automatically Generated by Checkbox on the SUT

Tabbed report provides system specific information, test statistics, individual test case result detail, and logs Report

18 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

OEM Reports

Easily converted to common formats such as XLS, ODT, and CSV

Formatted to match the reporting requirements of our largest OEM partnersReview

19 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

Bug Time Tracker

Analyse

● Collects the history of every bug for a given project ● Displays bug statistics in table form● Allows the user to filter data on the same criteria● Plots filtered data● Figures can be saved to disk● Data can be exported to csv

20 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011

The Future

Enhance

● Checkbox and Checkbox Editor merging to produce a single more flexible tool for managing test cases, runs, and results.

● Server based automated trend reporting for all projects using Launchpad Lib, and Certification.Canonical.com APIs

● Remote test execution with the integration of Jenkins into Checkbox

● Consolidated test result reporting with Platform QA for better integration of test efforts

● Full tool and test case support across all relevant architectures (x86, x86-64, ARM)

Thank you

Questions ?