Upload
linuxfb
View
131
Download
0
Tags:
Embed Size (px)
Citation preview
Quality Assurance at Novell for Linux Products
Oliver [email protected] OPS Quality Assurance
© Novell Inc. All rights reserved
2
Agenda
• How Open Source QA is different
• Aspects of OpenSource QA at Novell– Test Approach
– Test Environments
– Test Automation
– Maintenance Testing
• openSUSE & QA– future plans
© Novell Inc. All rights reserved
3
Software Testing as in the Books
Unit Testing
Integration Testing
System Testing
Component Testing
Stress Testing
Regression Testing
Acceptance Testing
Code
Product
© Novell Inc. All rights reserved
4
Multi Phase Test Approach
UNIT TEST
Autobuild
-Package dependencies-Correct libraries- Integrated tests and sanity checks
COMPONENT TEST
TIDB/Cerberus Harness
- Functional tests for all packages- Test Cases from PDB- Priority Based (CORE->ADDON)
FEATURE TEST
Code
Package
Feature Doc TIDB
- Manual test of all features includedin the feature document
SYSTEM TEST
Installed Systems Cerberus
- Stress and reliability tests- Industry test suites- Certification tests
REGRESSIONTESTS
TIDB/Cerberus
- Regression of all automated tests- Validation of fixed bugs- Regression of failed feature tests
automated manual/interactive
© Novell Inc. All rights reserved
5
Multi Phase Test Approach II
Beta 1
Beta 2
Beta 3
Beta 4
RC 1
RC 2
Unit Test ComponentTest
Packages
ComponentTest
SpecialFocus:
- Kernel- Network- SysMgmt- GUI
SystemTest
Stress - Reliability - Stability - Scalability
System test
Cert Tests
Feature Test
RegressionTest
Test Environment
© Novell Inc. All rights reserved
7
Overview of the Test Environment
PDB
Feature Document
Autobuild
TIDB /TESTLINK
EngineerTest Machines
KOTD(kernel of the day)test machines
i386 x86-64 IA64 s390PPC
i386 x86-64
QA LABNetwork
QA Kernel Laband ISV Test(Oracle, Veritas,...)
Reference Hosts
SLES9 SLES8 NLD SLOX
Test Automation
© Novell Inc. All rights reserved
9
Automation Architecture
CTCS
Apache testscript
Samba testscript
Bind testscript
Mount testscript
... testscript
Startssl
Httpd Smbd
Nmbd
Smbclient
Smbmount
named Mount /dev/nnn
mount -t nfs
App 1
App 2
App 3
Start Results
Application Layer
Test Control Files
"PASSED""FAILED"
start check and update
TXT, HTML, CSV, SQL, XML
create reports
Create reports
SQL db
...
Maintenance Update Testing
© Novell Inc. All rights reserved
11
MainCoord initiates SWAMPincl. Prio, etc.Patch is build and pkg. available
QA checks onhttp://w2d.suse.de/abuildstat/patch-statusneeded arch and products
MainCoord initiates SWAMPincl. Prio, etc.Patch is build and pkg. available
SLES 8i386
SLES 8s390
SLES 9x86_64
OESi386
SLES N<ARCH>
...
Update Test
Bug Fix Test
Feature Test
Update Test
Bug Fix Test
Feature Test
Update Test
Bug Fix Test
Feature Test
Update Test Update Test
Bug Fix Test
Feature Test
Collect ResultsTests passed ?
Approve Send report [email protected]
Reject
ISV Tests ISV Tests ISV Tests ISV Tests
Alert TPMs, Sec Team,Support (NTS), TAMs, ISVs
NO
YES
Feature Testfor Type A/B
Feature Testby OES team for Type C
Bug Fix Test
?
openSUSE & QA
© Novell Inc. All rights reserved
13
How can the community contribute?
• bugzilla / testopia– main interface and used by Novell internally as well
• Novell is opensourcing test tools / infrastructure / documentation
– QA code will be available in the Build Service
– Build Service will be able to run tests after a build job is done
– easier integration for Software maintainers
• building a QA community– awards & prizes for the best QA contributors (reports, test
cases, ...)
• better cooperation between the community and Novell– QA community manager
Testopia (off-line) Demo
© Novell Inc. All rights reserved
15
© Novell Inc. All rights reserved
16
© Novell Inc. All rights reserved
17
© Novell Inc. All rights reserved
18
© Novell Inc. All rights reserved
19
© Novell Inc. All rights reserved
20
© Novell Inc. All rights reserved
21
© Novell Inc. All rights reserved
22
© Novell Inc. All rights reserved
23
© Novell Inc. All rights reserved
24
Questions, Feedback, Discussions
© Novell Inc. All rights reserved
26
Thank you for your attention
Unpublished Work of Novell, Inc. All Rights Reserved.This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc. Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.
General DisclaimerThis document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. Novell, Inc. makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for Novell products remains at the sole discretion of Novell. Further, Novell, Inc. reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.