Upload
cory-sherman
View
217
Download
5
Embed Size (px)
Citation preview
Presented To:
April 21, 2023
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
Sean Campion [scamp]
27 June 2007 TopCoder Open, Las Vegas
TopCoder Component-Based Software Engineering (CBSE)
Quality Assurance (QA)
• CBSE Quality Assurance Specific Challenges
• scamp@home
• TopCoder Process from QA perspective
• Wrap-up
[ ] Agenda
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
[ ] CBSE Quality Assurance Specific Challenges
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
The following are common QA problems cited against the use of reusable components. I will discuss each issue, and what TopCoder does to address the issue.
[ ] CBSE Quality Assurance Specific Challenges
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
Issue: Lack of access to commercial component source code.
TopCoder: We provide both the source code and all design documentation for every component that we produce.
[ ] CBSE Quality Assurance Specific Challenges
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
Issue: Components are not readily testable.
TopCoder: All of our components are required to have unit tests of stress, failure, and accuracy.
[ ] CBSE Quality Assurance Specific Challenges
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
Issue: Difficult to use and interpret tests from component vendors
TopCoder: Our documentation provides examples on how to run the tests; we also provide the test results in the form of logs.
[ ] CBSE Quality Assurance Specific Challenges
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
Issue: Concern that components were not developed for a specific operating environment; and that developing a component test bed for all possible environments is too costly and time consuming.
TopCoder: We are developing scamp@home, a software system that will take advantage of participants diverse environments to test components.
[ ] scamp@home
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
scamp@home- Distributes component tests to participating
computers.- Takes advantage of diverse environments,
hardware, character sets, and so on.- Massively parallel- Can be setup at customer sites to start validating
existing components during architecture phase on actual target environment.
- Customers can use to test their own components.- Ideal for regression-testing when component is
updated.
[ ] scamp@home
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
Data WarehouseMetric Reporting and User DB
-End1
*
-End2
*
-End1
*
-End2
*
SCAMP@Home -- Macintosh (several versions : SCAMP@home
SCAMP@Home -- Windows(several versions : SCAMP@home
SCAMP@Home -- Linux(several versions) : SCAMP@home
SCAMP@Home -- PALM Mobile(several versions) : SCAMP@home
SCAMP@Home -- WindowsMobile(several versions) : SCAMP@home
SCAMP@Home -- PALM Mobile(several versions) : SCAMP@home
Application Server
Metric Capturer
Environment Tracker
Component DistributorSCAMP@home
CPUIdleIdentification
Component Downloader
Metric Upload
Metric Capturer
Environment Capturer
Cell Phone Synchronization
-End3
* -End4 *
<<ETL>>
<<JDBC>>
<<HTTPS>>
Test Case Execution User Registration
User Security and Administration
[ ] TopCoder Process from a QA Perspective
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
Specification Phase- Mid and end-point peer reviews of the
application specification, conducted by 2 Project Managers and 1 Architect.
- High-level test cases, tied to application level functionality, developed.
- Customer review and sign-off on application specification.
[ ] TopCoder Process from a QA Perspective
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
Architecture Phase- Mid and end-point peer reviews of the
architecture design, conducted by 1 Project Manager and 2 Architects.
- High-level test cases expanded into detailed test cases and scenarios.
- Customer review and sign-off on architecture design.
- Future: Existing components identified for reuse can be validated using scamp@home, including on the customer target systems if desired.
[ ] TopCoder Process from a QA Perspective
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
Component Production Phase - Design- Design competitions; designers know they are
competing against each other.- Automated submission screening tool.- Manual screening phase.- Review Board.- Scorecard-based evaluation.- Final fixes.- Winning designer required to support the
development phase.
[ ] TopCoder Process from a QA Perspective
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
Component Production Phase - Development- Development competitions; developers know
they are competing against each other.- Automated submission screening tool.- Manual screening phase.- Review Board.- Scorecard-based evaluation.- Final fixes.- Unit tests: stress, failure, accuracy- Coverage Tool: minimum of 85% line coverage
from unit tests.
[ ] TopCoder Process from a QA Perspective
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
Assembly Phase- Assembly competitions; teams know they are
competing against each other.- Manual screening phase.- Review Board.- Scorecard-based evaluation.- Final fixes.
[ ] TopCoder Process from a QA Perspective
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
Assembly Phase - Testing- Testing competitions; teams know they are
competing against each other.- Produces automated tests that will be run
against the assembly competition submissions.
[ ] TopCoder Process from a QA Perspective
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
Certification Phase- Replicate the customer’s target environment on
a TC server.- Deploy and rigorously test the winning assembly
submission.- Run automated tests against the submission.- Run manual tests against the submission.- Early customer access to the TC testing
environment for immediate feedback.
[ ] TopCoder Process from a QA Perspective
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
User Acceptance Testing Phase- Customer has option of performing their own
User Acceptance Testing.- TopCoder will directly support any UAT.
[ ] TopCoder Process from a QA Perspective
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
Deployment Phase - TopCoder will support the deployed application
for 90 days.- Customer receives all source code, including
component source code, and all design documentation from all phases.
[ ] Metrics
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
Here’s how TopCoder compares to industry standards:
Industry Defects per KLOC: 3.43
TopCoder Defects per KLOC: 2.10
[ ] Wrap-Up
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
- Quality Assurance addressed at every stage of the process in one form or another.
- TopCoder continuously looking to improve QA through processes and tools.
- scamp@home addresses a primary concern of reusable components.
- Quality software distinguishes TopCoder from other software.
[ ] References
CONFIDENTIAL Copyright 2007, TopCoder, Inc.
- “Testing and Quality Assurance for Component-Based Software”, Jerry Zeyu Gao, H.-S. Jacob Tsao, & Ye Wu; Artech House, Boston, 2003.
- “Risk Management in Component-Based Development: A Separation of Concerns Perspective”, Awais Rashid, Gerald Kotonya, Computing Department, Lancaster University, UK.
- “The Trouble With Testing Components”, Elaine J. Weyuker, in “Component Based Software Engineering”, Chapter 28, Addison-Wesley, 2001.
- “CBS Testing Requirements and Test Case Process Documentation Revisited”, Nasib S. Gill, Pradeep Tomar, Department of Computer Science and Applications, Haryana India, March 2007. From ACM SIGSOFT Software Engineering Notes, Volume 32 Number 2.
- “The QSM Software Alamanac: Application Development Series”, QSM Inc., 2006.