Managing Global Expansion Of Testing · Expansion in India n Need to consider local team structure — Need local QA Manager n Recruit the QA Manager and team leaders first — Add

Managing Global Expansion Of Testing

Alan Green, Mentor Graphics, UK

T4

Managing Global TestingEurostar 2006

Alan GreenQA ManagerIntegrated Electrical Systems Division (IESD)

Agenda1. Introduction

2. Organization1. IESD QA Organization2. Expansion in India

3. Lifecycle1. IESD Process Model2. IESD Development Lifecycle3. IESD Testing Lifecycle4. Reviews5. Integrating into development

process

4. Infrastructure1. Tool Support2. Environment testing

Introduction

Mentor GraphicsHeadquarters: Wilsonville, Oregon U.S.A.

n 300,000 Square Feet of Office & Laboratory Spacen 3,900 Employees Worldwide

— 1,000 at Wilsonville site

Mentor Graphics – IESD

IESD Offices

Wilsonville,U.S.

Altrincham &Newbury, UK

Hyderabad,India

IESD Markets

Capital Harness Systems (CHS)

n CHS is a suite of tools.n When combined the

tools cover a flow from logical systems design through to wiring harness design and manufacture.

n IESD is split into five teams, each working within a specific part of this flow.

HarnessSolutions

AnalysisSolutions

DesignSolutions

ViewSolutions

EnterpriseSolutions

Organization

IESD QA Organization

Expansion in Indian Need to consider local team structure

— Need local QA Managern Recruit the QA Manager and team leaders first

— Add them to the recruitment process

n Recruit the right mix of skills— QA guys

n Need to recruit some experienced testers— Will provide knowledge/experience to assist other testers— Will ensure that the test process is understood/followed— But may not be familiar with your domain

n Train testers in the use of your applications n Train testers in the domainn Provide marketing contacts who can answer any questions

— Domain guys n Understand the software to be tested but are not used to

“testing”— Can be trained in QA processes— Must sit QA Aptitude tests during interview process

Expansion in India (2)n Plan for several weeks of training

— In your QA processn Provide training in the TMap test processn Train testers in the required test techniques n Provide checklists to help ensure nothing is missed

— In your applications

n Establish a working relationship— Bring main team members to UK for training/staff introductions— Use regular inter-site visits to build the relationship

n Provide guidance & assistance— We initially used UK based QA Team leaders

n Ensure QA process is followedn Allocate tasks n Specify deadlinesn Review deliverables

— Aim for transfer to India based team leadersn Identify future team leaders and ensure they learn the processes you want

used

Keys to successn Communication

— Internal to QA teamn Management reporting

— Important to present QA reports to all teams— Ensure key information is given to all staff

n Status informationn Post release reportsn Results

n Team Working— Ensure you use all tools at your disposal to aid team

working— Greater interaction = Greater success

— Between QA & Developmentn where testers/developers are at different sites

— provide local QA contact to support open issuesn help keep things movingn easy to forget that someone is on another time zone

Additional Benefitsn Working around the clock

— We have been able to use the multi-location of the team to our advantage during some time critical tests/operations.

n S ite’s hand -off to next site at the start of the next day

n Larger team— Lower Staffing costs allow a larger team to be assembled

n Enabled standalone Service Pack test teamn Allowed a larger automation team to be created

Problem Areasn Staff retention in India

— Expect regular turnovern It is the norm to change jobs every 2 years

— Tempting overseas jobsn US/UK based job offers are compelling events

— Expectation of team leadershipn Staff expect to become team leaders in 2 years

n Solutions — Recruit staff whose family is local to office

n This means they are likely to remain in the area— Check past history

n Look for people that have worked overseas alreadyn How often have they changed jobs in the past?

— Demonstrate personal developmentn Provide ISEB Trainingn C reate job roles for “tool experts”n Allow staff to grow into team leading positions

Lifecycle

IESD Process ModelRational Unified Process

IESD Development Lifecycle

n Problems solved— How long should the constructions be?— Which features to put in C1 and which to put in C2?

Inception

Elaboration

Construction

Integration Test / Fix

Transition

Release

IESD Testing Lifecycle

Specification

Execution

Completion

Release

Planning

n Problems solved— How to fit TMap into an iterative cycle?— H ow to m ake the first test execution phase “useful”?— Integration week benefits: Snag meetings, pre-release builds

Detailed Testing LifecycleInception Elaboration Construction Transition

Functionality Development Test/Fix

Planning.

Identify FEATsIdentify QA teamsRisk AssessmentCreate Test PlanReview Test Plan

Test Planningn Tasks

— Identify features to be delivered— Identify inter-dependencies of the features

n Features may affect several tools when completen Q A assign “w hole” feature to ow ning team

— Ensure feature is tested across the flow as a single task— Find the implementation gaps?— Tester needs to understand the whole flow

— Assess the size of development teams in this releasen Developers may have moved teams to allow more functionality

to be added in a particular tooln Aim to make the QA team sizes proportional to development

team sizes on each tool— May need to move testers between tools— Good for x-tool knowledge/understanding

Test Planning (2)n Considerations

— Locations of testersn Try to keep teams of testers in single location

— Same time zones, aids team working, aids knowledge transfer— Experience of testers

n You need a QA lead with good knowledge of relevant applications

— Sometimes one is not available locally so off-site lead is assignedn Need to identify the next local team leadn Aim to enable this person during current releasen Give exposure to team lead activitiesn Allocate higher risk featuresn Invite them to management meetings

— Location of development teamn Is it different to location of QA team?

— Need to assign additional tester to team from development siten Acts as local QA contactn Aids communication and turnaround times

Risk-based testingn Determine the contents of each iterationn Hold a risk assessment for each product team

— based on severity and probabilityn Severity estimated by Marketing (what if not delivered?)n Probability estimated by Development (how complex?)

— Identifies amount of testing required per feature— Establishes an order to testing

n Higher risk = Earlier Test— Record the results in the Test Plan

Med

MedHigh

High

High

Low

Low

Med

Severity of Impact

Probability of Occurrence

Detailed Testing Lifecycle

Specification.

Write Test ConditionsReview Test ConditionsWrite Test CasesPrioritize Test CasesIdentify Test Environments

Inception Elaboration Construction TransitionFunctionality Development Test/Fix

Planning.


Test Specificationn Test Specification process

— Record test conditions (from use-case and functional spec)— Formal review of test conditions

n Ensure good coverage of testsn Good learning opportunity for new testers

— Document test cases — Prioritize test cases (P1, P2 or P3)

n Maximum 10% can be P1— Make testers really think about priority

n P1 test cases executed in Breadth Testingn P2 & P3 executed in Depth Testing

n Need to ensure Consistency— Use Test Techniques to generate Test Cases— Use a Test Specification template

Detailed Testing Lifecycle

Specification.


Completion

Defect ValidationRegression TestingMigration TestingCreate Automation scripts

Stabilization

Snag MeetingsConfirm Test Specs


Execution.

System Testing•Breath (P1)•Depth (P2 & P3)•Flow testing

Acceptance Testing (MAT)Defect Review MeetingsReport Weekly Metrics

Planning.


Test Automationn Dedicated Automation team in India

— Team employs java programmers

n Data driven approach being used— Testers use a custom Excel editor to create data files (that drive

tests)n Testers create automation directly from test specificationn Excel editor retrieves information about functions from a database

— Automators write java functions n Functions use a 3rd party tool to operate CHS toolsn Add Function name & parameters to the database

n Flow— Tester creates Excel data file & test data— Tester tests the Excel data file— Testers delivers to automation team

n Automation team own regression suite— Testers responsible for executing the test suite

IESD Testing Lifecycle (2)

Specification.


Completion

Defect ValidationRegression TestingMigration TestingCreate Automation scripts

Stabilization

Snag MeetingsConfirm Test Specs


Execution.

System Testing•Breath (P1)•Depth (P2 & P3)•Flow testing

Acceptance Testing (MAT)Defect Review MeetingsReport Weekly Metrics

Post-release Review

Planning.


Post release reviewsn IESD hold a post-release review after every release

— Aim: to review and amend our processes for the next release

n Representatives from the following groups present a summary to the meeting:— Marketing, Development, QA, Technical Writers— India technical centre

n We evaluate:— What went well in the release and why?— What went badly and why?

n We seek to gain consensus going forward:— What elements of the process should we ensure that we repeat

consistently?— What do we need to change and how?

n Finally, agree a plan for introduction of agreed measures: who, how & when.

Document Reviewsn Critical part of process

— Especially when teams split around the world

n Dual Purpose— Find defects— Ensure common understanding

n Main documents being reviewed— Use-Cases— Functional Specifications— Test Conditions— Test Plan/Test Schedule— Technical Publications

n Initially used Walkthrough reviews— Some issues:

n Reviews were taking too long— Too many unanswered questions in documents— Too many attendees

n Difficult to ensure consistency across different teams

Document Reviews – Growing Painsn N ext: started to introduce “roles”

— We allocated 2 roles to each meetingn Compare to sourcen Question software limitations

— Aim for more consistency— Aim to solve known problem areas

n Now: Moving to a formal review process— S ix “roles” defined— C reated a “pool” of staff from each team for each “role”— R eview ers are selected from the “pool” w hen m eeting is arranged— Entry criteria established for review process— Meeting is more controlled with less discussion allowed

Integrating into development processn Helping provide added value to whole development

lifecycle— Same process for all sites— Aim to build quality into the tools

n Examples— Requirements

n Use-case templates— Development

n Functional specification templaten Static Analysis of coden Automated Unit Testsn User Interface checklist

QA Regression Suite Usagen Initially regression suite used only on executables by testers.

n Problem: Needed a way to establish some entry-criteria for system testing.— Ensure basic functionality working.— R em o ve “false starts” from test execution.

n Solution: Introduce regression suite to the development environment.— H ad to create som e “developm ent environm ent” specific startup

scripts.

n Automation suite now executed nightly in development environment.— At all 4 development sites.— On local code base for that site (before delivery to mainline).— Unit tests and regression suite are executed.— Automated emails sent when incident occurs.

Tracking the status of release featuresn Problem: How to monitor that the process is being

followed by everyone and identify exact status of each feature?

n Solution: Internal system created to track features through the lifecycle— Identifies features selected for a particular release

n Tracks Marketing, Dev, QA, Tech Author activities— System provides:

n Links to all documentation during the lifecyclen Identifies key staff working on this FEATn A utom ated em ails on “state” changen Mandatory fields ensure minimum information is enteredn Web interface for all sitesn Custom reports available

Infrastructure

Tool Supportn Important to use same tools at all sites to ensure consistency

n Consider:— Availability

n Tools must be available 24 hours a day.— Accessibility

n Tools with a good web interface are easier to use and implement for everyone.

— Performancen It is important that all tools perform equally well at all sites.

— Processn The process for using the tool must be documented to ensure

consistent multi site use of a tool— Licensing

n Global licenses can help save money when working in multiple time zones.

— Commitmentn Management at all sites must be committed to use of the tools.

Where do IESD use tool support?n Code Development

— Single Java IDE used by all developersn Makes training of new staff consistent & allows a single training course to be

createdn Allow standardised set of checks for Static Analysis

n Configuration Management— Single tool used for CM

n Only a single process need be definedn Ensures code from all sites can be combined

n Defect Management— Single tool used for DM

n Lifecycle is defined and process documentedn Tool helps enforce the defect lifecylen Tool has excellent web interfacen Used by developers, users, customer support and QA

Where do IESD use tool support? (2)n Test Automation

— Standard tools allow automation to be used at all sites— Automation process is documented for automation team and testers

n Time Tracking— Web based time tracking system chosen

n Provides input for release metrics and trackingn IESD can report on time spent per activity post releasen Used to improve estimation

n Document Management— Web based document management system

n Allows fast access to documents from all sitesn Documents stored on ftp servern Access can be controlled to each document

Environment Testingn Aim: To test CHS on maximum number of supported

environments & configurations during test execution.

n Factors: QA teams at multiple sites gives greater flexibility for simultaneous testing.

n Ensure test team has access to mix of supported hardware/software configurations.

n But ensure test environments are not duplicated across the sites.

n Solution: Use environment test plan to detail environments to be tested per location per week— Prioritise H/M/L based on classification data.— Allocate configurations to sites based on hardware available at that

site.— Assign test execution week numbers to the configurations.— Tester at each site responsible for setting up weekly environments.— Arrange purchase & install of new hardware/software as needed.

n Cost of hardware may not be same? Maybe from a different budget?