W7 Test Management
5/1/2013 1:45:00 PM
Taming the Beast: Test/QA on Large-
scale Projects
Presented by:
Shaun Bradshaw
Zenergy Technologies, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Shaun Bradshaw
Cofounder of Zenergy Technologies, Inc., a QA and agile solutions consulting firm (zenergytechnologies.com), Shaun Bradshaw has spent the past fifteen years advising, teaching, and mentoring clients to improve their QA and test processes with effective testing and test management techniques. He is the coauthor, editor, and trainer of a suite of methodologies covering testing, test management, and test metrics. A popular speaker at many of the major QA industry conferences, Shaun is known for his presentations on test metrics, the S-Curve, and the Zero Bug Bounce.
4/16/2013
1
AGENDAAGENDAAGENDAAGENDA
• What constitutes a large-scale project?
• QA Approach
• Data Strategy
• Environments / Infrastructure
• Team Structure
• Communications
• Integration Testing
• Performance Testing
• Key Take Aways
4/16/2013
2
LARGELARGELARGELARGE––––SCALESCALESCALESCALE PROJECTSPROJECTSPROJECTSPROJECTS
• What constitutes a “largeWhat constitutes a “largeWhat constitutes a “largeWhat constitutes a “large----scale project”?scale project”?scale project”?scale project”?
� Number of resources?
� Duration?
� Number of integrated components?
� Cost?
� Other?
LARGELARGELARGELARGE----SCALESCALESCALESCALE PROJECTSPROJECTSPROJECTSPROJECTS
• What constitutes a “largeWhat constitutes a “largeWhat constitutes a “largeWhat constitutes a “large----scale project”?scale project”?scale project”?scale project”?
� Number of resources? 100+
� Duration? 2+ years
� Number of integrated components? 6 (major)
� Cost? $70 million
� Other? Multiple external vendors operating in
strict waterfall methodology
4/16/2013
3
TERMINOLOGYTERMINOLOGYTERMINOLOGYTERMINOLOGY
• System TestingSystem TestingSystem TestingSystem Testing
� Performed at the component level
� Includes functional, data transformation/validation, and security tests
• Integration TestingIntegration TestingIntegration TestingIntegration Testing
� Performed across multiple components of the solution
� Includes integration, data quality validation, and cross-component security tests
QAQAQAQA APPROACHAPPROACHAPPROACHAPPROACH
• Align with overall development methodologyAlign with overall development methodologyAlign with overall development methodologyAlign with overall development methodology
� Don’t create “culture shock” by following completely different test methodology� If the project is waterfall, utilize the V-model of QA or
something similar
� If the project is agile, help establish iterations that QA can work with and ensure open collaboration across the project
� Ensure the QA approach is thoroughly communicated across the project team
4/16/2013
4
QAQAQAQA APPROACHAPPROACHAPPROACHAPPROACH---- PHASEDPHASEDPHASEDPHASED TESTINGTESTINGTESTINGTESTING APPROACHAPPROACHAPPROACHAPPROACH
• Here was our approachHere was our approachHere was our approachHere was our approach
QAQAQAQA APPROACHAPPROACHAPPROACHAPPROACH---- HIGHHIGHHIGHHIGH LEVELLEVELLEVELLEVEL TIMELINETIMELINETIMELINETIMELINE
4/16/2013
5
DETERMINE THE DATA STRATEGYDETERMINE THE DATA STRATEGYDETERMINE THE DATA STRATEGYDETERMINE THE DATA STRATEGY
• Data AcquisitionData AcquisitionData AcquisitionData Acquisition
� Create test data � From component systems
� Based on test criteria
� Pull data from production� Ensure proper data quality
� Data meets the needs of the testing
� Data is in “proper” form
� Data masking?
� Data Volume?
ENVIRONMENTS/INFRASTRUCTUREENVIRONMENTS/INFRASTRUCTUREENVIRONMENTS/INFRASTRUCTUREENVIRONMENTS/INFRASTRUCTURE
• Determine, as early as possible, what environments Determine, as early as possible, what environments Determine, as early as possible, what environments Determine, as early as possible, what environments
are available & necessaryare available & necessaryare available & necessaryare available & necessary
� Component-level system test environment
� Integration test environment
� Performance test environment (possibly
production if this is the first instance of the
application)
� UAT environment
4/16/2013
6
TEAM STRUCTURETEAM STRUCTURETEAM STRUCTURETEAM STRUCTURE
Key RolesKey RolesKey RolesKey Roles
� Test Architect – Creates overall test strategy
communicates/socializes it to project team
� Test Manager – Manages day-to-day test activities
� Test Leads – Component-based/Technology-based
� Test Analysts – Develop and execute tests
� Test Project Manager – Ensures test activities are
properly scheduled
� Integration Lead – Pulls together integration test effort
TEAM STRUCTURETEAM STRUCTURETEAM STRUCTURETEAM STRUCTURE
• Key Roles Key Roles Key Roles Key Roles (continued)(continued)(continued)(continued)� Performance Lead/Engineers – Develops performance
test plan, scenarios, and executes� Test Data Modeler – Determines data needs &
acquires test data� Business Leads– provide user-type input/review of test
scenarios and test cases� Technical Leads – Assist in creation of QA environment
and data needs� Release Manager – Helps manage all configuration
and component movement
4/16/2013
7
COMMUNICATIONSCOMMUNICATIONSCOMMUNICATIONSCOMMUNICATIONS
• The BIGGEST Challenge for QAThe BIGGEST Challenge for QAThe BIGGEST Challenge for QAThe BIGGEST Challenge for QA
� Utilize 360° communication strategy� QA approach and reasoning to upper management (early
and often)
� Resource and schedule requirements to peers
� Tactics, techniques, and goals to down line resources
� Beware of assumptions made due to lack of
communication
COMMUNICATIONSCOMMUNICATIONSCOMMUNICATIONSCOMMUNICATIONS
• The BIGGEST Challenge for a LargeThe BIGGEST Challenge for a LargeThe BIGGEST Challenge for a LargeThe BIGGEST Challenge for a Large----Scale ProjectScale ProjectScale ProjectScale Project
� Identify the right resources to include when
discussing� QA Approach / Test Strategy
� Data Requirements / Strategy
� Resources & Schedules
� Test case framework
� Test phases
4/16/2013
8
INTEGRATIONINTEGRATIONINTEGRATIONINTEGRATION TESTINGTESTINGTESTINGTESTING
• The 2The 2The 2The 2ndndndnd BIGGEST Challenge for a LargeBIGGEST Challenge for a LargeBIGGEST Challenge for a LargeBIGGEST Challenge for a Large----Scale ProjectScale ProjectScale ProjectScale Project
� Logistical challenges
� Political challenges
• Purpose Purpose Purpose Purpose –––– demonstrates that the IT processes and demonstrates that the IT processes and demonstrates that the IT processes and demonstrates that the IT processes and
systems/components built to support business systems/components built to support business systems/components built to support business systems/components built to support business
processes are mutually integrated correctlyprocesses are mutually integrated correctlyprocesses are mutually integrated correctlyprocesses are mutually integrated correctly
INTEGRATIONINTEGRATIONINTEGRATIONINTEGRATION TESTINGTESTINGTESTINGTESTING
• Integration Test ApproachIntegration Test ApproachIntegration Test ApproachIntegration Test Approach
� Map out critical data flows between components (IDEF is a good model)
� Determine “day in the life” scenarios ensuring that all critical data flows are covered
� Develop integration test cases using previously created system test cases to minimize rework (use a modular test framework)
� Utilize a “bottom-up” approach where possible (start with C2C scenarios, then E2E scenarios)
4/16/2013
9
INTEGRATION TESTINGINTEGRATION TESTINGINTEGRATION TESTINGINTEGRATION TESTING
DRAFTDI
(Includes
COPS/RDM)
DW
(ODS/ODM)
BI
NXG
Ess
Base
PS
Formatted
Data to
JGen
(Batch
Data)
Combo Edit
Explosions
SIF Formatted Files
Direct to E2
Formatted Files
Processing Schedule
Attachments
Journal Line Attachments
Hyperion Out:
Actuals and
Budget
Hyperion In:
Budget Data
GL91I0014
Webservice PS Chart of
Accounts (partial or
complete) to be validated
by DI (PCR 33)
Reports
PeopleSoft UI
Spreadsheet Uploads
Converted e2
to PS Data
Manual Data Entries
Data Dictionary / SIF
NXG
Balance
Life Com
COPS Audit Info
Application and
User Security
Internal ControlsBusiness Rules
Rejected File
Self Service Balance
(Daily Balance File)
Config/Settings:
Open Periods, Mapping
Rules, Ledger Code,
Chart Fields, Open
Periods, Trees, Business
Units
JE Lines/
Hdr
(Nightly
Batch)
AutoSys
Excel Active Directory
PS Query
Results
E2 Mapping Rules
Black Box e2
Formatted Files
E&Y
CIMS
Server/Mainframe
Environment
Business Rules
Error Theshold
E2 to PS Chart field
translation - flat file
Input (PCR 15)
GLCDS
Converted Rejection Info
Fin Detail
Table (Nightly
Batch)
Compass Integration Model A1101711 1545.vsd
Date Dimension File (On
Demand)
Via Data Analyst Package
to RDM
Budget
On
Demand
(ETL)
Financial
Transactions
File (FTF)
E2 Balances
Manual Upload
Of Opening /
Outstanding Items
Match/Pass RulesRemitter
Lookup
Table
Account
Structure
Extract
Extract
Criteria
Template
Manual
Upload
Outstanding
Template
Trigger
for FTF
Trigger for
Balance
Reconciliation and Reporting Data
Common View CSA
(Webpage)
PS Universe
(Read Only)
DI Staging Tables
(Read Only)
ODS/ODM/RDM
Universe (Read
Only)
COPS Universe
(Read Only)
Non Chart Field Table
(Excel File) Via Data
Analyst Package to RDM
Stop
File
VPD
Security
Oracle
Webservice NDM
Informatica
Self
Service
Business
Object
Reports
Self
Service
EssBase
Reports
80 Compass
Developed
Reports
BO XIR3 Oracle DBAs
BO Portal
(Internet)
Financial ControllerHyperion EssBase
9.3.3 (Arc Cube) Consumers
InfoView
Oracle
Excel Add-in
Accurate Webclient
Informatica
Business Objects
Commonview/CSA
Prod Application
Admin
Informatica
DA Tool Email Server
WLK Day
Transactions
Outstanding Items
Conversion Data
Remitter
Number Table
Excel
(min
2007)
Perl and
Korn Shell,
Acs Scripts
NXG BO
Universe
Reports
NXG Admin
NXG Reconciler
BO User
Web
Driver
Generator
Ariba
JE Lines/Hdr
Ledger
(Trickle
Feed)
PeopleSoft Autosys
Adapter
SFTP
JGen Mappings
Users
RDM
Universe
(Read Only)
Webservice
Validation
Data (PCR
33)
ETL Controls
for ODS data
movement: All
but ODM data
EIS Trees (ARC)
EAS Ledger
Table Data
(ARC)
Date
Dimension
CDS
GL
Data Analyst
Admins
Admins
PS Users
Manual Open/Close Process
Processing Schedule
Processing Schedule
Excel Upload Template
Date/Time
Manual Schedule
Processing Schedule
Date/Time
Internal Controls
Internal Controls
Internal Controls
Internal ControlsProcessing Schedule
Date/Time
Application and
User SecurityApplication and
User Security
Application and
User Security
Active Directory
System
Connectivity
Oracle InformaticaOracle Active Directory
AutoSys
System
Connectivity
RDM
01
02
03
040506
01
02
03
04
05
30 31 1011
13
1514
12
1514
01
02
03
010203
040506
10
11
30 31 32 10
30 31 32
1617
10
11
12
30 31
03
01
02
04
10111213
1415
01
03
Application and
User Security
Server/Mainframe
EnvironmentSystem
Connectivity
Server/Mainframe
EnvironmentSystem
Connectivity
AutoSys
InformaticaOracle Active Directory
Server/Mainframe
EnvironmentSystem
Connectivity
AutoSys
Webservice NDM
Server/Mainframe
EnvironmentSystem
Connectivity
AutoSysServer/Mainframe
Environment
System
Connectivity
Active Directory
Consumers
Shared File
System
Validation Error Report Shared File
System
Translated PS File
Admins
Cost Center Validation
Cost Center BU Account
Detail Extract via SFTP
PS
Upstream
Sources
Upstream
Sources
Upstream
Sources
e2
e2
NXG
Team
e2
Annuity
Team
BI Team
Journal Entry
Preparers
Accurate
WebClient
NXG Data Base View
Author: Alan Smith
Trigger
for FTF
Self Service Balance
(Daily Balance File)
Tree Files
Balance
Files
BO Reports
SQL Data
Load
CDS
ICAS
Cost Center Validation
Data
Modified ICAS Report
Excel File Manual Load
Policy Level Claims
Reports
Processing
Schedule
Hourly JE extract for spreadsheet upload
Policy and Company Number
Business Objects
Treasury
Tax
Invest
Accntg
Trigger for
Balance
01
0203
30 31
01
30 31
30 3101020304
Server/Mainframe
Environment
System
Connectivity
Webservice
Controls
for Trickle
Feed and
Batch for
JE Lines
etc. data
movement
Journal Entry
Preparers
Journal Entry
Preparers
12
04
32
Send controlled by DI but
does not go through DI
13
BI Team
Notes
1617
40
111213
1840
33
40
30
40
32
40
33
02
05
10
40
10
40
32
40
32
40
1718
How to read the model
ACTIVITY
A
ACTIVITY
B
A Sends to B
B Pulls from A
SourceInput
DestinationOutputPrimary Flow
Controls, Contraints,
Rules, Timings
Mechanisms,
Systems, Users
Mechanisms and non-data generated
Controls grouped to remove clutter of lines
Red lines/text indicate High Risk
Green line is for Reports only
Dotted line is for PCR 33 only
Labels,
Descriptions
10ICOM
Number
Input 01-09
Output 10-19
Control 30-39
Mech 40-49
INTEGRATIONINTEGRATIONINTEGRATIONINTEGRATION TESTINGTESTINGTESTINGTESTING
• Validation ApproachValidation ApproachValidation ApproachValidation Approach
� Focus on positive scenarios during integration
(use system testing for most of the negative test
scenarios)
� Determine the best approach given time,
resources, and risk� Validation of Execution
� Validation of Results
4/16/2013
10
INTEGRATIONINTEGRATIONINTEGRATIONINTEGRATION TESTINGTESTINGTESTINGTESTING
• Validation of ExecutionValidation of ExecutionValidation of ExecutionValidation of Execution� Assumes detailed validation of expected results was
completed during system testing and system test scripts are being re-used as part of integration
� Generally, the most efficient method of validating results for integration scripts, but increases the risk of missing potential defects
� Best used when a component has been thoroughly system tested - it is acceptable to simply verify the process executed
INTEGRATIONINTEGRATIONINTEGRATIONINTEGRATION TESTINGTESTINGTESTINGTESTING
• Validation of ResultsValidation of ResultsValidation of ResultsValidation of Results
� Doesn’t assume the correctness of results from
previous testing phases
� Necessary to fully validate the process outputs at
a detailed level via external calculations (i.e.
“tool”, spreadsheet, SQL query, manual
calculations, etc.)
� Although dependable, low risk this method
time/resource intensive
4/16/2013
11
INTEGRATIONINTEGRATIONINTEGRATIONINTEGRATION TESTINGTESTINGTESTINGTESTING
• ExecutionExecutionExecutionExecution
� Its all about COMMUNICATION
� Because an individual tester may not have
requisite knowledge to validate every step of an
integration scenario “hand-offs” are necessary� Assign a resource to manage the communication of the
hand offs (we don’t want tests sitting)
� Hold daily triage and schedule meetings
� Be flexible ☺
PERFORMANCEPERFORMANCEPERFORMANCEPERFORMANCE TESTINGTESTINGTESTINGTESTING
• Application PerformanceApplication PerformanceApplication PerformanceApplication Performance
� Verify each component is properly tuned as early
as feasible
� Align with system test phase
• Infrastructure PerformanceInfrastructure PerformanceInfrastructure PerformanceInfrastructure Performance
� Utilize an integrated environment to determine
issues in the infrastructure of the solution
� Align with integration test phase
4/16/2013
12
KEY TAKE AWAYSKEY TAKE AWAYSKEY TAKE AWAYSKEY TAKE AWAYS
• Develop an overall test strategy that aligns with the development methodology and culture of the organization
• Ensure key roles are properly filled� QA Architect� QA PM� Business Owners� Data Modeler
• Determine where/how you will get your data (the data strategy)
KEY TAKE AWAYSKEY TAKE AWAYSKEY TAKE AWAYSKEY TAKE AWAYS
• Don’t rush to integration testing
• Identify key business and technical resources to assist in developing the integration scenarios
• Utilize a modular testing framework so creation of integration tests requires little additional work
• Performance test key components as early as possible and ensure the entire solution is performance tested
• Communicate, communicate, communicate
4/16/2013
13
Questions?Questions?Questions?Questions?
Thank you!Thank you!Thank you!Thank you!