Embed Size (px)
Citation preview
W8 Test Techniques 5/4/16 13:45
Testing in the New World of Off-‐the-‐Shelf Software
Presented by:
Gerie Owen
Eversource Energy
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.stareast.techwell.com/
Gerie Owen Eversource Energy Gerie Owen is a test architect at Eversource Energy, Certified ScrumMaster, conference presenter, and author on testing/test management topics. She is developing a test center of excellence and consults on test processes and tools. Gerie manages large, complex projects involving multiple applications; coordinates test teams across multiple time zones; and delivers high-‐quality projects on time and within budget. She enjoys mentoring new QA leads and brings a cohesive team approach to testing. Learn more about Gerie on her website, her Software Testing in the Trenches blog, and on Twitter @GerieOwen.
Testing in the New World of
Off-The-Shelf Software
Gerie Owen & Peter Varhol
About Us Gerie Owen
Test Architect at Eversource
Speaker and Writer on Testing topics
Experienced Tester, Test Lead, & Test Architect
Peter Varhol
Evangelist at Dynatrace
International speaker on technology topics
Technology Evangelist and Writer, Product Manager, University Professor
Meet Our Cats... Max, Stephanie & Monica
Agenda • What is Off-The-Shelf Software?
• Why Test COTS?
• Challenges of Testing COTS
• Differences from other types of testing
• Framework for Testing COTS
• The Test Team
What are COTS Applications?
• Commercial Off The Shelf Applications are: o Developed by a vendor
o Sold, leased or licensed to business organizations
o Typically serve enterprise-wide functions
Examples of COTS Application
• Many are Enterprise Resource Planning (ERP) or
Customer Relationship Management (CRM) o Workday
o Workforce
o SAP
o Salesforce.com
o Peoplesoft
o Oracle Financials
Examples of COTS Applications
• Some are smaller, niche products o Geospatial Information Systems (GIS)
• SmallWorld
• ArcGIS
Challenges of COTS Testing
o COTS “projects” are often considered an implementation effort rather
than a project
o No access to source code
o Traditional requirements are not what’s needed
o Potential for compatibility issues
Challenges of COTS Testing
o Vendor’s level of quality is unknown
o Vendor’s development processes are not always known
o Lack of control over upgrade schedules and quality
Why Test COTS Applications?
• No organization uses them completely “off the
shelf”
• Most COTS applications will be integrated with
internal legacy applications
• Data will be converted and migrated
• Interoperability and performance must be tested
How is testing COTS Different than other types of
testing? • The Focus is on Business Process
o It’s NOT testing core functionality
• The Focus is on Integrations
• The Focus is on Data
• The Focus is on Performance
• Multiple Tester Skill Sets are needed
COTS Test Framework • Four Areas of Focus
• Business Process
• Integrations
• Data Migration & Conversion
• Performance & Interoperability
• End to End Test
Business Process • GUI Configurations and Customizations
• Business Processes o Does it fully support the existing or planned business processes?
GUI Configurations & Customizations
• What’s the difference o Configurations
• Changes made through an administration module by the vendor or
integration
• Often based on organization-specific business rules
o Customizations
• Changes made to the core product code or database
• Often made to accommodate organization-specific business
processes
GUI Configurations & Customizations
• Testing Implications: o Both are the high priority areas for functional testing
o Understand what requirements are met by each
o For customizations, understand what code was changed
o Regression test all customizations
Business Processes • Business Processes
o A set of related activities, including inputs and outputs
o Executed in a specific order
o that accomplish a goal
• Activities and order may change in a COTS
implementation, but accomplish the same goal
Business Processes • Testing Implications
o Design test cases by business component
o Role Based
• Include negative test cases
o Link test cases into a complete business process flow
Integrations • All systems that interact with the COTS applications
o All integrations need to be documented
o End-to-end workflow of how data flows among the integrations
o Where the COTS application fits within the overall workflow
Integrations • Testing Implications:
o What is the nature of the interaction?
• Does the COTS application send to, receive from or both
o How is data transmitted?
• i.e, ETLs, API, .csv
o When does the data transmission take place
• Real-time, batch job, scheduled or manually pushed
o How frequent is the data transmission?
o How will the code in each integration be changed?
Integrations • Develop an Integration Map:
o Place the COTS system in the middle, if appropriate
o Place integrating application in order of data flow
o Indicate the direction of the data flow using arrows
o Indicate method and frequency of transmission using color codes or text
• This can also be done as a matrix
Integration Map
Integration Matrix
Integration No. Integration Name
(Owner) Integration Description
Source System Name
Source System
Integration Owner
Destination System Name
Destination System Integration Owner
Integration Type
Integration Frequency /
Volume / Speed
1 Daily Customer Information Daily delta feed to COTS CustInfo Mgr Name COTS Mgr Name FTP nightly
2
Shadow Database Extract Process Shadow Database extract file XYZ Mgr Name COTS Mgr Name FTP Every two weeks
3 New Orders realtime feed NewORD Mgr Name COTS Mgr Name webMethods realtime
4
5
6
7
Data Migration & Conversion
• What’s the difference? o Data Migration
• The process of transferring data from one system to another while
changing storage, database or application.
o Data Conversion
• The transformation of data from one form to another.
o Often, a data cleansing effort must be undertaken prior to migration and
conversion
• Begin data review and cleansing as early as possible in the project
Data Migration & Conversion
• Testing Implications: o Functional testing of data
o Real-time conversion process
Data Migration & Conversion
o Functional testing of data
• Validate data mapping
• Extract Transform Load (ETL) process
• Data states
o New, open, approved, closed
Data Migration & Conversions
• Real-time Conversion Process o How long does the conversion take?
o Can it be completed in the Go-Live timeframe?
Performance • Testing Implications:
o Ensure that the COTS application performs under the specified workload
• Stress testing is as important as load if not more
• Determine if the application will support "worst case" production load
conditions.
o Data-Intensive
• Number of Scenarios
• Volume of Data
Performance • Focus is transaction-based
Test Type Definition
Endurance Applying varying loads to the application under test for an extended period of
time to validate that the performance requirements related to production loads
and durations of such loads are met. This can be considered a component of load
testing.
Capacity Applying concurrent transactions to determine how many users and/or
transactions the application will support and still meet performance
requirements.
Response Time Measures how long it takes the application to return back a response when a
request is sent. Sometimes called round trip time.
Soak Applying high loads of transactions usually more than expected in production to
assess application stability over time.
End To End Test • Simulates actual usage of the COTS application
o Final Regression test after all integration, data and performance testing is
completed
o Test Scenarios include every business process
o Test Cases document data inputs and outputs between the COTS
application and the integrations
o Ideally, include UAT testers who are real business users
The Test Team • Testers with different skills sets are needed:
o Front-End Testers
• Understand business processes
• Develop both positive and negative test cases
• Experienced at GUI testing
o Back-End Testers
• Integration and Data Specialists
• Write SQL queries
• Understand the extract/transform/load process
• Ability to simulate data integrations
o Performance Engineers
Helpful Hints • The Test Manager should review the Statement of
Work (SOW) before it is signed, if possible. o Data Privacy Compliance
o Defect severity and priority definitions
o Defect turnover requirements
o Acceptance criteria
Summary • Yes, COTS applications require testing
o But a different kind of testing
o Fitness for purpose
o Integration
• You need a clear understanding of organizational
goals
• And how to test to those goals
Questions?
