Embed Size (px)
DESCRIPTION
FOCUS Roundtable:. Advances in. Advances in. Advances in. Advances in. Advances in. Advances in. Advances in. Advances in. Advances in. Advances in. Advances in. Advances in. FOCUS Quality Assurance. 10 of them stacked up. One move by the arrows every 60 seconds. Joel Starkman - PowerPoint PPT Presentation
Citation preview
Copyright 2007, Information Builders. Slide 1
Joel StarkmanDirector of Operations
June, 2008
FOCUS Quality AssuranceAdvances inAdvances inAdvances inAdvances inAdvances inAdvances inAdvances inAdvances inAdvances inAdvances inAdvances in Advances in
FOCUS Roundtable:
Copyright 2007, Information Builders. Slide 2
Advances in FOCUS Quality AssuranceCustom-Built Testing System Envelope
Catalogs and executes all tests across the entire company• Captures results and compares to approved base results
Web-based interface to view results, drill down to code, intelligent cross-platform analysis
Runs 5000-10000 test-focexecs every night on every platform on several different daily gens of the product - mainframe (z/OS and CMS), plus Linux, AS400, Windows, and many Unix platforms
Many test-focexecs contain 5+ tests each. Many others generate hundreds or thousands of test variations via Dialog Manager loops
• Date and datetime combinations spanning centuries • Thousands of /format variations everywhere formats can appear
New IBI invention to run interactive tests in batch (FOCREPLAY)
All combined, about 100,000 tests every night QA and Programming staff examine results daily
• Discrepancies are reported and fixed for the next cycle• Overnight coverage via staff located world-wide
Copyright 2007, Information Builders. Slide 3
Advances in FOCUS Quality AssuranceNew Testing Concepts
Random test-generation and test combinations
• Hits combinations we could never predict Random execution paths simulate live sessions
• Non-predictable results, different every day,
ok as long as it finishes cleanly Real customer applications simulated in-house Parallel runs of random test combination against prior release
• Compare results to each other rather than to approved base results
The FOCUS language has combinations.
How can you possibly test every one of them?
Copyright 2007, Information Builders. Slide 4
New QA Tools and MechanismsAll created in the last 3 years
WHERE GeneratorSU Stress
Mechanism
SU ScenariosMODIFY, MAINTAIN, HLI
TABLE
Syntax Generator
Parser
Stress Tests
System Stress Tests
Intelligent Testing System
Customer Applications
Copyright 2007, Information Builders. Slide 5
New QA Tools and MechanismsSU Scenarios
WHERE GeneratorSU Stress
Mechanism
SU ScenariosMODIFY, MAINTAIN, HLI
TABLE
Syntax Generator
Parser
Stress Tests
System Stress Tests
Intelligent Testing System
Customer Applications
Copyright 2007, Information Builders. Slide 6
New QA Tools and MechanismsSU Scenarios
Plays every logical combination (scenario) that forces two or more users to collide on specific records or database paths with conflicting intentions to Include, Update and Delete that record or path
Results in FOCCURRENT return codes that identify the conflict, or predictably affects a TABLE that is processing at the same time
Plays all of these
against each other: MAINTAIN MODIFY TABLE HLI Commit
SU ScenariosMODIFY, MAINTAIN, HLI
Copyright 2007, Information Builders. Slide 7
New QA Tools and MechanismsParser Stress Tests
WHERE GeneratorSU Stress
Mechanism
SU ScenariosMODIFY, MAINTAIN, HLI
TABLE
Syntax Generator
Parser
Stress Tests
System Stress Tests
Intelligent Testing System
Customer Applications
Copyright 2007, Information Builders. Slide 8
New QA Tools and MechanismsParser Stress Tests
Use every FOCUS keyword in every possible position in the TABLE syntax to uncover language parser inconsistencies
TABLE FILE BY SUM BY/BY_BY AS 'BY'BY BY PRINT BY COMPUTE BY = BY/BY; BY BY BY BY SUBTOTALON BY SUBHEAD "BY <BY " END
Parser
Stress Tests
Copyright 2007, Information Builders. Slide 9
New QA Tools and MechanismsTABLE Syntax Generator
WHERE GeneratorSU Stress
Mechanism
SU ScenariosMODIFY, MAINTAIN, HLI
TABLE
Syntax Generator
Parser
Stress Tests
System Stress Tests
Intelligent Testing System
Customer Applications
Copyright 2007, Information Builders. Slide 10
New QA Tools and MechanismsTABLE Syntax Generator
Randomly generates valid TABLE syntax Random keyword selection and order, within syntactic rules Random number of keywords, fields, sorting, prefixes, selection criteria,
computations, text, summary lines, SET environments Generates thousands of different combinations every day Compares current release to any other release
TABLE
Syntax Generator
Copyright 2007, Information Builders. Slide 11
New QA Tools and MechanismsCustomer Applications
WHERE GeneratorSU Stress
Mechanism
SU ScenariosMODIFY, MAINTAIN, HLI
TABLE
Syntax Generator
Parser
Stress Tests
System Stress Tests
Intelligent Testing System
Customer Applications
Copyright 2007, Information Builders. Slide 12
New QA Tools and MechanismsCustomer Applications
Customer passes Customer passes application to Information application to Information
BuildersBuilders
Customer environment is Customer environment is duplicated or closely duplicated or closely
simulated on IBI hardwsimulated on IBI hardw
Database is snapshot of Database is snapshot of actual data or contrived actual data or contrived
data valuesdata values
Application is validated on Application is validated on current gen of next release current gen of next release
– bugs fixed– bugs fixed
Customer informed of Customer informed of pass/fail results and pass/fail results and
migration adjustmentsmigration adjustmentscustomer applications running in-house today!
12345678
Customer Applications
Copyright 2007, Information Builders. Slide 13
New QA Tools and MechanismsWHERE Generator
WHERE GeneratorSU Stress
Mechanism
SU ScenariosMODIFY, MAINTAIN, HLI
TABLE
Syntax Generator
Parser
Stress Tests
System Stress Tests
Intelligent Testing System
Customer Applications
Copyright 2007, Information Builders. Slide 14
New QA Tools and MechanismsWHERE Generator
Generates thousands of complex, syntactically valid WHERE syntax combinations
Parenthetical expressions, calculations, subroutine calls, random field selections, varied comparison operators
Dynamically incorporated into standard TABLE syntax and executed
WHERE Generator
Copyright 2007, Information Builders. Slide 15
New QA Tools and MechanismsIntelligent Testing System
WHERE GeneratorSU Stress
Mechanism
SU ScenariosMODIFY, MAINTAIN, HLI
TABLE
Syntax Generator
Parser
Stress Tests
System Stress Tests
Intelligent Testing System
Customer Applications
Copyright 2007, Information Builders. Slide 16
New QA Tools and MechanismsIntelligent Test System
Randomly combines cataloged individual tests into consecutive executions, looking for unpredicted interactions
Runs twice - against current and previous releases Compares results, extracts suspicious activity, isolates the causal
test, reruns separately, reports analysis to QA for further investigation Daily alters test code and test environments on the fly to expose tests to
originally-unintended conditions
Intelligent Testing System
Copyright 2007, Information Builders. Slide 17
New QA Tools and MechanismsSU Stress Mechanism
WHERE GeneratorSU Stress
Mechanism
SU ScenariosMODIFY, MAINTAIN, HLI
TABLE
Syntax Generator
Parser
Stress Tests
System Stress Tests
Intelligent Testing System
Customer Applications
Copyright 2007, Information Builders. Slide 18
New QA Tools and MechanismsSU Stress Mechanism
SU Stress Mechanism
Objective: Crash the sink machine or damage a database, and capture the evidence for evaluation
Random number of tasks per user, random task selection based on weighted frequencies, random inter-task and intra-task time delays, random data selection. Conflicts are unpredictable, like real applications.
Randomly performs Includes, Updates, Deletes, TABLE extracts, volume loads, intentional delays, user out to lunch or crash, HLI via COBOL
Settable # of simultaneous users; users drop off – new ones start up Run it for 10 minutes or 10 days Tracks every activity, generates traces, analyzed after run completes
Tens to 100’s of simultaneous users randomly perform a large variety of
tasks against sink machine databases in real time
Copyright 2007, Information Builders. Slide 19
New QA Tools and MechanismsSystem Stress Tests
WHERE GeneratorSU Stress
Mechanism
SU ScenariosMODIFY, MAINTAIN, HLI
TABLE
Syntax Generator
Parser
Stress Tests
System Stress Tests
Intelligent Testing System
Customer Applications
Copyright 2007, Information Builders. Slide 20
New QA Tools and MechanismsSystem Stress Tests
Concentrate specifically on how FOCUS interacts with the operating system environment
Expect files to be sized, stored, opened, closed, appended, removed, etc. properly at expected times with expected return codes
Disks, directories, pds’s, temp areas, etc. to be accessible and utilized as expected
FOCUS’s awareness of other environmental conditions and changes that affect its processing, with proper messaging and handling
System Stress Tests
Copyright 2007, Information Builders. Slide 21
Advances in FOCUS Quality Assurance
WHERE GeneratorSU Stress
Mechanism
SU ScenariosMODIFY, MAINTAIN, HLI
TABLE
Syntax Generator
Parser
Stress Tests
System Stress Tests
Intelligent Testing System
Customer Applications
![[List University Name] Quality Assurance Program ... Assurance Foms/University... · [List University Name] Quality Assurance Program Description Document ... Quality Assurance Program](https://img.pdfslide.us/doc/110x75/5ab8c3d37f8b9aa6018d08ac/list-university-name-quality-assurance-program-assurance-fomsuniversitylist.jpg)
