Testing Analysis and Design Models Review Class 3 Overview of Testing Analysis and Design Models Models and Testing Guided Inspection Analysis Model Design

Testing Analysis and Design Models

• Review Class 3• Overview of Testing Analysis and

Design Models• Models and Testing• Guided Inspection• Analysis Model• Design Model

CEN 5076 Class 4 – 09/26

CEN 5076 Class 4 - 09/26 2

Testing Process Overview• Testability:

1. The degree to which a system or component facilitates the establishment of test criteria and the performance of tests to determine whether those criteria have been met. [IEEE 610]

2. The degree to which a requirement is stated in terms that permit establishment of test criteria and performance of tests to determine whether those criteria have been met. [IEEE 610]

• Test criteria- The criteria that a system or component must

meet in order to pass a given test. [IEEE 610]

CEN 5076 Class 4 - 09/26 3

Testing Process OverviewThere are two types of testing criteria:1. Test data selection criterion –

represents a rule used to determine which test case to select.

2. Test data adequacy criterion – a rule used determine whether or not sufficient testing has been performed.

• Test data selection criterion serves as the basis for picking a test set to satisfy some goal, while a test data adequacy criterion checks to see whether a previously selected test set satisfies the goal. [Weyuker 93]

CEN 5076 Class 4 - 09/26 4

Risk Analysis – A Tool for Testing• Risk

1. Anything that threatens the successful achievement of a project’s goal.

2. An event that that has some probability of occurring, and if it does there will be some lost.

• Risk-based testing principle:- Test most heavily those portions of the system

that pose the highest risk to the project to ensure that the most harmful faults are identified.

• Risk analysis – a procedure for identifying risks and for identifying ways to prevent potential problems from becoming real.

CEN 5076 Class 4 - 09/26 5

Use Case Test Plan• Describes the system level test to be derived

from a single use case.

• Incorporated by reference into both the integration and systems test plan.

• Types of use cases: High-level – are abstract use cases that are the basis

for being extended to end-to-end use cases.

Functional sub-use cases – are aggregated into end-to-end system-level use cases.

End-to-end – represents a complete transaction in the operation of the system.

CEN 5076 Class 4 - 09/26 6

Use Case Test Plan• Other uses cases:

Report – access information in the system, summarize it, and format it for presentation to the user.

Boundary – describe startup, shutdown, and exceptional conditions.

• Figs 3.18, 3.19, and 3.20, Pgs 102, 103, and 104.

CEN 5076 Class 4 - 09/26 7

Overview of Testing Analysis and Design (A & D) models

• Main activity is the guided inspection when testing the A & D models.

Steps:• Organization of the guided inspection.

– Allocation of roles, tasks and schedule– Identify objectives of inspection

• Preparing for the inspection.- Creating test cases- Completing checklist

CEN 5076 Class 4 - 09/26 8

Overview of Testing A & D models

- Symbolically testing the model using test cases- If prototype available, execute the test cases- Distribute results of pre-inspection documentation

• Guided inspection- Developers walk the inspectors through the model

using the test cases.– Testers analyze the walkthroughs done by the

developers.– A test report is created that states the problems

encountered during the completion of the checklists and the symbolic execution of the test cases.

CEN 5076 Class 4 - 09/26 9


Recall: • An inspection is a formal small group

procedure for detailed scrutiny of a technical work product.

• Cost effective and efficient for defect prevention.

• Process has a formal entry-exit criteria, defined checklist, captures process and product metrics, has a follow up session.

CEN 5076 Class 4 - 09/26 10


Guided inspection:

• Verifies the models as they are created and validates the completed model against the project requirements.

• Is an enhanced what inspection technique.

• Focuses on what should be there not what is there.

• Applies the testing perspective very early in the development process.

CEN 5076 Class 4 - 09/26 11


Guided inspection cont:

• Boundaries of the inspection – scope and depth of the information to be inspected.

• Scope – the set of uses cases.

• Depth – levels of containment in the composition hierarchy.

CEN 5076 Class 4 - 09/26 12

Models and Testing

• Testing searches a space for those points (or inputs) that uncovers bugs.

• It is usually impossible to search the entire input domain in order to uncover bugs.

• Therefore searching the input space during testing should be: systematic, focused and automated. [Binder 2000]

• Systematic - every targeted combination is tried.

• Focused – where the bugs are likely to exist.

CEN 5076 Class 4 - 09/26 13

Models and Testing cont

• Automated – required if we are to run a large number of consistent and repeatable tests.

• Testing using models achieved the above objectives.

• A model captures essential relationships but is easier to develop or analyze than the system under study. [Binder 2000]

• Properties of a model:

- must have a well-defined subject.

CEN 5076 Class 4 - 09/26 14

Models and Testing cont.- must have a point of view and theory.

- must have a notation.

- are complex artifacts i.e., requires skill to develop.

Validation, verification, and testing?

• Model validation – establishes confidence in the sufficiency of a formal abstract component w.r.t. its informal behavioral requirements.

• Verification – shows that the implementation is correct w.r.t. its representation, without executing it. Informal (checklist) formal (proof)

CEN 5076 Class 4 - 09/26 15


• Consistency checking – evaluates a representation as an instance of its meta-model. meta-model – definition of a modeling technique

i.e. notation and rules.

instance-model – a representation constructed using the technique defined by the meta-model.

• Responsibility-based testing – evaluates whether observable behavior conforms to the representation.

CEN 5076 Class 4 - 09/26 16


• Implementation-based testing – evaluates observable behavior w.r.t. a test model derived from the implementation.

• Product validation – evaluates conformance of observable behavior to requirements.

• A test-ready model contains enough information to produce test cases for the implementation automatically.

Note automated test case generation for s/w may not be practical.

CEN 5076 Class 4 - 09/26 17


• How do test practitioners use models during testing?

As with all models, we ignore the complexities that can invalidate the model and use what we can apply with ease. After all, models for testing are intended to give us insights into effective test case design – they’re intuition joggers, so it does not matter that they’re imperfect as long as the resulting tests are good. [Bezier 1990]

• Most OOA/D methodologies provide a loose graphical syntax and symbol set.

CEN 5076 Class 4 - 09/26 18


• These OOA/D methodologies are referred to as cartoons. [Binder 2000].

• Cartoons

- do not demand complete information, consistent usage, or useful integration with other facets of the model.

- Useful for sketching, refining, and documenting solutions, but are not test-ready i.e., do not have the content and consistency necessary to produce executable test cases.

CEN 5076 Class 4 - 09/26 19

Basics of Guided Inspection

A guided inspection provides a means of objectively and systematically searching a work product for faults by using explicit test cases.

Steps:

1. Define the test space.

2. Select values from the test space using a specific strategy.

3. Apply the test values to the product being tested

4. Evaluate the results and the percentage of the model covered by the test (based on some criteria).

CEN 5076 Class 4 - 09/26 20

Basics of Guided Inspection cont

Elaboration of the steps for a guided inspection:

1. Specify the scope and depth of inspection

- Scope - body of material or set of use cases

- Depth - level of detail to be covered

2. Identify basis from which the model under test (MUT) was created.

3. Develop test cases for the evaluation criteria to be applied using the contents of the basis model as input.

CEN 5076 Class 4 - 09/26 21


4. Establish criteria for measuring test coverage e.g. every class is touched by the test set.

5. Perform static analysis using the appropriate checklist.

6. “Execute” the test cases.

7. Evaluate the effectiveness of the tests using the coverage measurement.

8. If the coverage is insufficient, expand the test suite and apply the additional tests.

CEN 5076 Class 4 - 09/26 22


• In UML models, coverage is defined in terms of structural or behavioral entities e.g., classes, relationships, objects, and messages.

• Evaluation criteria attempts to answer the following questions as the MUT is inspected:

- Is the model correct?

- Is the model a complete representation of the information?

- Is the model internally consistent and consistent with the basis model?

CEN 5076 Class 4 - 09/26 23


Correctness:

• Measures accuracy a of the model.

• Analysis level – accuracy of the problem description.

• Design level – accuracy in representing the solution to the problem.

• The degree of accuracy is judge by the “oracle” (usually a human expert).

CEN 5076 Class 4 - 09/26 24


Correctness cont:

• Testing determines that the model is correct w.r.t. a test case if the result of the execution is the result that was expected (the oracle).

• Note sometimes the oracle can be incorrect.

• Domain experts are usually separated into two teams:

1. Team to construct model.

2. Team to develop test cases.

CEN 5076 Class 4 - 09/26 25


Completeness:

• Measures the inclusiveness of the model.

• Checks if any elements of the model are missing.

• Effectiveness of the completeness criteria is the quality of the test coverage.

• Model is judged complete if the results of executing the test cases can be adequately represented using only the contents of the model.

CEN 5076 Class 4 - 09/26 26


Completeness cont:

• Example: A sequence diagram may be constructed to represent a scenario. Need to check that all objects for the sequence diagram comes from classes in the class diagram (part of Rat Rose model checking).

CEN 5076 Class 4 - 09/26 27


Consistency:

• Measures if there are contradictions within the model or between the current model and the basis model.

• Testing identifies inconsistencies by finding different representations within the model for similar test cases.

• Possible area of inconsistencies:- execution of test cases on the current model and

basis model.

CEN 5076 Class 4 - 09/26 28


Consistency cont:

- During incremental development, consistency is judged locally until the current increment is integrated with the larger system.

• Checks whether there are contradictions or conflicts present either in the internal to a single diagram or between two diagrams.

• Example – a sequence diagram requires a relationship between two classes but the class diagram shows none (provided by Rat Rose).

CEN 5076 Class 4 - 09/26 29


Other criteria [Dustin]:

• Testability (or verifiability) – confirms it is possible to create a test for every req., and that an expected result is known and can be programmatically or visually verified.

• Feasibility – ensures a requirement can be implemented given the budget, schedules, technology, and other resources available.

CEN 5076 Class 4 - 09/26 30


• Necessity – verifies that every requirement in the spec is relevant to the system. Test requirements against system goals:

- Does this req. contribute to the sys goals?

- Would excluding this req. prevent the sys from meeting the goals?

- Are any other reqs. dependent on this req.?

- Some reqs. are really proposed solutions.

CEN 5076 Class 4 - 09/26 31


• Prioritization – allows everyone to understand the relative value to stakeholders of the req. Need to balance perspective of the user against the cost and technical risk associated with the proposed requirement.

• Unambiguousness – ensures the reqs. are stated in a precise and measurable way.

• Traceability – each req. is identified so that it can be associated with all parts of the system where it is used.

CEN 5076 Class 4 - 09/26 32


• Traceability cont:

- For any change in reqs. it is possible to identify all parts of the system where this change has an effect.

- Allocate reqs. to subsystems, or to sequential releases based on priority.

- Allocate reqs. to subsystems so that there is high cohesion (internal connections) and low coupling (connection among groups).

CEN 5076 Class 4 - 09/26 33

Organization of Guided Inspection

Assign inspection team e.g.,

• Domain expert – people that provide a source of “truth” (or at least the expected results).

- Define the expected system response for a specific input scenario.

• Tester – people that develop a set of test cases from the use cases.

- Often creators of the basis model.

- When at the system-wide level test case writers are often the system test team.

CEN 5076 Class 4 - 09/26 34

Organization of Guided Inspection cont

• Developer – creators of the MUT provide info not captured in the model.

- Walks the inspectors through the model, tracing actions on diagrams, showing the relationships between diagrams, and providing the actual system response at a level appropriate to the current maturity of the development.

• Moderator defines the inspection boundaries, schedules the inspection session(s), distributes materials, keeps session moving forward, completes the final report.

CEN 5076 Class 4 - 09/26 35

Preparing for the Inspection• Specifying the inspection

- Scope and depth of material to be inspected should be specified.

- Scope - specifies a set of use cases, a set of packages, or abstract classes/interfaces. Determines the starting points for scenarios, other entities are pulled in scope as required by the scenarios.

- Depth - specifies layers in aggregation hierarchies under which messages are sent. Bottom layer classes simply return values with no indication of how the values was computed.

CEN 5076 Class 4 - 09/26 36

Preparing for the Inspection• Selecting test cases for inspection

- Motivated by the criteria being addressed

Approaches include:

- Orthogonal Defect Classification (ODC) – selects test cases to identify defects by covering the different categories of system actions that trigger defects. (see P. 125 in text)

- Use profile – selects test cases to give confidence in the reliability of the product by identifying the parts of the system are used the most.

- Risk – similar to use profile but focuses on risk during development.

CEN 5076 Class 4 - 09/26 37

Preparing for the Inspection

• Creating test cases

- Test cases for a guided inspection are scenarios represented in the MUT.

- The use case template identifies three main components:

1. Use scenario – the “sunny day scenario”, most often taken path.

2. Alternative Paths – may list several scenarios that differ from the use scenario, but still represent valid executions.

3. Exceptional Paths – scenarios that result in error conditions.

CEN 5076 Class 4 - 09/26 38

Preparing for the InspectionTest cases from use cases

• Test case consists of:

1. Purpose

2. A set of preconditions

3. A stimulus (inputs)

4. Expected response

• Use the scenarios in the use case and provide specific values (??).

• Exercise the 3 sections of the use case: use scenario, alternative path, exceptional path.

CEN 5076 Class 4 - 09/26 39

Preparing for the Inspection

• Completing checklist

- Examine models for certain syntactic information that can be evaluated form the model.

- Concerned with form of the model not content.

- Go through checklist Fig. 4.12 on P. 129.

The design phase checklist focuses on:

- the analysis model and MUT

- issues internal to the MUT

CEN 5076 Class 4 - 09/26 40

The Inspection Session

• Organized in two ways depending on whether the model is automated or not.

• If prototype (or executable model) is available the testing session is an interactive session.

• No executable model symbolic execution is performed on the models i.e., the developers walk the testers through the scenarios provided by the test cases.

• Roles assigned at the inspection session: moderator, recorder, drawer.

CEN 5076 Class 4 - 09/26 41

The Inspection Session

Moderator

• Controls the session and advances the execution through the scenario.

• Keeps the session moving, i.e., session not intended to fix bugs (developers) or expand requirements (domain experts).

Recorder (usually a tester)

• Makes annotations on the reference models as the team agrees faults are found.

CEN 5076 Class 4 - 09/26 42

The Inspection SessionRecorder cont

• Makes certain faults are considered in the latter parts of the scenario.

• Maintains a list of issues that are not resolved during the testing session; maybe faults, deferrals.

Drawer

• Constructs parts of the model as scenarios are executed, used to check existing models, e.g. a sequence diagrams.

CEN 5076 Class 4 - 09/26 43

Testing Specific Types of Models

• Guided inspection technique usually stays the same form on development phase to another. Areas of change:

- Level of detail in the model becomes greater

- Amount of information increases

- Interpretation of evaluation criteria becomes more specialized

- Members of the inspection team changes

CEN 5076 Class 4 - 09/26 44

Requirements Model (Use Case Model)

• Requirements are captured by the use case model.

• Use case diagram captures relationships between use cases. Relationships include: uses (includes), extends.

• Acceptance testing often finds faults with the requirements. Typical problems include:

- Missing reqs. (incomplete requirements model)

CEN 5076 Class 4 - 09/26 45

Requirements Model (Use Case Model)

- Contradictions of reqs. (inconsistent model)

- Behavior does not agree with client expectations (incorrect model)

• Guided inspections help with finding bugs early in the development process. How?

• The act of writing test cases for the guided inspection identify problems with the reqs.

• There is no UML basis model for requirements, therefore domain documents must be examined.

CEN 5076 Class 4 - 09/26 46

Analysis Models

• Two types of analysis models: domain and application.

• Domain analysis model:

- Represents a domain of knowledge that pertains to the application to be constructed.

- Derived from the domain literature.

- Encompasses structure of domain entities and standard domain algorithms.

- Useful to partition the domain experts into two groups: developers (larger) and testers (smaller).

CEN 5076 Class 4 - 09/26 47

Analysis Models cont

• Application analysis model:

- Large projects might have several domain models.

- App model merges required pieces from several domains in large project.

- A single domain makes checking the criteria of the app model easier than an app model generated from several domains.

- Be careful not to include design information in the analysis model. How ?

CEN 5076 Class 4 - 09/26 48

Analysis Models

• Criteria for domain model inspection Fig 4.24 P. 139.

• Criteria for application analysis model inspection Fig 4.27 P. 141.

CEN 5076 Class 4 - 09/26 49

Model-Testing Checklist

Guided Inspection Process Checklist Step:

• Decide how completeness, consistency and correctness will be judged for the particular use in the model under test (MUT).

• Determine which scenarios to sample from the use case model to use the test cases.

• Create test cases by supplementing the scenarios with specific data.

• Select the model/notation that will record the results of each execution.

CEN 5076 Class 4 - 09/26 50

Model-Testing Checklist cont

Guided Inspection Process Checklist Step cont:

• Select the model/notation that will record the results of each execution.

• Conduct tests.

• Evaluate the results of the test executions to determine which tests the model passed and which it failed.

• Record the results for use in guiding the repair and testing processes in the next iteration.

CEN 5076 Class 4 - 09/26 51


• Each member of the team individually completes the checklist before the inspection meeting.

• Checklist completion:

- Trace the entities in the use cases to the class diagram in the analysis model.

- Compare the class diagram from the analysis model with the class diagram in the design model.

Go through checklist on page 112 and 129.

CEN 5076 Class 4 - 09/26 52


Project: steps for the guided inspection:

• Assign roles to team members.

• Prep for meeting

- complete checklists

- assign one use case per team member and create two test cases for that use case

- perform symbolic execution of the test cases using the use case and sequence diagram.

CEN 5076 Class 4 - 09/26 53


• During meeting

- each team member presents one test case by symbolically executing the use case and sequence diagram.

- compare the results of the individual checklist, identify any discrepancies, and make recommendations.

- Each team create a statechart diagram for one control object.

Documents

Testing Analysis and Design Models Review Class 3 Overview of Testing Analysis and Design Models Models and Testing Guided Inspection Analysis Model Design