View
220
Download
0
Category
Preview:
Citation preview
8/6/2019 Reviews Test Analyst
1/54
1
6 Reviews
6.1 Introduction
6.2 The Principles of Reviews
6.3 Types of Reviews
6.4 Introducing Reviews6.5 Success Factors for Reviews
8/6/2019 Reviews Test Analyst
2/54
2
6 Reviews6.1 Introduction
A successful review process requires planning,participation and follow-up.
Test managers have responsibilities for the planningand follow-up activities.
Testers / Test Analysts must be active participantsin the review process, providing their unique views.They should have formal review training to better
understand their respective roles in any technicalreview process.
8/6/2019 Reviews Test Analyst
3/54
3
6 Reviews
6.1 Introduction
6.2 The Principles of Reviews
6.3 Types of Reviews
6.4 Introducing Reviews6.5 Success Factors for Reviews
8/6/2019 Reviews Test Analyst
4/54
4
Reviews Learning Objectives
The Learning Objective for the video session is to: (K2) Compare review types with each other and show
their relative strengths, weaknesses and fields of use.
The concept of checklists will also be introduced but a
practical session on their use is deferred to the Liveclassroom event
(K3) Use a review checklist to verify code andarchitecture from a testers perspective
(K3) Use a review checklist to verify requirements anduse cases from a testers perspective
8/6/2019 Reviews Test Analyst
5/54
5
Quiz 6.2a6.2 The Principles of Reviews
a) To find defects
b) Not only to find defects but find defects earlier inthe life cycle
c) Not only to find defects but find defects earlier inthe life cycle and remove the causes from the
process
Which of the following best describes the benefits ofReviews?
8/6/2019 Reviews Test Analyst
6/54
6
Quiz 6.2a (suggested answer)6.2 The Principles of Reviews
a) To find defects
b) Not only to find defects but find defects earlier in the life cycle
c) Not only to find defects but find defects earlier in the life cycleand remove the causes from the process
The answer is (c).
a) Not just find defects
b) Not just find defects early but !
Which of the following best describes the benefits ofReviews?
8/6/2019 Reviews Test Analyst
7/54
7
Review6.2 The Principles of Reviews
Review: a type of static testing
an evaluation of software elements or project statusto ascertain discrepancies from planned results andto recommend improvement
(IEEE 1028 Standard for Software Reviews)
Objective:
not just to find defects, but also, to find defectsearlier in the life cycle and to remove the causesfrom the process
8/6/2019 Reviews Test Analyst
8/54
8
Why reviews?6.2 The Principles of Reviews
Objective: to find defects, but also, to find defects earlier and to
remove the causes from the process
Early defects are often the most important
defects have the characteristic to multiply themselvestop-down
cost of rework rises exponentially
60% of the defects have already been made before
coding/implementation has started
8/6/2019 Reviews Test Analyst
9/54
9
Cost Escalation Model6.2 The Principles of Reviews
Cost of fixing errors escalates as we move project towards field use
8/6/2019 Reviews Test Analyst
10/54
10
More Benefits6.2 The Principles of Reviews
Other defects (not failures!!) are found compared todynamic testing
e.g. deviations from std.s, missing requirements,
insufficient maintainability
Development productivity improvements Reduced lead-times
Reduced dynamic testing costs & time(less test cycles and re-testing)
Improved communication
8/6/2019 Reviews Test Analyst
11/54
11
Code Reviews & Dynamic Testing6.2 The Principles of Reviews
Review ofsource code (faults)
Dynamic testing(failures)
8/6/2019 Reviews Test Analyst
12/54
12
Code Reviews & Dynamic Testing6.2 The Principles of Reviews
Code reviews are more cost effective than dynamictesting
dynamic testing done well finds fewer defects thanstatic testing done well
But code reviews cannot find all the faults, e.g.:
Complier introduced errors
Processor specific errors
Environment specific problems What about Static Analysis?
8/6/2019 Reviews Test Analyst
13/54
13
IEEE Std 1028-19976.2 The Principles of Reviews
The ISTQB syllabus is broadly based on thisstandard
The syllabus identifies key principles and provides acomparative description of commonly used formalreview types including:
Management review
Audit
Walkthrough Technical review
Inspection
8/6/2019 Reviews Test Analyst
14/54
14
Basic Review Process6.2 The Principles of Reviews
PlanningKick-off
Individualpreparation
Reviewmeeting
Rework/follow-up
Metrics/sign-off
8/6/2019 Reviews Test Analyst
15/54
15
Source Documents6.2 The Principles of Reviews
Requirements & Specifications
Standards Rules /checklists
Reviews are more effectivewhen source documents areavailable
Without source documents areview is only likely to find errorsof omission and internalinconsistency in the productdocument
8/6/2019 Reviews Test Analyst
16/54
16
Checklists6.2 The Principles of Reviews
A list of questions
A negative answer identifiesan error
Interpret the rules Identify frequently made
mistakes
Must be maintained
8/6/2019 Reviews Test Analyst
17/54
17
Checklist questions6.2 The Principles of Reviews
Does this document conform to your standards andhouse-style?
Is this document clear and unambiguous to theintended readership?
Are all statements consistent with other statementsin the same or related documents?
8/6/2019 Reviews Test Analyst
18/54
8/6/2019 Reviews Test Analyst
19/54
19
Possible Review Outcomes6.2 The Principles of Reviews
Is it fit for purpose?
Possible outcomes:
Product can be used as is;
Issues need to be dealt with,but a further review is notrequired;
Issues need to be dealt withand a further review is required
8/6/2019 Reviews Test Analyst
20/54
20
Review Roles6.2 The Principles of Reviews
Leader (for the completereview process)
Author
Scribe
Reviewers
Presenter
Chair
8/6/2019 Reviews Test Analyst
21/54
21
Quiz 6.2b6.2 The Principles of Reviews
What are the benefits of reviews compared todynamic testing and to other static testingtechniques?
Five minutes to note down your ideas!
8/6/2019 Reviews Test Analyst
22/54
22
Quiz 6.2b (suggested answer)6.2 The Principles of Reviews
Benefits of reviews compared to dynamic testingand to other static testing techniques include:
Help to improve communications via the reviewmeeting participation
Help to increase our understanding of key aspects ofthe application via walkthroughs
Provide an opportunity to check the effectiveness ofproject management via management review
Provide the evidence to check process via audit
Can be very cost effective compared with dynamictesting and static analysis earlier in SDLC
Can validate requirements conformance whereasstatic analysis checks intrinsic code quality
8/6/2019 Reviews Test Analyst
23/54
23
Key Learning Points6.2 The Principles of Reviews
You should now be able to
Explain the benefits of reviews compared todynamic testing and other static testing techniques
8/6/2019 Reviews Test Analyst
24/54
24
6 Reviews
6.1 Introduction6.2 The Principles of Reviews
6.3 Types of Reviews
6.4 Introducing Reviews6.5 Success Factors for Reviews
8/6/2019 Reviews Test Analyst
25/54
25
Review Types and Objectives6.3 Types of Review
Review types (from Foundation): Informal
Walkthrough
Technical review
Inspection
Other review types include Management Reviewand Audit
Choice of review depends upon: Objectives of the review
Culture and maturity of the organization performingthe review
8/6/2019 Reviews Test Analyst
26/54
26
Learning Objectives6.3 Types of Review
Compare review types with each other and showtheir relative strengths, weaknesses and fields ofuse. (K2)
8/6/2019 Reviews Test Analyst
27/54
27
Types of Formal Review6.3 Types of Review
Formal Review types: Management review
Audit
Previously covered in Foundation
Walkthrough
Technical review
Inspection
8/6/2019 Reviews Test Analyst
28/54
28
Types of Formal Reviews6.3 Types of Review
Management review (IEEE 1028-1997):A systematic evaluation of a software acquisition,
supply, development, operation, or maintenanceprocess performed by or on behalf of management
that monitors progress, determines the status of plansand schedules, confirms requirements and their systemallocation, or evaluates the effectiveness ofmanagement approaches used to achieve fitness for
purpose.Strength: forum for monitoring and control
Weakness: may be biased by management over-control
Field of use: progress reporting, problem solving
8/6/2019 Reviews Test Analyst
29/54
29
Key Characteristics Management Review6.3 Types of Review
To monitor progress, assess status, and makedecisions Carried out by or for managers having direct responsibility for the
project or system Carried out by or for a stakeholder or decision maker, e.g. a
higher level manager or director Checks consistency with and deviations from plans, or adequacy
of management procedures Includes assessment of project risks Outcome includes action items and issues to be fixed Participants expected to prepare, decisions are documented
Note that test managers should participate in and mayinstigate management reviews of testing progress.
8/6/2019 Reviews Test Analyst
30/54
30
Types of formal reviews : Audits6.3 Types of Review
Audit: An independent examination of a software product,software process, or set of software processes to assesscompliance with specifications, standards, contractualagreements, or other criteria (IEEE 1028:1997)
Strengths: Audits can be extremely formal, and are usually
performed to demonstrate conformance to some set of expectations,most likely an applicable standard or a contractual obligation, e.g.
ISO 9000:2000 QA Audit
CMMI /TMMi
Weaknesses: Audits are the least effective at revealing defects inthe product
Field of use: Evaluate/find defects in/conformance to the process!
8/6/2019 Reviews Test Analyst
31/54
31
Key Characteristics Audit6.3 Types of Review
Main purpose: provide independent evaluation ofcompliance to processes, regulations, standards etc.
A lead auditor is responsible for the audit and acts asthe moderator
Auditors collect evidence of compliance throughinterviews, witnessing and examining documents
Outcome includes observations, recommendations,corrective actions and a pass/fail assessment
R i t W lkth h
8/6/2019 Reviews Test Analyst
32/54
32
Review type : Walkthrough6.3 Types of Review
Walkthrough (IEEE 1028-1997)A static analysis technique in which a designer orprogrammer leads members of the development team andother interested parties through a software product, and the
participants ask questions and make comments aboutpossible errors, violation of development standards, and otherproblems.
Strength: easy to do, little prep, also educational
Weakness: less evidence (review records)
Field of use: Design trade-offs and proof of concepts
R i t T h i l R i
8/6/2019 Reviews Test Analyst
33/54
33
Review type : Technical Review6.3 Types of Review
Technical review (IEEE 1028-1997)A systematic evaluation of a software product by a team of
qualified personnel that examines the suitability of thesoftware product for its intended use and identifies
discrepancies from specifications and standards. Technicalreviews may also provide recommendations of alternativesand examination of various alternatives.
Strength: Technical rigour, evidence record, discuss alternatives
Weakness: analysis paralysis! Time consuming!
Field of use: Technical solution, integrity!
Review type : Inspection
8/6/2019 Reviews Test Analyst
34/54
34
Review type : Inspection6.3 Types of Review
Inspection (IEEE 1028-1997)A visual examination of a software product to detect andidentify software anomalies, including errors and deviationsfrom standards and specifications. Inspections are peerexaminations led by impartial facilitators who are trained
in inspection techniques. Determination of remedial orinvestigative action for an anomaly is a mandatory element ofa software inspection, although the solution should not bedetermined at the inspection meeting.
Strength: proven track record, review records!Weakness: needs formal training, checklists & time
Field of use: critical systems, process/product improvement
R i Q i 6 3
8/6/2019 Reviews Test Analyst
35/54
1. Agreeing control activities to manage reduced time scales
2. Identifying defects and process improvements in a requirements document
3. Checking whether a team has conformed to a stated process flow
4. Assessing the effectiveness of metrics and reporting on a project
5. Getting feedback when time is short but you need quick ideas
6. Agreeing solutions to problems in a system architecture
7. Introducing a new business process to a group of business users
8. Checking whether a design process is a suitable choice for a team to use
35
Reviews Quiz 6.36.3 Types of Review
Suggest a suitable review type for each situation:
R i Q i 6 3 S t d A
8/6/2019 Reviews Test Analyst
36/54
36
Reviews Quiz 6.3 Suggested Answer6.3 Types of Review
1. Agreeing control activities to manage reduced time scales Managementreview
2. Identifying defects and process improvements in a requirements documentInspection
3. Checking whether a team has conformed to a stated process flow Audit4. Assessing the effectiveness of metrics and reporting on a project
Management review
5. Getting feedback when time is short but you need quick ideas Informalreview or a walkthrough
6. Agreeing solutions to problems in a system architecture Technical review7. Introducing a new business process to a group of business users
Walkthrough
8. Checking whether a design process is a suitable choice for a team to useAudit for example for compliance to a specific CMMi level
Suggest a suitable review type for each situation:
R i Q i 6 2
8/6/2019 Reviews Test Analyst
37/54
a) Planning, preparation, kick-off, meeting & follow-up
b) Kick-off, planning, preparation, meeting, rework &follow-up
c) Preparation, planning, kick-off, meeting, rework &follow-up
d) Planning, kick-off, preparation, meeting, rework &follow-up
37
Reviews Quiz 6.2c6.2 The Principals of Reviews - revision
Can you remember: in what order would thephases of a formal review usually occur?
R i Q i 6 2 S t d A
8/6/2019 Reviews Test Analyst
38/54
38
Reviews Quiz 6.2c Suggested Answer6.2 The Principals of Reviews - revision
a) Planning, preparation, kick-off, meeting & follow-up
b) Kick-off, planning, preparation, meeting, rework &follow-up
c) Preparation, planning, kick-off, meeting, rework &follow-up
d) Planning, kick-off, preparation, meeting, rework& follow-up
Can you remember: in what order would thephases of a formal review usually occur?
K L i P i t
8/6/2019 Reviews Test Analyst
39/54
39
Key Learning Points6.3 Types of Review
You should now be able to: Compare review types with each other and show
their relative strengths, weaknesses and fields ofuse. (K2)
A i t 6 4 I t d i R i
8/6/2019 Reviews Test Analyst
40/54
40
Assignment 6.4 Introducing Reviews
1. What is your current experience with reviews?2. What review types have you participated in?
3. What worked well?
4. Do you use any check lists?
if so, document examples
5. What is your typical reading rate?
6. What suggestion for improvements do you have?
7. If you have not used reviews where could you applythem?
Document your response ready for the virtual classroomsession.
8/6/2019 Reviews Test Analyst
41/54
41
6 Reviews
6.1 Introduction6.2 The Principles of Reviews
6.3 Types of Reviews
6.4 Introducing Reviews6.5 Success Factors for Reviews
Success Factors
8/6/2019 Reviews Test Analyst
42/54
42
Success Factors6.5 Success Factors for Review
Technical Factors Organizational Factors
People Factors
Success Factors Technical Factors (1/2)
8/6/2019 Reviews Test Analyst
43/54
43
Success Factors - Technical Factors (1/2)6.5 Success Factors for Review
Ensure the defined process for the review type isfollowed correctly, particularly for more formal types ofreview such as inspection
Record the costs of reviews (including time spent) andbenefits achieved
Review early drafts or partial documents to identifypatterns of defects before they are built into the wholedocument
Ensure that the documents or partial documents arereview-ready before starting a review process (i.e. applyentry criteria)
Use organization-specific checklists of common defects
Success Factors Technical Factors (2/2)
8/6/2019 Reviews Test Analyst
44/54
44
Success Factors - Technical Factors (2/2)6.5 Success Factors for Review
Use more than one type of review, depending onobjectives: document cleanup, technical improvement,transfer of information, or progress management
Review or inspect documents on which importantdecisions will be made, for example, inspect a proposal,contract or high level requirement before a managementreview authorizing major expenditure
Sample a limited subset of a document for assessment
Encourage finding the most important defects: focus oncontent not format
Continuously improve the review process!
Success Factors Organizational Factors (1/2)
8/6/2019 Reviews Test Analyst
45/54
45
Success Factors - Organizational Factors (1/2)6.5 Success Factors for Review
Ensure managers allow adequate time to be spent onreview activities, even under deadline pressure
Remember, time and budget spent are not in proportionto the defects found
Allow adequate time for rework of defects identified byreviews
Never use the metrics from reviews for individualperformance appraisal
Ensure that the right people are involved in the differenttypes of review
Provide training in reviews, particularly the more formalreview types
Success Factors Organizational Factors (2/2)
8/6/2019 Reviews Test Analyst
46/54
46
Success Factors - Organizational Factors (2/2)6.5 Success Factors for Review
Support a test analyst forum to share experience,checklists and ideas for improvement
Ensure everyone participates in reviews and everyonehas their own documents reviewed
Apply the strongest review techniques to the mostimportant documents
Ensure a well-balanced review team of people withdifferent skills and backgrounds
Support process improvement actions to addresssystemic problems
Recognize improvements gained through the reviewprocess
Success Factors People Factors
8/6/2019 Reviews Test Analyst
47/54
47
Success Factors - People Factors6.5 Success Factors for Review
Educate stakeholders to expect that defects will be foundand to allow time for rework and re-review
Ensure the review is a positive experience for the author
Welcome the identification of defects in a blame-free
atmosphere Ensure comments are constructive, helpful and
objective, not subjective
Do not review if the author does not agree or is not
willing Encourage everyone to think deeply about the most
important aspects of the documents being reviewed
Quiz 6 5a
8/6/2019 Reviews Test Analyst
48/54
48
Quiz 6.5a6.5 Success Factors for Reviews
A group of test analysts who has been on a testingcourse return enthusiastic about implementing reviewson their project.
What factors do they need to explain to their managerto support reviews being introduced successfully?
Quiz 6 5a (suggested answer)
8/6/2019 Reviews Test Analyst
49/54
49
Quiz 6.5a (suggested answer)6.5 Success Factors for Reviews
A group of test analysts who has been on a testing course returnenthusiastic about implementing reviews on their project.
What factors do they need to explain to their manager to supportreviews being introduced successfully?
Technical Factors: for example, make sure everyone agrees and
understand the review process, make sure the documents arereviewed at the right time, use the right types of review
Organizational Factors: for example, management support in timeand budget, use of metrics to measure efficiency and effectivenessof reviews (Return on Investment) but NOT measurement of people
People Factors: for example, helping the motivation of the authorand reviews by use of objective reporting of defects, helpstakeholders to understand why the reviews are helping
Quiz 6 5b - Checklists
8/6/2019 Reviews Test Analyst
50/54
50
Quiz 6.5b - Checklists6.5 Success Factors for Reviews
A requirements document has the paragraph:For all new investment products, whenever a newaccount is opened the customer record must beupdated with the new account number and investment
product code. Please note that for joint accounts bothclient records must be updated with the investmentcode.
What observations do you have and what type ofchecklist points might help you identify defects?
Quiz 6 5b Checklists (Suggested Answer)
8/6/2019 Reviews Test Analyst
51/54
51
Quiz 6.5b Checklists (Suggested Answer)6.5 Success Factors for Reviews
A requirements document has the paragraph:(ref?)Forallnew investment products, whenever a new account isopened thecustomerrecord must be updated with the new accountnumber andinvestment product code. Please note that for jointaccounts bothclientrecords must be updated with theinvestment code.
The words in red are potential defects. For example, is the investmentproduct code the same data item as the investment code? Is thecustomer record the same as the client record? Checklists would helpus identify items for consistency, completeness, accuracy, ambiguity,adherence to standards and traceability. Examples are
Consistency
is each data item uniquely and consistently named?
Traceability do the requirements have unique references so they can betracked when they change? Is there sufficient traceability to allow us totrack when requirements have been tested?
Accuracy and ambiguity the words ALL and NEVER are warning signsdo we mean that?
A i 6 5
8/6/2019 Reviews Test Analyst
52/54
52
Assignment 6.5
As a Test Analyst you should be using checklists to : Verify code and architecture from a testers
perspective
Verify requirements and use cases from a testers
perspective Thinking about the quiz you have just done,
investigate the topic of checklists for the aboveareas and produce what you believe to be
appropriate checklists Have your response ready for the Live Virtual
Classroom session
Conclusion
8/6/2019 Reviews Test Analyst
53/54
53
Conclusion6 Reviews
Reviews are an effective and efficient means toimprove the quality of the software product
requirements, specification, code, test specifications
There is more than inspection
Objective determines the review type
Lots of proof and knowledge available
JUST DO IT!
End of Chapter 6 Summary
8/6/2019 Reviews Test Analyst
54/54
End of Chapter 6 Summary6 Reviews
6.1 Introduction6.2 The Principles of Reviews
6.3 Types of Reviews (1 LO)
6.4 Introducing Reviews
6.5 Success Factors for Reviews
The other 2 LOs (K3) for use of checklists are
covered further in your assignments and in theVirtual Classroom
A table of review types is in your Reader.
ReaderChapter 6
Recommended