View
213
Download
0
Tags:
Embed Size (px)
Citation preview
PolitecnicoPolitecnicodi Milanodi Milano
Web qualityWeb quality
Luciano BaresiLuciano BaresiPolitecnico di Milano – Dipartimento di Elettronica e InformazionePolitecnico di Milano – Dipartimento di Elettronica e Informazione
Piazza L. da Vinci, 32 – 20133, Milano (Italy)Piazza L. da Vinci, 32 – 20133, Milano (Italy)[email protected]@elet.polimi.it
Web Quality - Buenos Aires (Argentina) July 21-25 2
Who am I?Who am I?
Research positionsAssociate professor at Politecnico di Milano (Italy)Visiting positions at
University of Oregon (USA)University of Paderborn (Germany)
Professional ActivitiesCo-chair of GT-VMT 01, ICECCS 02, UMICS 03PC member of several conferences (ICWE 04)
Co-founder and senior partner of INQuaS
www.elet.polimi.it/www.elet.polimi.it/~baresi~baresi
Web Quality - Buenos Aires (Argentina) July 21-25 3
Politecnico di MilanoPolitecnico di Milano
The oldest technical university in ItalyOne of the oldest in Europe
5 schools1000 faculties25,000 students
PolitecnicoPolitecnicodi Milanodi Milano
QualityQuality
Web Quality - Buenos Aires (Argentina) July 21-25 5
A definitionA definition
A methodological approach to the analysis and management of all company processesThis activity aims at
Reducing wastes Improving products and servicesImproving processes
We all know what quality is
Web Quality - Buenos Aires (Argentina) July 21-25 6
Different perspectivesDifferent perspectives
Quality is quality but different roles have different views. For example
StakeholdersThey “see” the business
UsersThey want to “use” the system
DevelopersThey must develop and maintain the system
Web Quality - Buenos Aires (Argentina) July 21-25 7
Process quality vs. product quality Process quality vs. product quality
Process qualityWe aim at introducing special-purpose characteristics to control
CostsTime-to-deliveryQualityCompetitiveness
Product qualityWe aim at products with particular quality indicators no matter of the process behind themThey must be able to match explicit and implicit requests from customers
Web Quality - Buenos Aires (Argentina) July 21-25 8
Quality & development processQuality & development process
RUP
Web Quality - Buenos Aires (Argentina) July 21-25 9
Quality dimensionsQuality dimensions
UsabilityDependability/ReliabilityCorrectnessRobustnessScalabilityPerformabilitySecurity
internal vs. external properties
PolitecnicoPolitecnicodi Milanodi Milano
Web applicationsWeb applications
(at least their architectures)(at least their architectures)
Web Quality - Buenos Aires (Argentina) July 21-25 11
Web applications vs. Web sitesWeb applications vs. Web sites
Web Quality - Buenos Aires (Argentina) July 21-25 12
Why Web applications?Why Web applications?
The Web is heterogeneous by definitionToo many technologies (Applet, Servlet, JSP, ASP, XML, …)Much more importance to presentation and communicationDistribution…
Internet
ClientFirewall
Internal network
internal users
SMTP serverWeb serverDB server
N-tier architecture
Web Quality - Buenos Aires (Argentina) July 21-25 13
HeterogeneityHeterogeneity
ComponentsHTMLScripting languagesDatabasesMultimedia contents
ExpertiseSystem designersDB administratorsDesignersProgrammersTesters
Development and maintenance are complex (more complex?)
Web Quality - Buenos Aires (Argentina) July 21-25 14
Client/ServerClient/Server
Entities that are logically distinct, but linked by a network, that work together to obtain a given task
Requests
Services
Web Quality - Buenos Aires (Argentina) July 21-25 15
Logical componentsLogical components
PresentationUser interactions
Data managementAccess, persistency, queries
Application logicBusiness oriented computations
In some cases some of these elements can be absentEmbedded systems have no presentation layers
PresentationPresentation Applicationlogic
Applicationlogic
Datamanagement
Datamanagement
Web Quality - Buenos Aires (Argentina) July 21-25 16
Client and server responsibilitiesClient and server responsibilities
PresentationPresentation
LogicLogic
PresentationPresentation
LogicLogic
Serv
er
Clie
nt
Netw
ork
PresentationPresentation
LogicLogic
PresentationPresentation PresentationPresentation
LogicLogic
LogicLogic
Data management
Data management
PresentationPresentation
LogicLogic
Datamanagement
Datamanagement
A B C D E
Data management
Data management
Data management
Data management
Data management
Data management
Data management
Data management
Web Quality - Buenos Aires (Argentina) July 21-25 17
Client and server responsibilitiesClient and server responsibilities
Distributed presentationThe server owns all the knowledgeThe client only interacts with the server
Example: HTML form (no control on the quality of data)Remote presentation
Presentation is fully in charge of the clientDistributed logic
The logic is partly on the server and partly on the clientRemote data access
Presentation and logic are on the client that interacts with a server to access data (SQL interface)
Distributed databaseThe functionality to manage data are partially on the client and partially on the server (e.g., Distributed Relational Database Architecture di IBM)
Web Quality - Buenos Aires (Argentina) July 21-25 18
Fat client vs. Fat serverFat client vs. Fat server
Clients are fat if the implement part of the application logic (C, D, E)Why fat clients
The system can handle user inputs more quicklyThe system interacts with the user with a finer granularityDifferent clients can implement interfaces that are specific to different usersThe system is more scalable
Web Quality - Buenos Aires (Argentina) July 21-25 19
Fat client vs. Fat serverFat client vs. Fat server
Why not fat clientsInteractions with the server can become too frequent during complex computations
Servers could better control data access if they control the computation
We loose data encapsulationThe client must know in detail how data are organized on the server
System management and maintenance become more complex
Clients and server must be both updated It would be easier if we could update only the server
Web Quality - Buenos Aires (Argentina) July 21-25 20
Three-tier architecturesThree-tier architectures
In the ideal model the mid component contains all the application logic
In many real cases the logic is also spread on client and server
This way we can distinguish between presentation and application logic
Different distributed data sourcesDifferent client types
PresentationPresentation Applicationlogic
Applicationlogic
Datamanagement
Datamanagement
ServerServerClientClient
Intermediatecomponent
Intermediatecomponent
Web Quality - Buenos Aires (Argentina) July 21-25 21
Another exampleAnother example
Client
Shared Application Services
DataSources
Order Entry
Customer Service
Scheduling
Credit Check
Customer Information
Distribution Service
Inventory Manager
Credit Service
CustomersDistribution Inventory
Web Quality - Buenos Aires (Argentina) July 21-25 22
From two-tier to n-tierFrom two-tier to n-tier
PresentationPresentation Applicationlogic
Applicationlogic
Datamanagement
Datamanagement
ServerServerClientClient Intermediate
component
Intermediatecomponent
ServerServerClientClient
2-tier
n-tier
Intermediatecomponent
Intermediatecomponent Intermediate
component
Intermediatecomponent
Web Quality - Buenos Aires (Argentina) July 21-25 23
Computational modelsComputational models
Computational modelComputational model
Host/terminalmodel
Host/terminalmodel
Models based onDistributed computation
Models based onDistributed computation
Client/serverClient/server
Models basedon events
Models basedon events
Models based onfile transfer
Models based onfile transfer Peer-to-PeerPeer-to-Peer
Object-OrientedObject-
Oriented
PolitecnicoPolitecnicodi Milanodi Milano
What can we do?What can we do?
Web Quality - Buenos Aires (Argentina) July 21-25 25
Basically, we canBasically, we can
MeasureWe “understand” the quality of our applications by measuring itSeveral different measures
AnalyzeWe “understand” the quality by studying the artifacts we haveInteresting, but complex
TestWe can “understand” the quality by trying with some special executions
No magic solutions !!!
PolitecnicoPolitecnicodi Milanodi Milano
Design for testabilityDesign for testability
Web Quality - Buenos Aires (Argentina) July 21-25 27
TestabilityTestability
proc foo ( ) x: integer; y: char; begin xxlskd ;
xxl;
Definition ofanalysis models
Direct control over source code(usually not feasible)
? P
Interestingproperties
Algorithm control ofproperty P’
? P´
Modelproperty
Implication
Web Quality - Buenos Aires (Argentina) July 21-25 28
Important factorsImportant factors
Systems can be tested more easily if they:Work decently (operability)Can be controlled (controllability)Can be observed (observability)Do not add useless complexity (simplicity)Are documented consistently and completely (understandability)Are well-known (suitability)Are stable (stability)
Web Quality - Buenos Aires (Argentina) July 21-25 29
Five dimensionsFive dimensions
RequirementsWe must consider the way we represent them
DesignWe should anticipate as many constraints as we can
ImplementationTest oracles should be “added” during the implementation
Test We should identify test cases as soon as we can
DocumentationBetter testability if we have good documentation
Web Quality - Buenos Aires (Argentina) July 21-25 30
Some very preliminary commentsSome very preliminary comments
Design projects as simple as possibleNo added (useless) complexity
Use/add contracts (assertions) to our componentsMaximize the visibility of all productsConsider comments and documentationWrite “standard” codePrivilege formal (semi-formal) notations instead of informal models
Is this true with the Web?
Web Quality - Buenos Aires (Argentina) July 21-25 31
nowebnoweb
PolitecnicoPolitecnicodi Milanodi Milano
Test and analysisTest and analysis
Web Quality - Buenos Aires (Argentina) July 21-25 33
Why test and analysisWhy test and analysis
Software is never correctNo matter of the domainNo matter of the techniques we use
Any software must be verified/validatedTest and analysis are
Important to control and assess the quality of products
But impact on the processUsually expensiveDifficult, but interesting
Good compromises
Web Quality - Buenos Aires (Argentina) July 21-25 34
IEEE terminologyIEEE terminology
ErrorWhat causes the problem (deviation) between the product and the ideal programErrors and faults are not consistentFor example, typos, cut&paste, wrong requirements
FaultProgram elements that do not correspond to the expectations Faults do not respect locality and are not consistent with failuresFor example, the program has a multiply operator instead of a sum operator
FailureBehavior not consistent with system specificationFor example, 4 + 3 = 12
Web Quality - Buenos Aires (Argentina) July 21-25 35
PropertiesProperties
Process oriented (internal) propertiesReusabilityMaintainabilityModularity
External properties that can be verifiedInteroperabilityTimeliness
External properties that cannot be verifiedUser-friendlinessUsability
Dependability propertiesCorrectnessRobustnessSafetyReliability
Web Quality - Buenos Aires (Argentina) July 21-25 36
DependabilityDependability
Robust
Safe
Robust but not safe: we can have catastrophic errors
Correct, but not safe or robust: the specification is not enough
Reliable
Reliable, but not correct: We can seldom have failures
Correct
Safe, but not correct: We can have “light” failures
Web Quality - Buenos Aires (Argentina) July 21-25 37
Validation and verificationValidation and verification
Requirements
Formal description
system
validation verificationInclude usability test, user feedback
Includes test, inspection, static analysisbuilding the right systembuilding the system right
Web Quality - Buenos Aires (Argentina) July 21-25 38
Validation vs. VerificationValidation vs. Verification
If we say that the page must display quickly, we cannot verify the property, but we can validate itIf we say that the page must display in 30 seconds, we can verify it.
Web Quality - Buenos Aires (Argentina) July 21-25 39
The real problemThe real problem
Correctness properties are undecidable
Decisionprocedure
property
program
yes/no
Web Quality - Buenos Aires (Argentina) July 21-25 40
What do they offer?What do they offer?
We must settle for some kind of inaccuracyto be able to deal with the problem
Perfect verification
Optimistic approximation(testing)
Pessimistic approximation(analysis, proof)
Simplified properties
Sample theinput space
Web Quality - Buenos Aires (Argentina) July 21-25 41
Impact of the software typeImpact of the software type
The software type and its characteristics impact test and analysis activities in different ways
Different emphasis on the same propertyTimeliness“Correctness”
Different propertiesUsabilityUser-friendliness
New techniquesPresentationNavigation
Web Quality - Buenos Aires (Argentina) July 21-25 42
PrinciplesPrinciples
The basic principles areSensitivity: it is better to fail every time than sometimeRedundancy: we should make our intentions explicitPartitioning: divide et imperaRestriction: we should try to reduce the scopeFeedback: we should use the experience to improve the process
Web Quality - Buenos Aires (Argentina) July 21-25 43
SensitivitySensitivity
Better to fail every time than sometimesConsistency helps:
A test selection criterion is better if any selected test gives the same results, that is, if the program fails with a given test, it should fail with all tests selected with that criterionFor example, deadlock analysis at run-time is better if it does not depend on the machine, that is, if the program fails on a machine within a given execution, then it should fail on all machines within that execution
Web Quality - Buenos Aires (Argentina) July 21-25 44
RedundancyRedundancy
Make decisions explicitRedundant control can increase the capability of capturing errors in advance or more efficiently
Static type check is redundant with respect to testing, but it can solve many problems in advanceThe validation of requirements is redundant with respect to the validation of the final product, but it can solve several problems in advance and more efficientlyTest and model checking are redundant, but they are used together in some cases to increase the confidence on the right behavior of the product
Web Quality - Buenos Aires (Argentina) July 21-25 45
PartitionPartition
Divide et imperaDifficult problems can be treated by partitioning the input space
Criteria for both functional and structural test selection identify meaningful partitions of the input spaceVerification techniques partition the input space by grouping data that are homogeneous with respect to the properties that we want to prove
Web Quality - Buenos Aires (Argentina) July 21-25 46
RestrictionRestriction
Simplify the problemClever restrictions can make problems that are difficult (and undecidable) simple and tractable
In some cases a weaker property can be easier to verify
For example, we cannot demonstrate that pointers are used in the right way, but if we use Java we can impose it easily
In other cases, an heavier property can be easier to verify
For example, in general we cannot demonstrate that we do not have type errors with languages that have a dynamic type system, but we can demonstrate it if the language is statically typed.
Web Quality - Buenos Aires (Argentina) July 21-25 47
FeedbackFeedback
Fine tune the development processLearn from the experience:
Checklists are built on errors discovered in the pastThe way errors are classified can help define meaningful criteria to select test casesThe mechanisms to revise the process are based on the fact that we must improve the process to improve the product
Web Quality - Buenos Aires (Argentina) July 21-25 48
Test and analysis in a development processTest and analysis in a development process
Activities related to quality control and types of development processesDegrees of freedom and compromises
How to balance budget, risks, and qualityError analysis and feedbackImpact of the development process on test activitiesResponsibilities of a test group
Web Quality - Buenos Aires (Argentina) July 21-25 49
Course outlineCourse outline
UsabilityAccessibility
Functional correctnessTesting and analysis techniques
Robustness and scalabilityPerformanceSecurityTest processToolsConclusions
PolitecnicoPolitecnicodi Milanodi Milano
UsabilityUsability
Web Quality - Buenos Aires (Argentina) July 21-25 51
UsabilityUsability
It is the measure of how a software system satisfies the needs of its users
Ease of useEfficacy and efficiencyEase of storing produced artifactsLow number of errors and ease to recoverSatisfaction while using the product
Jakob Nielsen
Web Quality - Buenos Aires (Argentina) July 21-25 52
Some design principlesSome design principles
Design of pages based on the deviceCorrect visualization with different browsersCorrect visualization no matter of the screen characteristicsLight-to-load pages
Ease to access supplied contentsStructure of contents to facilitate their readingCoherence in the page styleVisibility of links and their meaningConsistency of the presentation style with respect to users
Ease of navigation
Web Quality - Buenos Aires (Argentina) July 21-25 53
Page design and devicePage design and device
Web Quality - Buenos Aires (Argentina) July 21-25 54
Correct visualizations and browsersCorrect visualizations and browsers
The semantics of tags can change in different browsersThe design can
Use the minimum common set, trying not to use the last version of the languageUse the latest version
To stimulate users to updateMany users may not to be able to access some contents
We should test our applications with all well-known browsersIf we knew the users, we could better understand what we need
Specific classes of usersSkilled users
We can start developing the application with old versions of the languages and
Add the extensions laterAll main contents must be published using such a format that can be accessed by all users
Web Quality - Buenos Aires (Argentina) July 21-25 55
Screen-independent designScreen-independent design
800x600 is the standard resolutionMany displays are already set to this resolutionIf we used higher resolutions, the visualization would use horizontal sliding bars
Vertical bars are acceptable, but we should avoid horizontal ones
We could use relative dimensions (percentages)The visualization adapts to the current resolution and the current screen sizeBut we must be sure that all visualizations are meaningful
Web Quality - Buenos Aires (Argentina) July 21-25 56
Design light pagesDesign light pages
Some studies have revealed that0.1 second is the limit to give the user the perception that the system is reacting
We do not need any message, but we need to start displaying the result
1 second is the limit within which any human thought is not interrupted
The user perceives the delay, but it is still acceptable
10 seconds is the limit to keep the user attention on the dialogue
If delays are higher, the user changes his focusIf we deliver pages in more than 10 seconds, this means that we loose the client
Web Quality - Buenos Aires (Argentina) July 21-25 57
Possible problems and solutionsPossible problems and solutions
The speed can be influenced by the number and size of images
Light and few imagesMultimedia effects used only if they are really helpful
They improve the way information is perceivedNo pages should be loaded in more than 20 secondsWe should check page loading in the different situations
Maybe even with different cache settingsFirst screen at a glance
We should reduce the time needed to load the first pageThis page (the first part) should provide as much information as possible (more text than images)
Web Quality - Buenos Aires (Argentina) July 21-25 58
Neat and clear informationNeat and clear information
We should use some 50% less text than “standard” newspapers
When we read on a screen, we are 25% slowerUsers do not love to scroll windows
Contents should be well-structured: titles, paragraphs, and itemized lists are useful to locate contents
Rule of the pyramid (taken from journalism)Each page should start with a short conclusion (summary) and then present all details
Web Quality - Buenos Aires (Argentina) July 21-25 59
Hypertextual structureHypertextual structure
We should use hypertext to split long text in pagesThe hypertext should not be used to break linear informationEach information peace should focus on a specific and well-defined argument
We should be able to identify the entities that are more relevant and then identify their subcomponents
We should then identify the relationships that exist between entities and between subcomponents within the same entity
Web Quality - Buenos Aires (Argentina) July 21-25 60
Homogeneous visualization styleHomogeneous visualization style
We should choose coherence and uniformityWe should plan the themes, information structures, and navigation paths that contribute to the uniformity of the siteWe should define a visual schema that should be adopted in all pages
Same fonts, colors, rendering of contents to communicate the adopted schema and facilitate the comprehension to users
We should avoid to change the styleLinear transitions and uniformity give the user the idea of being in the same site (known borders)Better ease of use and learning since users can find well-known models
Web Quality - Buenos Aires (Argentina) July 21-25 61
An example (I)An example (I)
Web Quality - Buenos Aires (Argentina) July 21-25 62
An example (II)An example (II)
Web Quality - Buenos Aires (Argentina) July 21-25 63
Linear transitionsLinear transitions
Same backgroundSame fontsSame grid to partition the pageSame positioning of the different page elementsSame use of empty spaces
Example: national gallery of Washington (www.nga.gov)
Web Quality - Buenos Aires (Argentina) July 21-25 64
Facilitate access to contentsFacilitate access to contents
Ease the interactionIdentification of links and their meaning
Link typesStructural links: They define the structure of the information space and allow the user to move within itAssociation links: Textual links that let the user deeper the knowledge of the text used as anchorLists of alternative pointers: They can help users find what they need
Starting rhetoricThe user must be able to identify the added value that he will find in the target page
Ending rhetoricThe target page must clarify the ending context and give value to the source page
Web Quality - Buenos Aires (Argentina) July 21-25 65
Descrivere i link testualiDescrivere i link testuali
We should not use long anchorsThe must only be pointersToo many words do not allow us to identify the meaning of the link (max 4 words)We should use simple words that clearly identify the target
Examples– To know about my job click here– Here is my job
We can add text to supply further informationIt can be used to differentiate similar links
Web Quality - Buenos Aires (Argentina) July 21-25 66
Suitable presentation stylesSuitable presentation styles
The visual style must give the right impression that we want to communicateWe should choose the right styleFor example
http://www.nasa.govhttp://kids.msfc.nasa.gov
Web Quality - Buenos Aires (Argentina) July 21-25 67
Ease of navigationEase of navigation
Where am I? Where was I? Where can I go?We should always define the page titleBreadcrumbs link identify the path in the site followed to reach a given pageA link to the home page (or to the starting pages of the different sections) from each page help the user to understand where he is We should be careful with the links defined so far
Web Quality - Buenos Aires (Argentina) July 21-25 68
Site organizationSite organization
Each application must have a home page It provides a summary and links to some important pages
A Web site is the set of pages that are linked to the home page through links
The should supply an homogeneous set of information
HP
Web Quality - Buenos Aires (Argentina) July 21-25 69
Linear structureLinear structure
It guides the user along a pathUseful when the presentation implies that the user follow a predefined path
examples: lessons, book chapters, collections, etc.
User navigate forward and backward by following a predefined pathEach page has a link to the initial page
HP
Argument Argument Argument…
SubArgument1 SubArgument2
Web Quality - Buenos Aires (Argentina) July 21-25 70
Network structureNetwork structure
We need a navigation bar to guide the user
Typical in many simple applications (personal site)Links to and from any page in the application
Home Page
Link1Link2Link3....
LinkN
HP
Web Quality - Buenos Aires (Argentina) July 21-25 71
Hierarchical structure Hierarchical structure
Areas identify and organize contentsIn each page we can have links to the home page and to the other pages of the area
Home Page
Sub areapage
Contentspage
…Area page Area page Area page
Sub areapage
Contentspage
Contentspage
Contentspage
Contentspage
Contentspage
Contentspage
PolitecnicoPolitecnicodi Milanodi Milano
Usability testUsability test
Web Quality - Buenos Aires (Argentina) July 21-25 73
A simple exampleA simple example
Information density is inverse proportional to the search time and the capability of finding what we were searching forA useful page should not contain an excessive number of non homogenous dataWe have many pages, even professional ones, that are not useful according to this criterion
Web Quality - Buenos Aires (Argentina) July 21-25 74
Usability testUsability test
Test with sample users to study their interaction with the system
The human participation is basilarTest of HTML code to verify they are compliant with the W3C guidelines
It can be easily made in an automatic wayThey are complementary techniques
Web Quality - Buenos Aires (Argentina) July 21-25 75
Usability testUsability test
The tests with sample users apply techniques to collect qualitative and quantitative data while they interact with the product to stress particular featuresWe have two main approaches with sample users
The use of formally defined tests to validate or invalidate particular hypotheses (not so used)The use of iterative tests to gradually expose usability problems
The goals areIdentification and correction of usability problems before releasing the applicationFacilitate the creation of products that
Are easy to learn and useSatisfy user needsSupply useful functionality to the set of target users
Create a repository of tests to be used for future releases
Web Quality - Buenos Aires (Argentina) July 21-25 76
Non technical goalsNon technical goals
Minimize the costs to support product maintenanceIncrease the number of sold productsAcquire a competitive advantage with respect to competitors
Usability is a key feature for many productsMinimize risks before the release
Web Quality - Buenos Aires (Argentina) July 21-25 77
Types of usability testsTypes of usability tests
Analysis of user
requirementsRequirementsspecification
Preliminary design
Detailed design
Implementation
Explorative tests
Evaluation tests
Validation tests
Comparison tests
Web Quality - Buenos Aires (Argentina) July 21-25 78
Explorative tests (I)Explorative tests (I)
WhenAt the beginning of the development processWhen we know the user profile and how the system will be used, but we are working on the functional specificationsBefore working on the detailed project
GoalsUnderstand the validity of the preliminary design Verify the conceptual idea that the user has of the product (high abstraction level)Verify the hypothesis on the user
Web Quality - Buenos Aires (Argentina) July 21-25 79
Explorative tests (II)Explorative tests (II)
MethodologyWe use product prototypes to make them be evaluated by representative usersAt the beginning, we can also use static interfaces, maybe even on whiteboards (story boards)We must highly interact with participants to
Verify the efficacy of the concepts on which the preliminary project is basedHelp fill the gap as to not-yet-implemented functionality
Web Quality - Buenos Aires (Argentina) July 21-25 80
Evaluation tests (I)Evaluation tests (I)
It is the most widely used usability testWhen
At the beginning or during the development cycleAfter the definition of the preliminary project
GoalsWe want to extend the results with explorative tests to evaluate the usability of low level operationsWhile explorative tests work on the skeleton of my product, these tests consider all characteristicsWe do not want to evaluate how intuitive the product is, but how a user can complete realistic tasks and maybe identify possible problems
Web Quality - Buenos Aires (Argentina) July 21-25 81
Evaluation tests (II)Evaluation tests (II)
Methodology:The user always works on tasks, instead of surfing around and making commentsWe not want to understand mental processes, but we consider the actual behaviorWe collect quantitative measures
Web Quality - Buenos Aires (Argentina) July 21-25 82
Validation tests (I)Validation tests (I)
Called also verification test When
Late in the development cycleIt is used to certify the usability of the productBefore releasing the product
GoalsWe want to compare the product against predefined standards or standard used by competitorsWe want our product to comply with standards before releasing it
If it is not compliant with want to understand whyThese standards are defined at the beginning of the development cycle along with the usability goals
Web Quality - Buenos Aires (Argentina) July 21-25 83
Validation tests (II)Validation tests (II)
Usability goalsThey are defined by thinking of
Usability tests on previous versionsMarket analysisInterviews with users
They are characterized by means ofCriteria to measure performance (speed, user accuracy while working on a task)Criteria for user preferences
Web Quality - Buenos Aires (Argentina) July 21-25 84
Validation test (III)Validation test (III)
MethodologySimilar to evaluation testBefore testing, we need to identify the reference standardsWe must define also the tolerance with want to use to accept results (e.g., % of failures)We propose specific tasks to participantsThey do not interact with the test monitorWe collect quantitative data
Web Quality - Buenos Aires (Argentina) July 21-25 85
Comparison tests (I)Comparison tests (I)
WhenIt is not associated with any specific step in the development processIn the first phases it can be used to compare different alternatives through exploratory testsIt can be used to access the efficacy of a single componentIt can be used to compare the product with what developed by competitors
GoalsIt can be associated with any of the other testing methodsIt can be used to understand pros and cons of different projectsIt can be used to understand the effectiveness of different designs
Web Quality - Buenos Aires (Argentina) July 21-25 86
Comparison tests (II)Comparison tests (II)
MethodologyWe propose the vis-a-vis comparison between two or more alternativesFor each alternative, we collect information and observations on performance and preferencesThe format we use depends on what the want to getIn many cases we discover that some alternatives can be the winning ones The best results come when we compare projects that are radically different (instead of just similar)
Web Quality - Buenos Aires (Argentina) July 21-25 87
The phases of usability testThe phases of usability test
Definition of test planSelection and recruiting of participantsPreparation of test materialExecuting the testDebriefingConclusions and recommendations
Web Quality - Buenos Aires (Argentina) July 21-25 88
Definition of test planDefinition of test plan
Objectives: We want to describe the reasons why we want to test the application
For example: We have user feedback on some particular problems with the application
Test objectives: We must describe the test questions in a clear and neat way
For example: is the on-line help easier through hot-keys or the mouse?Is the on-line help enough and self-contained?Is the navigation good enough to allow the user to always know where he is?Is the product usable?
This is wrong because it is too vague
Web Quality - Buenos Aires (Argentina) July 21-25 89
Test plan: how to design testsTest plan: how to design tests
We must define the test procedure by identifying the steps that testers should follow and the “tools” they should useWe must decide the distribution of participants with respect to
Experience, background, age, sex, ..Execution order of tasks
They need a reference guide for all participants (test monitor and external observers)Different test monitors can manage the same test in equivalent ways
Web Quality - Buenos Aires (Argentina) July 21-25 90
Test plan: final reportTest plan: final report
It must identify all elements that are significant to evaluate the application and that must be collectedFor example
For each task of group of themAssigned time framePercentage of participants that have completed their task successfullyPercentage of participants that have done some mistakesPercentage of participants that have not been able to complete their task
Web Quality - Buenos Aires (Argentina) July 21-25 91
Selection and recruiting of Selection and recruiting of participantsparticipants
Identification of classes of possible usersFor example: professors, expert students, novice students
Definition of the optimum sample (10-12 participants)Some participants from each class according to the frequencies in the real worldFor example
3 professors, 1 professor of computer science5 students that are used to navigate in Internet2 novice students
Definition of the minimal sample (4-5 participants)Identify the minimum number for each class
1 professor, better if not expert2 expert students1 novice student
Web Quality - Buenos Aires (Argentina) July 21-25 92
Preparation of test materialPreparation of test material
Screening questionnaire to select participantsQuestionnaire to study their backgroundTools to collect data
What dataPerformance: measures on the behavior of the application (e.g., number of errors, mean time to answer)Preferences: opinions on the product (e.g., preferences between two possible versions)
How to classify themDeclarations to comply with privacy rulesPre-testPost-test
Web Quality - Buenos Aires (Argentina) July 21-25 93
Test executionTest execution
IntroductionPresentation of the system and testFilling of a questionnaire to know the participants
Identification of goalsLet the participants know what we want with the test
We want to evaluate the system not the participantWe should be careful with cameras and microphones
ExecutionEach participant must have a list of tasks that must be completed in a given time frameThe test monitor oversee the process, but does not influence it
Web Quality - Buenos Aires (Argentina) July 21-25 94
DebriefingDebriefing
We interview participants on what they did during the testIt is useful to understand the reasons of discovered problems and how to solve themWe should not judge participants and try to score themWe must try to convince participants to tell their comments
Web Quality - Buenos Aires (Argentina) July 21-25 95
Conclusions and recommendationsConclusions and recommendations
We collect and summarize data (mean values and accuracy)We analyze data to:
Identify those tests that did not match quality criteriaIdentify the origin/nature of the problemsDiscover new problemsClassify priorities: Criticality = Severity + Probability
Severity (to be defined with the development team):
– Unusable– Hard– Mild– Not important
Web Quality - Buenos Aires (Argentina) July 21-25 96
Analysis of resultsAnalysis of results
After executing tests, the test monitor analysis results based on questioners and discussions with participantsHe produces a report to identify problems and limits of the system (as to usability)He can define new requirementsWho is responsible for the system can decide that the system must be reworkedThe test monitor decides if and what tests must be re-executed
PolitecnicoPolitecnicodi Milanodi Milano
AccessibilityAccessibility
Web Quality - Buenos Aires (Argentina) July 21-25 98
W3C WAIW3C WAI
The WAI – Web Accessibility Initiative defines 14 steps that must be followed to make a Web application accessibleEach step is associated with a priority level based on the impact it has on the accessibilityAccording to the priority, we have 3 certification levels (A, AA, AAA)
A : This means that the application complies with all MUST stepsAA : This means that the application complies with all MUST and SHOULD stepsAAA : This means that the application complies with all MUST, SHOULD, LIKE TO steps
Web Quality - Buenos Aires (Argentina) July 21-25 99
PrioritiesPriorities
[Priority 1] A Web content developer must satisfy this checkpoint. Otherwise, one or more groups will find it impossible to access information in the document. Satisfying this checkpoint is a basic requirement for some groups to be able to use Web documents.
[Priority 2] A Web content developer should satisfy this checkpoint. Otherwise, one or more groups will find it difficult to access information in the document. Satisfying this checkpoint will remove significant barriers to accessing Web documents.
[Priority 3] A Web content developer may address this checkpoint. Otherwise, one or more groups will find it somewhat difficult to access information in the document. Satisfying this checkpoint will improve access to Web documents.
Web Quality - Buenos Aires (Argentina) July 21-25 100
ExampleExample
In General (Priority 1) 1.1 Provide a text equivalent for every non-text element (e.g., via "alt",
"longdesc", or in element content). This includes: images, graphical representations of text (including symbols), image map regions, animations (e.g., animated GIFs), applets and programmatic objects, ascii art, frames, scripts, images used as list bullets, spacers, graphical buttons, sounds (played with or without user interaction), stand-alone audio files, audio tracks of video, and video.
2.1 Ensure that all information conveyed with color is also available without color, for example from context or markup.
4.1 Clearly identify changes in the natural language of a document's text and any text equivalents (e.g., captions).
6.1 Organize documents so they may be read without style sheets. For example, when an HTML document is rendered without associated style sheets, it must still be possible to read the document.
6.2 Ensure that equivalents for dynamic content are updated when the dynamic content changes.
7.1 Until user agents allow users to control flickering, avoid causing the screen to flicker.
14.1 Use the clearest and simplest language appropriate for a site's content. And if you use images and image maps (Priority 1)
1.2 Provide redundant text links for each active region of a server-side image map.
9.1 Provide client-side image maps instead of server-side image maps except where the regions cannot be defined with an available geometric shape.
PolitecnicoPolitecnicodi Milanodi Milano
Functional correctnessFunctional correctness
Traditional testingTraditional testing
Web Quality - Buenos Aires (Argentina) July 21-25 102
Granularity levelsGranularity levels
Acceptance testing: the software behavior is compared with end user requirementsSystem testing: the software behavior is compared with the requirements specificationsIntegration testing: checking the behavior of module cooperation.Unit testing: checking the behavior of single modulesRegression testing: to check the behavior of new releases
Web Quality - Buenos Aires (Argentina) July 21-25 103
The test case generation problemThe test case generation problem
How to generate test data Partition testing: divide program in (quasi-) equivalence classesrandomfunctional (black box)
based on specificationsstructural (white box)
based on codefault based
based on classes of faults
Web Quality - Buenos Aires (Argentina) July 21-25 104
White vs black boxWhite vs black box
Black boxit depends on the specification notationit scales up(different techniques at different granularity levels)it cannot reveal code bases testing (same specification implemented with different modules)
White boxit is based on control or data flow coverageit does not scale up (mostly applicable at unit and integration testing level)it cannot reveal missing path errors(part of the specification that is not implemented)
Web Quality - Buenos Aires (Argentina) July 21-25 105
Specification-based TestingSpecification-based Testing
From formal specificationscan be automatedEXAMPLES: Test case generation from
Algebraic specificationsFinite state automata (UML class diagrams)Grammars
From semi-formal specificationspartitions can be easily identifiedcan be partially automated
Web Quality - Buenos Aires (Argentina) July 21-25 106
Test-case Generation from Informal Test-case Generation from Informal Specifications (Natural Language)Specifications (Natural Language)
cannot be automatedsome structure (e.g., organization standards) can helpguidelines to increase confidence level and reduce discretionality: at least on test case for each:
subsets of “valid” homogeneous data“non valid” (combination of) databoundary dataspecific data (treated independently, error prone,...)
Web Quality - Buenos Aires (Argentina) July 21-25 107
Fault-based TestingFault-based Testing
Identify a set of program locations (related to specific faults)generate alternate programs by seeding faults in the original program in the identified locationsgenerate test cases to estimate adequacy in detecting real faults from adequacy in detecting seeded faults
Web Quality - Buenos Aires (Argentina) July 21-25 108
Partition TestingPartition Testing
Basic idea: Divide program input space into (quasi-) equivalence classes
Underlying idea of specification-based, structural, and fault-based testing
Web Quality - Buenos Aires (Argentina) July 21-25 109
The Category-Partition Method The Category-Partition Method
STEP 1: Analyze the specification: Identify individual functional units that can be tested separately. For each unit identify:
parameters and characteristicsenvironment and characteristics
classify units into categoriesSTEP 2: Partition the categories into choicesSTEP 3: Determine constraints among the choicesSTEP 4: Write tests and documentation
Web Quality - Buenos Aires (Argentina) July 21-25 110
The Category-Partition Method: The Category-Partition Method: an examplean example
......... *Command:
findSyntax:
find <pattern> <file>Function:
The find command is used to locate one or more instances of a given pattern in a file. All lines in the file that contain the pattern are written to standard output. A line containing the pattern is written only once, regardless of the number of times the pattern occurs in it.The pattern is any sequence of characters whose length does not exceed the maximum length of a line in the file. To include a blank in the pattern, the entire pattern must be enclosed in quotes (“). To include a quotation mark in the pattern, two quotes in a row (““) must be used.
...........
* From Ostrand, Balcer, The Category-Partition Method for Specifying and Generating Functional Tests
Web Quality - Buenos Aires (Argentina) July 21-25 111
Step A - analyze the specification: Step A - analyze the specification: identify categoriesidentify categories
find is an individual function that can be tested separatelyparameters: pattern, filecharacteristics (pattern)
explicit (immediately derivable from specs):pattern lengthpattern enclosed in quotespattern contains blankspattern contains enclosed quotes
implicit (“hidden” in specs):quoted patterns with/without blanksseveral successive quotes included in the pattern........
Web Quality - Buenos Aires (Argentina) July 21-25 112
Step B - partition categoriesStep B - partition categories
Parameters:Pattern size:
emptysingle charactermany characterslonger than any line in the
fileQuoting:
pattern is quotedpattern is not quotedpattern is improperly
quotedEmbedded blanks:
none oneseveral
Parameters (cont.....)Embedded quotes: none one severalFile name:
....Environment:
Number of occurrences of pattern in a file:
noneoneseveral
Pattern occurrences on target line:
....
Web Quality - Buenos Aires (Argentina) July 21-25 113
Step C: Determine ConstraintsStep C: Determine Constraints
Parameters:Pattern size:
empty [property Empty]single character [property NonEmpty]many characters [property NonEmpty]longer than any line in the file [single]
Quoting:pattern is quoted [property Quoted]pattern is not quoted [if NonEmpty]pattern is improperly quoted [error]
........
Environment:Number of occurrence of pattern in a file:
none [if NonEmpty] [single]one [if NonEmpty] [property Match]
.....
Web Quality - Buenos Aires (Argentina) July 21-25 114
Some Considerations onSome Considerations onthe Category Partition Methodthe Category Partition Method
a practical implementation of general principles:partition testingboundary testingerroneous conditions
other approaches with similar goals, but different procedures:
condition tablescause effect graphsequivalence partitioning
Web Quality - Buenos Aires (Argentina) July 21-25 115
Structural Coverage TestingStructural Coverage Testing
(In)adequacy criteria If significant parts of program structure are not tested, testing is surely inadequate
Control flow coverage criteriaStatement (node, basic block) coverageBranch (edge) coverageCondition coveragePath coverageData flow (syntactic dependency) coverage
Attempted compromise between the impossible and the inadequate
Web Quality - Buenos Aires (Argentina) July 21-25 116
Statement CoverageStatement Coverage
One test datum (N=1, A[0]=-7, X=9) is enough to guarantee statement coverage of function selectFaults in handling positive values of A[i] would not be revealed
int select(int A[], int N, int X) {
int i=0;while (i<N and A[i] <X) {
if (A[i]<0) A[i] = - A[i];
i++;}return(1);
} i++
i<N and A[i] <X
A[i]<0
A[i] = - A[i];
return(1)
TrueFalse
TrueFalse
i=0
Web Quality - Buenos Aires (Argentina) July 21-25 117
Branch CoverageBranch Coverage
i=0
i<N and A[i] <X
A[i]<0
A[i] = - A[i];return(1)
TrueFalse
TrueFalse
We must add a test datum (N=1, A[0]=7, X=9) to cover branch False of the if statement. Faults in handling positive values of A[i] would be revealed. Faults in exiting the loop with condition A[i] <X would not be revealed
int select(int A[], int N, int X) {
int i=0;while (i<N and A[i] <X) {
if (A[i]<0) A[i] = - A[i];
i++;}return(1);
} i++
Web Quality - Buenos Aires (Argentina) July 21-25 118
Condition CoverageCondition Coverage
i=0
i<N and A[i] <X
A[i]<0
A[i] = - A[i];return(1)
TrueFalse
TrueFalse
Both conditions (i<N), (A[i]<X) must be false and true for different tests. In this case, we must add tests that cause the while loop to exit for a value greater than X. Faults that arise after several iterations of the loop would not be revealed.
int select(int A[], int N, int X) {
int i=0;while (i<N and A[i] <X) {
if (A[i]<0) A[i] = - A[i];
i++;}return(1);
} i++
Web Quality - Buenos Aires (Argentina) July 21-25 119
Path CoveragePath Coverage
i=0
i<N and A[i] <X
A[i]<0
return(1)
TrueFalse
TrueFalse
The loop must be iterated given number of times.PROBLEM: uncontrolled growth of test sets. We need to select a significant subset of test cases.
int select(int A[], int N, int X) {
int i=0;while (i<N and A[i] <X) {
if (A[i]<0) A[i] = - A[i];
i++;}return(1);
} i++;
A[i] = - A[i];
Web Quality - Buenos Aires (Argentina) July 21-25 120
Data Flow CoverageData Flow Coverage
DEF={i}
USE={i,N,A,X}
USE={A,i}
USE={i}DEF={i}
TrueFalse
True
False
Exercise Def-Use paths: selects paths based on effects on the variables, rather than number of iteration of loops
int select(int A[], int N, int X) {
int i=0;while (i<N and A[i] <X) {
if (A[i]<0) A[i] = - A[i];
i++;}return(1);
} USE={A,i}DEF{A}
DEF={A,N,X}
Web Quality - Buenos Aires (Argentina) July 21-25 121
The Infeasibility ProblemThe Infeasibility Problem
Syntactically indicated behaviors (paths, data flows, etc.) are often impossible
Infeasible control flow, data flow, and data statesAdequacy criteria are typically impossible to satisfy Unsatisfactory approaches:
Manual justification for omitting each impossible test case (esp. for more demanding criteria)Adequacy “scores” based on coverage
example: 95% statement coverage, 80% def-use coverage
Web Quality - Buenos Aires (Argentina) July 21-25 122
Regression TestingRegression Testing
Testing a new version (release): how can we minimize effort using results of testing of previous versions?On a previous release:
save scaffoldings (drivers,stubs,oracles)record test cases (<inputs,outputs>)
On the new release:keep track of changesevaluate impact of changes
Web Quality - Buenos Aires (Argentina) July 21-25 123
Create ScaffoldingCreate Scaffolding
initialization of non-local variablesinitialization of parameters
activation of the unit
DRIVER
“templates” of modules used by the unit (functions called
by the unit)“templates” of any other entity used by the unit
STUB
PROGRAM UNIT
ORACLE
check the correspondence between
the produced and the
expected result
Web Quality - Buenos Aires (Argentina) July 21-25 124
Problems and TradeoffsProblems and Tradeoffs
effort in developing drivers/stubs
effort in test execution and regression testing
poorly designed drivers/stubs
well designed drivers/stubs
high effort in test execution and regression testing
high effort in development
low effort in test execution and regression testing
low effort in development
Web Quality - Buenos Aires (Argentina) July 21-25 125
What is an oracle?What is an oracle?
1324535968....
An “Inspector” of executions:do test executions produce
acceptable results?
An oracle can be:human being
machinea former version of the same program
another program.....
Web Quality - Buenos Aires (Argentina) July 21-25 126
What is a good oracle?What is a good oracle?
Testing large, complex applications may require millions of test runs
ewfcdgvr09-4tfewv95gff9btbfd766dv0
erfjmvesf78-6rsdvv45fvf6bfdfb666gf7
gygirfnm55-8nyfng43nhynhum001bn4
hnhnghn88-7hgmh11vfnnhnhm888hh6
ewfcdgvr09-4tfewv95gff9btbfd766dv0
erfjmvesf78-6rsdvv45fvf6bfdfb666gf7
gygirfnm55-8nyfng43nhynhum001bn4
hnhnghn88-7hgmh11vfnnhnhm888hh6
................
................
...........
The size of the outputs to be
inspected exceed the capabilities of
human eyes
human eyes are slow and unreliable examiners even of small number of outputs
AUTOMATED ORACLES ARE ESSENTIAL!AUTOMATED ORACLES ARE ESSENTIAL!
Web Quality - Buenos Aires (Argentina) July 21-25 127
How can we build acceptable How can we build acceptable oracles? oracles?
There is NO universal recipe
application domains development environments development phases
• GUI• protocols• .....
• no specifications• informal specifications• formal specifications• .......• system testing
• regression testing• ........
Different solutions for different
Web Quality - Buenos Aires (Argentina) July 21-25 128
Oracles from DesignOracles from Design
Example: UML design notationsMessage sequence charts
A UML message sequence chart indicates a test case and expected outcome, which can be interpreted by a driver and oracle Typical of “scenario-based” oracles
scenarios combine test case with special oracleStateChart (finite state acceptor)
A UML finite state machine describes all permissible behaviors of a module
oracle can be used with large numbers of automatically generated test cases
Web Quality - Buenos Aires (Argentina) July 21-25 129
Oracles from Code DocumentationOracles from Code Documentation
Parnas’ tabular annotations precisely describe the functional behavior of the unit. The table can be evaluated with respect to the produced outputs to check for their correctness .
* from: Parnas, Madey, Iglewski, Precise Documentation of Well-Structured Programs, IEEE-TSE Vol.. 20 N. 12 Dec 1994
*DISPLAY 1 Display 1 Specification
Find(x,A,j,present)
R0(,) = ((1n) and [for all (1in) ‘A[i] ‘A[i+1]])
present’=
j’ |
i[(1in) and (‘A[i]=‘x)] =
true false
‘A[j]=‘x true
true false and NC(x,A)
Display 1 Programprocedure find (...)..........end {find}
Display 1 Specifications of the Invoked Programs
..........
Web Quality - Buenos Aires (Argentina) July 21-25 130
Harness vs. Embedded AssertionsHarness vs. Embedded Assertions
Driver
Stubs
Unit or Subsystem
OracleOracleDriver
Stubs
Unit or Subsystem
Oracle
Oracle
Oracle
Oracle
Oracle
Oracle
Embedded assertions act as oracles within the unit under test
Web Quality - Buenos Aires (Argentina) July 21-25 131
Assertions as OraclesAssertions as Oracles
/* * Alphabetic sort of an array of strings */void sort( char *words[ ], int nwords ) {
. . . assert( is_sorted(words, nwords) ); return; }
Web Quality - Buenos Aires (Argentina) July 21-25 132
Another example: HttpUnitAnother example: HttpUnit
The main class is WebConversationIt is a browser that interacts with a server
With this class we can buid several interactions
WebConversation wc = new WebConversation();
WebResponse resp = wc.getResponse("http://httpunit.sourceforge.net/doc/Cookbook.html");
WebLink link = resp.getLinkWith("response");
Web Quality - Buenos Aires (Argentina) July 21-25 133
An exampleAn example
Check
Check
Web Quality - Buenos Aires (Argentina) July 21-25 134
An exampleAn example
Check
! Check password
PolitecnicoPolitecnicodi Milanodi Milano
AnalysisAnalysis
Web Quality - Buenos Aires (Argentina) July 21-25 136
Software Inspection:Software Inspection:Low tech but effectiveLow tech but effective
Fagan Code InspectionsOne of many “walk-through” and inspection techniques; among the most successful
More formal and well-defined than “structured walk-throughs” etc.
Has been extended to designs, requirements, etc. with similar organizing principlesA completely manual technique for finding and correcting errors
Web Quality - Buenos Aires (Argentina) July 21-25 137
Software Inspection RolesSoftware Inspection Roles
Moderator: Typically borrowed from another project. Chairs meeting, chooses participants, controls process
Readers, Testers:Read code to group, look for flaws
Author:Passive participant; answer questions when asked
Web Quality - Buenos Aires (Argentina) July 21-25 138
Software Inspection ProcessSoftware Inspection Process
PlanningModerator checks entry criteria, choose participants, schedule meeting
OverviewProvide background education, assign roles
PreparationInspection (see ahead)ReworkFollow-up (& possible re-inspection)
Web Quality - Buenos Aires (Argentina) July 21-25 139
In the MeetingIn the Meeting
Goal: Find as many faults as possiblemax 2 x 2 hour sessions per dayapprox. 150 source lines/hour
Approach: Line-by-line paraphrasingReconstruct intent of code from sourceMay also “hand test”
Find and log defects, but don’t fix themModerator responsible for staying on track
Web Quality - Buenos Aires (Argentina) July 21-25 140
Checklists — NASA exampleChecklists — NASA example
About 2.5 pages for C code, 4 for FORTRANDivided into: Functionality, Data Usage, Control, Linkage, Computation, Maintenance, Clarity
Examples:Does each module have a single function?Does the code match the Detailed Design?Are all constant names upper case?Are pointers not typecast (except assignment of NULL)?Are nested “INCLUDE” files avoided? Are non-standard usages isolated in subroutines and well documented?Are there sufficient comments to understand the code?
Web Quality - Buenos Aires (Argentina) July 21-25 141
Inspection AutomationInspection Automation
Although a manual technique, many kinds of automated support are possible:
Automate trivial checks (e.g., formatting)Reference: Checklists, standards w/ examplesFocus (highlight, selection) on relevant partsAnnotation & CommunicationProcess guidance and (partial) enforcement
e.g., InspeQ will not allow check-off until all relevant parts of a document have been observed
Web Quality - Buenos Aires (Argentina) July 21-25 142
Why does inspection work?Why does inspection work?
The evidence says it is cost-effective. Why?Detailed, formal process, with record keepingCheck-lists; self-improving processSocial aspects of process, esp. for authorConsideration of whole input spaceApplies to incomplete programs
LimitationsScale: Inherently a unit-level techniqueNon-incremental; what about evolution?
Web Quality - Buenos Aires (Argentina) July 21-25 143
Data flow analysisData flow analysis
if (a < b)
tmp := a; a := b; b := tmp;
absdiff := a - b;
function absdiff (a, b: integer)
return integer is
tmp: integer;
begin
if (a < b) then
tmp := a;
a := b;
b := tmp;
end if;
return ( a - b );
end absdiff;
Web Quality - Buenos Aires (Argentina) July 21-25 144
Classic data flow analyses to find Classic data flow analyses to find program errorsprogram errors
Uninitialized variable“May” result from classic “avail” analysis
but conservative analysis can be annoying“Must” version is also possible (how?)
Dead assignment (no possible use)Classic “live variables” analysis In FORTRAN, Awk, BASIC, PERL, etc., usually indicates a misspelled variableless useful in languages requiring declarations
Web Quality - Buenos Aires (Argentina) July 21-25 145
Precision & SafetyPrecision & Safety
An analysis is conservative (safe) if it doesn’t miss errors An analysis is precise to the extent that it doesn’t report spurious errorsStatic flow analysis considers all (syntactic) program paths; it can be conservative or precise, but not both
An overly conservative, imprecise analysis may be useless.A well-defined but overly strict property may be preferable to spurious error reports
Web Quality - Buenos Aires (Argentina) July 21-25 146
Analysis of Models: Analysis of Models: State-Space ExplorationState-Space Exploration
Concurrency (multi-threading, distributed programming, ...) makes testing harder
introduces non-determinism; time- and load-dependent bugs escape extensive testing
Finite-state models can be exhaustively verified
accept E do ... ...
-E
?E
!E
Extract Combine Check
E
Web Quality - Buenos Aires (Argentina) July 21-25 147
Automated Finite-State VerificationAutomated Finite-State Verification
Example tool SPIN (one of many)
verifies simple program-like design modelhigh-level design of process interaction, ignoring other aspects of computation (e.g., functional behavior)
used for protocols, OS scheduling, ... useful despite limited capacity; best for verifying high-level design before coding
Domain-specific analysislimited “proof” of simple but critical properties in a limited domain
G. Holzmann, “The model checker SPIN.” IEEE TSE 23(5), May 1997
G. Holzmann, “The model checker SPIN.” IEEE TSE 23(5), May 1997
Web Quality - Buenos Aires (Argentina) July 21-25 148
State explosion is one face of a (provably) hard problem. The same fundamental limits appear in different form for
non-enumerative analyses
State explosion problemState explosion problem
Size of composite state graph is product of individual state graphs.
OK for a simple two-party protocol, but impossibly expensive for systems with many processes
Brute force state enumeration is limited to a few processes
Web Quality - Buenos Aires (Argentina) July 21-25 149
What is static analysis good for?What is static analysis good for?
Not a replacement for testingfocused, (mostly) automated analysis for limited classes of faults
More thorough than testing (within scope)conservative analyses are tantamount to formal verification
Also augments testing, e.g., dependence analysis for data flow testing
PolitecnicoPolitecnicodi Milanodi Milano
Load and performance testLoad and performance test
Web Quality - Buenos Aires (Argentina) July 21-25 151
Robustness and scalabilityRobustness and scalability
Robustness: It is the capability of behaving “decently” even in cases not explicitly considered in the requirements definition document
If we consider Web applications, the number of users is the key factor
Scalability: It is the capability of serving a given load, along with a predefined QoS, and being able to adapt to the evolutions of the load
Web Quality - Buenos Aires (Argentina) July 21-25 152
An example loadAn example load problem problem
Yahoo was attacked by sending thousands of emails and the servers collapsed
Web Quality - Buenos Aires (Argentina) July 21-25 153
ApproachesApproaches
We can analyze the architecture to discover problems in its components or in the way they interact
We can verify their design and interactionsWe can produce models
We can test the application (load test)
Web Quality - Buenos Aires (Argentina) July 21-25 154
Load testLoad test
We simulate a given load on the server by means of virtual users
These users behave like real users and test the capability of the system to support the loadWe must have the application before being able to work on load testing
With this test, we can Study the HW/SW configuration needed to support a given loadDiscover the maximum load we want offer with the current configuration
Web Quality - Buenos Aires (Argentina) July 21-25 155
HowHow
Manual testsWe use employees during the week-endsIt is really expensive and we cannot simulate exceptional situationsWe need millions of users for Web applications
Automatic tests (Mercury, Rational, Empirix):We do not need too many employeesWe can perform what-if simulationsWe can better collect statistics
Available tools record the activities done by real tools, create the scripts that virtual users should execute, and measure the mean time to answerScripts can be parametric and configurable and use a DB to store and retrieve data and generate coherent use cases
Web Quality - Buenos Aires (Argentina) July 21-25 156
Astra load test (Mercury)Astra load test (Mercury)
It can emulate thousands of users and provides a graphical monitor to identify and isolate problemsIt allows users to record browsing/interaction sessionsIt allows users to fill forms in a parametric way
San FranciscoPortlandAcapulco
Web Quality - Buenos Aires (Argentina) July 21-25 157
Astra load testAstra load test
The test monitor allows us to
Define the number of virtual usersIdentify the tests that we want to execute and are already stored in the right formatThe machines on which we want to execute the test
The monitor also displays the results even during the execution of the tests
PolitecnicoPolitecnicodi Milanodi Milano
PerformancePerformance
Web Quality - Buenos Aires (Argentina) July 21-25 159
PerformancePerformance
We should always consider our clients before designing the application
Its weight basically should depend on them
We should carefully plan performance testing at the beginning of our project
Our requirements should clearly state what we wantWe must clearly state pass and fail conditions
A transaction that takes too longs can be seen as an errorWhat is the meaning of “too long”?
Web Quality - Buenos Aires (Argentina) July 21-25 160
What should we test? Where?What should we test? Where?
We must identify those cases are we think can be critical
Trivial problems become important on the server because of the degree of parallelismDifferent applications have different criticalities
In client-server like architectures we should always identify the bottleneck
Is the server slow?Is available bandwidth limited on the client side?Are CPU and memory not sufficient?
Usually, we mainly consider problems on the server sideBefore starting we must clearly know the HW/SW architecture
Web Quality - Buenos Aires (Argentina) July 21-25 161
Three levelsThree levels
Ad hoc performance testingTesters validate if the application answers with a reasonable delay. They may record failures, but they do not locate them
Observational testingWe use the first measures, but we use watches and similar tools
Measured testingWe use objective measures and specific tools
Web Quality - Buenos Aires (Argentina) July 21-25 162
What do we measure?What do we measure?
Measures on the serverMegacycles (MCs)Memory footprint
Measures that consider the networkTime to last byte (TTLB)User-perceived response time
Other measuresBytes over the wire (BoW)
Web Quality - Buenos Aires (Argentina) July 21-25 163
Megacycles (MCs)Megacycles (MCs)
CPU cost = (CPU usage * number of CPUs * CPU speed in MHz) / requests per secondThis measure defines the computation cost on the CPU
If I know this cost and the number of users, I can foresee the number of CPUs I need
When I add a new CPU, at most it gives an improvement of .8 because of the hardware it shares (bus, ram, ...)
Web Quality - Buenos Aires (Argentina) July 21-25 164
ExampleExample
I have two bi-processor Web servers that work at 400 MHzTheir usage is equal to the 60% They must server 30 requests per second
The mean value of each measure is: Available MCs =(1+0.8)+(1+0.8)*400= 1440Used MCs =1440*0.60= 864MCs used for each request =864/30= 28.8Requests that we can serve per second =1440/28.8= 50
Web Quality - Buenos Aires (Argentina) July 21-25 165
Other measuresOther measures
Memory footprintIt is used to study the maximum usage of memoryIt is not interesting if we have only static pages and imagesIt is interesting when we have dynamic pages, accesses to DBs, and code that uses memory (Java and C#)
Time to last byte (TTLB)It measures the time from the instant at which the request leaves the client till the server sends the last byteIt does not consider the time used by the client to execute what receives (to execute a script, render a page)If the “execution” is complex and time-consuming, the user can have significant delays even if the TTLB is low
Web Quality - Buenos Aires (Argentina) July 21-25 166
Other measuresOther measures
User-perceived response timeIt is the time needed to fully load a page on the client and have it ready to interact with the userIt is usually greater than TTLB since we must consider the overhead necessary after the last byte
Bytes over the wire (BoW)It counts the number of bytes moved between client and server. It distinguished between:
First request: the cache is emptyFurther request: the cache contains some data and we must transmit only what changes (dynamic data)
Web Quality - Buenos Aires (Argentina) July 21-25 167
Model-based validationModel-based validation
Can provide quick feedback during the designAllow for the tuning of resources
page
Client
Internal Web
Internal LAN
Internal LAN Internet
ReqReq
Req
new page
page
pageReq
Web Quality - Buenos Aires (Argentina) July 21-25 168
Performance testingPerformance testing
We need testing toolsFor example: E-Test (Empirix)
We create a script that corresponds to a given browsing/interaction session
To measure the performance of the server with respect to the number of usersTo verify the performance of each single component and identify bottlenecks
Kb/sec
Pages/sec
Transactions/secMemory
usage by the DB
CPU usage by the DB
Web Quality - Buenos Aires (Argentina) July 21-25 169
Testing services through the WebTesting services through the Web
We can use some applications that allow us to test our applications without installing anything on our machinesFor example: www.netmechanic.comThey work on the load time for each page and on the “weight” of each page or componentLoad Time 38.71 seconds, height/width
problems
Load Time by Modem Speed Modem Speed Download Time 14.4k 75.42 seconds 28.8k 38.71 seconds 56k 20.68 seconds ISDN (128k) 10.26 seconds T1 (1.44 MB) 2.73 seconds
Size Object URL
78698 HTML http://www.tiscali.it/
6743 IMG http://www.tiscali.it/magazine/Italy/spettacoli/mytv/foto/ gino_maggio_2.jpg
5591 IMG http://www.tiscali.it/magazine/Italy/eventi/eventi/Media/Foto/2002/ aprile/19/junior_spazio.jpg
5246 IMG http://www.tiscali.it/magazine/Italy/eventi/eventi/Media/Foto/2002/ maggio/2/foto_epoca.jpg
4179 SCRIPT http://www.tiscali.it/inc/hp/cookie.js
Original: JPEG Size: 6743 BytesH: 130W: 150DL Time (28.8): 0:02
Type: JPEGQuality: 30Size: 4170 bytesDL Time (28.8): 0:01Savings: 38 %
•You may want to break this Web page into several smaller pages. •Click on an image's file size to reduce it with GIFBot. •Adding HEIGHT and WIDTH attributes to your images will help browsers display your page sooner. •Adding WIDTH attributes to your TABLE tags will help browsers display your page sooner. •Your page's overall rating was lowered one level because of HTML problems. •Correct the HTML problems to get our highest rating.
PolitecnicoPolitecnicodi Milanodi Milano
Security testSecurity test
????????
PolitecnicoPolitecnicodi Milanodi Milano
ProcessProcess
Web Quality - Buenos Aires (Argentina) July 21-25 172
Software Qualities and ProcessSoftware Qualities and Process
Qualities cannot be added after developmentQuality results from a set of inter-dependent activitiesAnalysis and testing are crucial but far from sufficient.
Testing is not a phase, but a lifestyleTesting and analysis activities occur from early in requirements engineering through delivery and subsequent evolution. Quality depends on every part of the software process
An essential feature of software processes is that software test and analysis is thoroughly integrated and not an afterthought
Web Quality - Buenos Aires (Argentina) July 21-25 173
The Quality ProcessThe Quality Process
Quality process: set of activities and responsibilitiesfocused primarily on ensuring adequate dependability concerned with project schedule or with product usability
The quality process provides a framework for selecting and arranging activities considering interactions and trade-offs with other important goals.
Web Quality - Buenos Aires (Argentina) July 21-25 174
Interactions and tradeoffsInteractions and tradeoffs
Example: high dependability vs. time to marketMass market products:
Better to achieve a reasonably high degree of dependability on a tight schedule Than to achieve ultra-high dependability on a much longer schedule
Critical medical devices:Better to achieve ultra-high dependability on a much longer scheduleThan a reasonably high degree of dependability on a tight schedule
Web Quality - Buenos Aires (Argentina) July 21-25 175
Properties of the Quality ProcessProperties of the Quality Process
Completeness: appropriate activities are planned to detect each important class of faults. Timeliness: faults are detected at a point of high leverage (as early as possible)Cost-effectiveness: activities are chosen depending on cost and effectiveness
cost must be considered over the whole development cycle and product lifethe dominant factor is likely to be the cost of repeating an activity through many change cycles.
Web Quality - Buenos Aires (Argentina) July 21-25 176
Planning and MonitoringPlanning and Monitoring
The quality process balances several activities across the whole development processselects and arranges them to be as cost-effective as possibleimproves early visibility
Quality goals can be achieved only through careful planningPlanning is integral to the quality process
Web Quality - Buenos Aires (Argentina) July 21-25 177
Process VisibilityProcess Visibility
A process is visible to the extent that one can answer the question
How does our progress compare to our plan?‘Are we on schedule? How far ahead or behind?
The quality process has not achieved adequate visibility if one cannot gain strong confidence in the quality of the software system before it reaches final testing
quality activities are usually placed as early as possible
design test cases at the earliest opportunity (not “just in time'') uses analysis techniques on software artifacts produced before actual code
Web Quality - Buenos Aires (Argentina) July 21-25 178
A&T StrategyA&T Strategy
Identifies company- or project-wide standards that must be satisfied
procedures required for obtaining quality certificates techniques and tools that must be useddocuments that must be produced
Web Quality - Buenos Aires (Argentina) July 21-25 179
Quality planQuality plan
a comprehensive description of the quality process that includes:
objectives and scope of quality activitiesdocuments and other items that must be available items to be testedfeatures to be tested and not to be testedanalysis and test activities staff involved in qualityconstraintspass and fail criteriascheduledeliverableshardware and software requirementsrisks and contingencies
Web Quality - Buenos Aires (Argentina) July 21-25 180
Improving the ProcessImproving the Process
Long lasting errors are commonIt is important to structure the process for
Identifying the most critical persistent faultstracking them to frequent errorsadjusting the development and quality processes to eliminate errors
Feedback mechanisms are the main ingredient of the quality process for identifying and removing errors
Web Quality - Buenos Aires (Argentina) July 21-25 181
Organizational factorsOrganizational factors
Different teams for development and quality?separate development and quality teams is common in large organizationsindistinguishable roles is postulated by some methodologies (extreme programming)
Different roles for development and quality?test designer is a specific role in many organizationsmobility of people and roles by rotating engineers over development and testing tasks among different projects is a possible option
Web Quality - Buenos Aires (Argentina) July 21-25 182
Example of Allocation of Example of Allocation of ResponsibilitiesResponsibilities
Allocating tasks and responsibilites is a complex job: we can allocate
Unit testing to the development team (requires detailed knowledge of the code)but the quality team may control the results (structural coverage)
Integration, system and acceptance testing to the quality teambut the development team may produce scaffolding and oracles
Inspection and walk-through to mixed teams
Regression testingto quality and maintenance teams
Process improvement related activities to external specialists interacting with all teams
Web Quality - Buenos Aires (Argentina) July 21-25 183
Product vs. process improvementProduct vs. process improvement
Product improvement: Fault is detected (by inspection, testing, user report, ...) Fault is diagnosed and repaired
Process improvementFaults are detected (and maybe repaired)Fault record is analyzed to tune process
Web Quality - Buenos Aires (Argentina) July 21-25 184
Fault analysisFault analysis
What are the faults?Categorize by kind (Memory leak, interface error, mis-feature, etc.)And by severity
When did they occur? And when found?Coding? Design? Requirements?
Why did they occur? Look for “root causes”
How could they be prevented?
Web Quality - Buenos Aires (Argentina) July 21-25 185
Categorizing faultsCategorizing faults
There is no “right” categorizationMay depend on design style, implementation language, process and documents, ... Should probably be revised occasionally
Goal is enough precision for “Pareto” analysis (80/20 rule) considering severity and cost
Categorization needn’t be perfect or painful, but keeping records is essential
Web Quality - Buenos Aires (Argentina) July 21-25 186
Fault severityFault severity
Typical breakdown (of failures): Critical: Product is unusableSevere: Product feature cannot be used; no workaroundModerate: Product feature can be used only with workaround (loss of efficiency, reliability, or significant loss of convenience)Cosmetic or minor inconvenience
Cost may be distinct from severity
Web Quality - Buenos Aires (Argentina) July 21-25 187
80/20 rule (a.k.a. Pareto analysis)80/20 rule (a.k.a. Pareto analysis)
Identify one or two “dominant” fault categoriesConsidering severity, cost, and frequencyFurther problem analysis is limited to these
Categories may “level” over timeA good time for rethinking the categories
Web Quality - Buenos Aires (Argentina) July 21-25 188
Test DocumentationTest Documentation
Must be an organization standardDepends on
organization (size,turnover,..), type of software (criticality, average life, complexity, number of versions,...)
It must include at least:test suite documentationtest case documentation
Web Quality - Buenos Aires (Argentina) July 21-25 189
Test Documentation (cont....)Test Documentation (cont....)
Documentation of test suites:software testedversiongoaloverall resultsauthor
Documentation of Test Cases:goal“environment” (driver,stub,oracle)inputexpected outputactual outputresultobservations
PolitecnicoPolitecnicodi Milanodi Milano
ToolsTools
Web Quality - Buenos Aires (Argentina) July 21-25 191
ToolsTools
Test process managementValidator/ReqTest DirectorRational RobotMercury WinRunner Segue SilkTest Compuware QA Run
UsabilityWebSATBobbyPage ValetW3C HTML Validator ServiceHTML Authoring ServiceXenuAlert LinkRunnerHTML Link ValidatorHTML Validator 5.0Web Link Validator
Web Quality - Buenos Aires (Argentina) July 21-25 192
ToolsTools
PerformanceWeb Performance TrainerHttploadWebserver Stress ToolHttp/s Load OpenSTAjMeter
ScaffoldingATTOL Unit Test (NOW Rational)Cantata++jUnitCactusHTTPunit
CoverageLogiscope TestCeckerDeep CoverjCoverLDRA TestBedAttol Coverage (Rational)
Memory leaksPurify (Rational)Sentinel
Static analysisLogiscope AuditLDRA TestBed
PolitecnicoPolitecnicodi Milanodi Milano
ReferencesReferences
Web Quality - Buenos Aires (Argentina) July 21-25 194
Web sitesWeb sites
www.w3c.orgwww.useit.comwww.softwareqatest.com/qatweb1.htmlwww.junit.org/index.htmwww.swquality.com/users/pustaver/index.shtmlstandards.ieee.org/catalog/olis/se.htmlwww.stickyminds.comwww.qaforums.comwww.qualitytree.com
www.cs.uoregon.edu/~michal/book/
Web Quality - Buenos Aires (Argentina) July 21-25 195
BooksBooks
UsabilityJakob Nielsen, “Designing Web Usability: The Practice of Simplicity”, New Riders Publishing, Indianapolis, 2000
Software engineeringIan Sommerville, "Software Engineering". Addison-Wesley, 2000Roger S. Pressman, "Software Engineering: A Practitioner's Approach". McGraw-Hill, 2000
Test in generalCem Kaner, Hung Quoc Nguyen, Jack Falk, "Testing Computer Software", John Wiley & Sons, 1999
Web Quality - Buenos Aires (Argentina) July 21-25 196
BooksBooks
Test for object-oriented softwareRobert V. Binder, "Testing Object-Oriented Systems: Models, Patterns, and Tools", Addison-Wesley, 1999
Software metricsNorman E. Fenton, Shari Lawrence Pfleeger, "Software Metrics: A Rigorous and Practical Approach, Revised", Brooks/Cole, 1998Stephen H. Kan, "Metrics and Models in Software Quality Engineering", Addison-Wesley, 1995
PolitecnicoPolitecnicodi Milanodi Milano
Grazie !!!!Grazie !!!!
Luciano BaresiLuciano BaresiDEI - Politecnico di MilanoDEI - Politecnico di Milano
Piazza L. da Vinci, 32 - 20133 Milano (Italia)Piazza L. da Vinci, 32 - 20133 Milano (Italia)tel: 02 2399 3638 tel: 02 2399 3638
email: [email protected]: [email protected]
www.elet.polimi.it/~baresiwww.elet.polimi.it/~baresi
Web Quality - Buenos Aires (Argentina) July 21-25 198
HomeworkHomework
Study and analyze one of the arguments presented in these days
Some particular methodologiesSome interesting examples…
Imagine you are a test manager and try to identify the activities that should be carried out to test a Web application
For example, www.amazon.comConsider what, how, and whenTry to estimate costs in terms of person months
Constraints10 days5 pages (at most)