View
218
Download
1
Tags:
Embed Size (px)
Citation preview
Abstract
ANSI standard ANSI/INCITS-354 (Common Industry Format (CIF)), adopted in 2001, provides a standard format for the sharing of usability-related data. As a newly ratified standard, the CIF has yet to gain industry-wide support, and is still being evaluated for roll-out within IBM. However, adoption of the CIF within IBM is important for several reasons. Several large companies (both competitors and customers) support the CIF, which may become a requirement for sales (similar to the Government section 508 Accessibility requirements). Early adopters of the CIF are Boeing, Kodak, Oracle Corporation, and State Farm Insurance. The CIF will lead to improvements in the User Centered Design (UCD) process within IBM through the standardization of reports across teams and products. The CIF will also become a necessity to maintain competitiveness (e.g., Oracle Corporation). The poster will provide a history and requirements of the CIF document, IBM corporate strategy regarding the CIF, a comparison to an existing process, UCD process improvements and the benefits to IBM.
What is the ANSI Standard ANSI/INCITS-354 Common Industry Format (CIF)?
Usability standard for direct comparison between competitive products
Most likely performed by an independent testing organization
Conducted after a product is released Compared with competitive product for usability Evaluate comparison and weigh differences for purchasing
decisions Help procurement and purchasing for large companies Document audience is primarily usability experts
The CIF does not tell you what to do; it tells you how to report on what you did
CIF has a dual nature—highlights both product strengths and weaknesses
History
1996 – NIST recognized the need to: Encourage software suppliers and consumer organizations to work together to
understand user needs and tasks. Develop a common usability reporting format for sharing usability data with consumer
organizations. Conduct a pilot trial to determine how well the usability reporting format works
and to determine the value of using this format in software procurement. Keith Butler (Boeing) started and drove the standards work group IBM UCD Advisory Council and Microsoft have provided feedback to the
core CIF team during development of the standard Ziff Davis Publishing wanted to make this a usability seal of approval, but
that idea was denied Anticipate a government tie into the ANSI standard (like accessibility
requirements) Being considered for inclusion in ISO Standard 9241 (Usability Standard)
Support of CIF and Early Adopters
Include user quotes here Boeing Kodak Oracle State Farm Insurance Microsoft HP
Standard Report Format
Title Page
Executive Summary
Introduction Full Product Description Test objectives
Method Participants Context of Product Use in Test Experimental Design Usability Metrics
Results Data Analysis Presentation of the Results
References
Appendices
Title Page
Identify the report as a Common Industry Format (CIF) document
State the CIF version State contact information (i.e., ‘Comments and questions:
[email protected]’) Product name and version/release tested Research lead and contact information Date(s) test was conducted Date the report was completed Report author
Executive Summary
Provides high level summary of the test
Intent is to provide information for procurement decision-makers in customer organizations
Identify and description of product Summary of the method(s) used in the
test Results expressed as mean scores or
other suitable measure of central tendency
Reason for and nature of the test Tabular summary of performance
results
Introduction
Full Product Description Formal product name and release or version What parts were evaluated Intended user population Any groups with special needs Brief description of environment the product
should be used in The work that is supported by the product
Test Objectives Describes objectives for the test Functions and components the user directly
or indirectly interacted with during the test Whether or not the function or component
tested was a subset of the total product. If so, provide reason for testing subset
Methods
Key technical section of the report Must provide enough information to
allow an independent tester to replicate the procedure used in testing
Participants Description of the user population and
test sample Total number of participants tested Segmentation of user groups tested Key characteristics and capabilities of
the user groups How participants were selected and if
they met essential characteristics and capabilities
Whether or not the participant sample included representatives of groups with special needs.
Business Sector
Company Job Title Storage Experience/ Responsibilities
Storage Software Experience
Storage Hardware Experience
Elements of Current Storage Environment
P1
P2
Pn
Methods
Context of Product Use in the Test Description of the tasks, scenarios and conditions in which
the test was performed Tasks
• Description of task scenarios used for testing• Explanation of why the scenarios were selected• Description of the source of tasks• Description of task data provided to the participants• Completion or performance criteria established for each task
Test Facility• Physical description of the test facility• Details of relevant features or circumstances that may affect
the quality of the results (e.g., recording equipment, one-way mirrors)
Participant’s Computing Environment• Software and hardware configuration details, display details,
audio details, and/or manual input details Test Administrator Tools
• Describe any hardware or software used to control the test or record data
• Describe any questionnaires used to collect data
Methods
Experimental Design Describes the logical design of the test Procedure
• Provide independent or control variables, operational definitions of measures, and any policies or procedures for task time limits, training, assistance, intervention or responding to questions.
• Provide sequence of events from greeting to dismissing participants
• Provide steps the evaluation team followed to execute the study and record data and the roles they played
• State details of non-disclosure agreements, informed consent/human subjects rights, and compensation
Usability Metrics• Explain what measures have been used
for each category of usability metrics: completion rates, errors, assists, time-on-task, completion rates, satisfaction ratings
3 Required CIF Usability Metric Categories
Effectiveness: Empowering users to succeed in their tasks
Efficiency: Enabling people to work faster to save time and money
Satisfaction: Reducing frustration and under-utilization
3 Additional IBM Usability Metric Categories
Flexible: Allowing people to work in ways that match their situation
Easy to learn: Reducing time to value with or without training
Safe: Preventing accidents and business errors
Results
Second major technical section of the report Describes how the data were scored, reduced
and analyzed and provides the major findings in quantitative formats
Data Analysis Provide sufficient detail to allow replication of
data scoring, data reduction, and analysis methods by another organization
Presentation of the Results Required to report effectiveness, efficiency, and
satisfaction results in tabular and graphical presentations to describe the data
A mixed-factor design was used for this study with 16 participants. The dependent variables were acceptability scores and qualityrating. The independent variables were:
Drives (Company A, Company B, Company C, and Company D) Environment (Office and Lab) Gender (Male and Female)
The design classification was:
P13, P14, P15, P16P9, P10, P11, P12P5, P6, P7, P8P1, P2, P3, P4Company DP13, P14, P15, P16P9, P10, P11, P12P5, P6, P7, P8P1, P2, P3, P4Company CP13, P14, P15, P16P9, P10, P11, P12P5, P6, P7, P8P1, P2, P3, P4Company BP13, P14, P15, P16P9, P10, P11, P12P5, P6, P7, P8P1, P2, P3, P4Company A
FemaleMaleFemaleMaleLabOffice
The treatment order was a partially counterbalanced Balanced Latin Square design. This treatment order was used to control forpresentation order bias and gender bias:
Drive 2Drive 1Drive 4Drive 34Drive 3Drive 2Drive 1Drive 43Drive 1Drive 4Drive 3Drive 22OrderDrive 4Drive 3Drive 2Drive 11Presentation
P4, P8, P12, P16P3, P7, P11, P15P2, P6, P10, P14P1, P5, P9, P13
Analysis of Variance (ANOVA) was first performed to determine significant differences between the perceived quality ofthe drives based on the overall quality score for each drive (question 3 of the survey). A Regression Analysis was thenperformed to determine what factors affected perceived quality based on the sound attribute scores (question 2 of thesurvey). An alpha level of 0.05 was used for all analyses.
In general, a significant difference was found between the overall quality scores of the drives for both the office and labsettings. Mean scores, standard deviations, and confidence levels for each drive in each setting were as follows:
0.60930.72899.41.24401.48807.5Company D1.48211.77288.51.57891.88868.2Company C1.94792.32996.51.48211.77285.5Company B0.62200.74406.42.17782.60495.3Company A
ConfidenceLevel
StandardDeviation
MeanConfidence
LevelStandardDeviation
Mean
Lab SettingOffice Setting
Appendices
Detailed study materials Customer questionnaires, participant general
instructions, participant task instructions, release notes.
References
Common Industry Format for Usability Test Reports (version 1.1, October 28, 1999). Available from [email protected]
P. Englefield (personal communication, June 6, 2003)D. Gonzalez (personal communication, April 25, 2003)E. Reinke (personal communication, June 9, 2003)K. Vredenburg (personal communication, May 13, 2003)
Improvements in Usability Coverage
Usability tests designed to meet CIF requirements Standardizes UCD reporting Standardizes core set of UCD metrics Provides a standard means to measure competitive
products’ usability
Benefits and Competitiveness
Improvements to UCD process will help drive user-friendly product that meet user requirements.
CIF will provide a yardstick to compare our usability to the competition – highlight areas that we can improve or exceed the competition.
Usability is a product differentiator.