Human Factors Experimental Design and Analysis Reference · 2011. 5. 14. · i Army Research Laboratory Aberdeen Proving Ground, MD 21005-5425 ARL-RP-0186 July 2007 Human Factors

Human Factors Experimental Design

and Analysis Reference

by Robert C. Willages

ARL-RP-0186 July 2007

A reprint from Virginia Polytechnic Institute and State University Technical Report Number HFEEC-06-01 dated December 2006.

Approved for public release; distribution is unlimited.

NOTICES

Disclaimers

The findings in this report are not to be construed as an official Department of the Army position unless so designated by other authorized documents.

Citation of manufacturers’ or trade names does not constitute an official endorsement or approval of the use thereof.

DESTRUCTION NOTICE⎯Destroy this report when it is no longer needed. Do not return it to the originator.

i

Army Research Laboratory Aberdeen Proving Ground, MD 21005-5425

ARL-RP-0186 July 2007

Human Factors Experimental Design

and Analysis Reference

Robert C. Willages Virginia Polytechnic Institute and State University

A reprint from Virginia Polytechnic Institute and State University Technical Report Number HFEEC-06-01 dated December 2006.

Approved for public release; distribution is unlimited.

ii

REPORT DOCUMENTATION PAGE Form Approved OMB No. 0704-0188 Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing the burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 1. REPORT DATE (DD-MM-YYYY)

July 2007 2. REPORT TYPE

Reprint 3. DATES COVERED (From - To)

5a. CONTRACT NUMBER 5b. GRANT NUMBER

4. TITLE AND SUBTITLE

Human Factors Experimental Design and Analysis Reference

5c. PROGRAM ELEMENT NUMBER 5d. PROJECT NUMBER

5e. TASK NUMBER

6. AUTHOR(S)

Robert C. Willages (VPI)

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)

U.S. Army Research Laboratory Human Research & Engineering Directorate Aberdeen Proving Ground, MD 21005-5425

8. PERFORMING ORGANIZATION REPORT NUMBER

ARL-RP-0186

10. SPONSOR/MONITOR'S ACRONYM(S)

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

11. SPONSOR/MONITOR'S REPORT NUMBERS

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited. 13. SUPPLEMENTARY NOTES

The contracting officer’s representative (COR) is Sam Middlebrooks, U.S. Army Research Laboratory, ATTN: AMSRD-ARL-HR-MV, Fort Hood, TX, telephone number 254-288-9379. 14. ABSTRACT

CADRE (computer-aided design reference for experiments) is a desktop computer tool for human factors and ergonomic researchers. The tool is in Acrobat1 format for cross-platform computer use and has more than 500 bookmarks for information search. This tool provides more than 850 Power-Point2 note pages of applied experimental design and analysis reference material. The reference material covers 25 topics and is divided into five major sections including introduction to experimental design, supplemental data collection design and analysis, basic analysis of variance (ANOVA) designs, advanced ANOVA designs, and empirical model building. References to the scientific literature are provided in each topic for supplemental reading. The CADRE tool also provides more than 200 pages explaining 40 examples of statistical analyses covered by the reference material. These examples are hyperlinked to Version 9.1.3 of the SAS3 (2004) statistical analysis software, and the sample SAS programs can be easily modified for user-specific data. 1Acrobat is a registered trademark of Adobe Systems, Inc. 2PowerPoint is a trademark of Microsoft. 3SAS, which is not an acronym, is a registered trademark of the SAS Institute, Inc. 15. SUBJECT TERMS

experimental design

16. SECURITY CLASSIFICATION OF: 19a. NAME OF RESPONSIBLE PERSON Robert C. Willages

a. REPORT UNCLASSIFIED

b. ABSTRACT UNCLASSIFIED

c. THIS PAGE UNCLASSIFIED

17. LIMITATION OF ABSTRACT

SAR

18. NUMBER OF PAGES

1,130 19b. TELEPHONE NUMBER (Include area code)

540-951-7243 Standard Form 298 (Rev. 8/98) Prescribed by ANSI Std. Z39.18

Technical Report HFEEC-06-01


Robert C. Williges

December 2006

Prepared For Collaborative Technology Alliance on Advanced Decision Architectures Micro Analysis and Design Colorado Springs, Colorado and U.S. Army Research Laboratory Human Research and Engineering Directorate Aberdeen Proving Ground, Maryland 21005-5425

Human Factors Engineering and Ergonomics Center Grado Department of Industrial and Systems Engineering

Virginia Polytechnic Institute and State University Blacksburg, Virginia 24061

CADRE Computer-Aided

Design Reference for Experiments

By

Robert C. Williges R.H. Bogle Professor Emeritus

Grado Department of Industrial and Systems

Engineering

2007

CADRE is a desktop computer tool for human factors and ergonomic research-ers. The tool is in PDF format for cross-platform computer use and has over 500 bookmarks for information search.

This tool provides over 850 PowerPoint note pages of applied experimental de-sign and analysis reference material by Williges (2006). The reference material covers 25 topics and is divided into five major sections including introduction to experimental design, supplemental data collection design and analysis, basic analysis of variance (ANOVA) designs, advanced ANOVA designs, and empiri-cal model building. References to the scientific literature are provided in each topic for supplemental reading,

The CADRE tool also provides over 200 pages in a companion appendix by Sla-ter and Williges (2006) explaining 40 ex-amples of statistical analyses covered by the reference material. These examples are hyperlinked to Version 9.1.3 of the SAS (2004) statistical analysis software, and the example SAS programs can be easily modified for user specific data.

CADRE Description

The CADRE project was supported by the Hu-man Research and Engineering Directorate (HRED) of the Army Research Laboratory un-der the direction of Dr. Sam Middlebrooks and Dr. Michael Strub of HRED and through its prime contractor Micro Analysis and Design.

Slater, C.R. and Williges, R.C. (2006). Appen-dix: SAS examples for human factors ex-perimental design and analysis reference. Blacksburg, VA: Virginia Polytechnic Insti-tute and State University. Technical Re-port HFECC-06-02.

Williges, R.C. (2006). Human factors experi-mental design and analysis reference. Blacksburg, VA: Virginia Polytechnic Insti-tute and State University. Technical Re-port HFECC-06-01.

Williges, R.C. (2007). CADRE: Computer-Aided Design Reference for Experiments (a tool for human factors and ergonomic research). Blacksburg, VA: Virginia Poly-technic Institute and State University. Electronic Book CD-ROM-07-01.

References


ii

Table of Contents

Topics Page Overview 0.1. Purpose of Reference Material ………………………………………………………….3

0.1.1. Applied Experimental Design 0.1.2. Human Factors Engineering Methods

0.1.2.1. Research Methods 0.1.2.2. User Interface Design Methods 0.1.2.3. Training System Design Methods 0.1.2.4. Usability Evaluation Methods

0.2. Presentation Approach …………………………………………………………………10 0.2.1. Format of Reference Material 0.2.2. Experimental Design References

0.3. Organization of Reference Topics ……………………………………………………..13 0.3.1. Introduction to Experimental Design 0.3.2. Supplemental Data Collection and Analysis 0.3.3. Basic Analysis of Variance Designs 0.3.4. Advanced Experimental Designs

Section 1. Introduction to Experimental Design Topic 1. Research Design Process 1.1. Stages of Research ……………………………………………………………….........21 1.2. Research Problem ………………………………………………………………………22 1.3. Research Approach ……………………………………………………………………..23 1.4. Critical Research Methods ……………………………………………………………..24

1.4.1. Variables 1.4.2. Procedures 1.4.3. Protection of Human Subjects 1.4.4. Equipment 1.4.5. Pretesting

1.5. Research Design Alternatives ………………………………………………………...30 1.6. Analyzing Results ……………………………………………………………………….31 1.7. Research Reports ……………………………………………………………………….32

1.7.1. Scientific Reports 1.7.2. Major Components of Reports

1.7.2.1. Introduction 1.7.2.2. Method 1.7.2.3. Results 1.7.2.4. Discussion

1.7.3. Additional Considerations 1.8. Summary …………………………………………………………………………………41 1.9. Supplemental Readings ………………………………………………………………..44


iii

Topic 2. Experimental Designs 2.1. Introduction ………………………………………………………………………………46

2.1.1. Threats to Validity 2.1.2. Quantitative Research Approach

2.2. Experimental Design Alternatives ……………………………………………………..51 2.2.1. Experimental Design Notation 2.2.2. Quasi-Experimental Designs

2.2.2.1. Single Group Designs 2.2.2.2. Nonequivalent Control Group Designs 2.2.2.3. Regression-Discontinuity Designs 2.2.2.4. Interrupted Time Series Designs

2.2.3. Randomized Experimental Designs 2.3. Summary …………………………………………………………………………….......69 2.4. Supplemental Readings ………………………………………………………………..70 Topic 3. Basic Statistical Concepts 3.1. Probability ………………………………………………………………………………..72

3.1.1. Compositional Techniques 3.1.2. Counting Techniques

3.2. Random Sampling ………………………………………………………………………81 3.3. Sampling Distributions ………………………………………………………………….82

3.3.1. Binomial Distribution 3.3.2. Normal Distribution 3.3.3. Student's t Distribution 3.3.4. Chi-Squared Distribution 3.3.5. F-Distribution

3.4. Statistical Estimation ……………………………………………………………………99 3.4.1. Estimators 3.4.2. Point Estimation 3.4.3. Interval Estimation 3.4.4. Summary of Statistical Estimation

3.5. Statistical Hypothesis Testing ………………………………………………………...109 3.5.1. Components of Hypothesis Testing

3.5.1.1. Null and Alternative Hypotheses 3.5.1.2. Format for Hypothesis Test 3.5.1.3. Types of Errors 3.5.1.4. Statistical Power

3.5.2. Single-Sample t-Test 3.5.3. Relationship to Statistical Estimation

3.6. Two-Sample t-Tests …………………………………………………………………...122 3.6.1. Sampling Distribution 3.6.2. Assumptions 3.6.3. Standard Format 3.6.4. Between-Subjects t-Test 3.6.5. Within-Subjects t-Test 3.6.6. Conclusion


iv

3.7. Summary ………………………………………………………………………………..139 3.8. Supplemental Readings …………………………………………………………........140 Section 2. Supplemental Data Collection and Analysis Topic 4. Supplemental Data Collection Methods 1.1. Background ……………………………………………………………………………..143

1.1.1. Types of Dependent Variables 1.1.2. Analysis Procedures

1.2. Nonparametric Procedures …………………………………………………………...147 1.2.1. Scales of Measurement 1.2.2. Classification Scheme

1.3. Subjective Measures …………………………………………………………………..154 1.3.1. Self Reports

1.3.1.1. Verbal Protocols 1.3.1.2. Critical Incidents

1.3.2. Questionnaires 1.3.3. Psychometric Scaling

1.4. Graphic Rating Scales ………………………………………………………………...164 1.4.1. Likert Rating Scales 1.4.2. Bipolar Adjective Scales 1.4.3. Rating Scale Reliability and Validity 1.4.4. Examples of Rating Scales

1.5. Summary ………………………………………………………………………………..177 1.6. Supplemental Readings …………………………………………………………........178 Topic 5. Analysis of Nominal Scale Data 5.1. Background ……………………………………………………………………………..180 5.2. Between-Subjects Tests ………………………………………………………………181

5.2.1. Chi-Square Goodness of Fit Test 5.2.2. Chi-Square Test of Independence

5.3. Within-Subjects Tests …………………………………………………………………..195 5.3.1. McNemar Change Test 5.3.2. Cochran Q Test

5.4. Summary ………………………………………………………………………………..202 5.5. Supplemental Readings ………………………………………………………………203 Topic 6. Analysis of Ordinal Scale Data 6.1. Background ……………………………………………………………………………..205 6.2. Between-Subjects Tests ………………………………………………………………206

6.2.1. Kolmogorov-Smirnov Tests 6.2.2. Kruskal-Wallis One-Way ANOVA

6.3. Within-Subjects Tests …………………………………………………………………..217 6.3.1. Wilcoxon Signed Ranks Test 6.3.2. Friedman Two-Way ANOVA

6.4. Summary ………………………………………………………………………………..226


v

6.5. Supplemental Readings ………………………………………………………………..227 Topic 7. Summary of Supplemental Data 7.1. Supplemental Data Collection …………………………………………………………229

7.1.1. Self Reports and Questionnaires 7.1.2. Rankings and Rating Scales

7.2. Supplemental Data Analysis……………………………………………….…………..232 7.2.1. Nominal Scale Data Analysis 7.2.2. Ordinal Scale Data Analysis

7.3. Supplemental Data Process……………………………...……………………………235 7.4. Summary………………………………………………………………………………... 238 7.5. Supplemental Readings………...…………………………………………….………..239 Section 3. Basic Analysis of Variance (ANOVA) Designs Topic 8. Introduction to ANOVA 8.1. Advantages of ANOVA Designs……………………………………………………....242 8.2. Basic Terms ……………………………………………………………………………..243 8.3. ANOVA Design Alternatives…………………………………………………………...246 8.4. ANOVA Statistical Models ……………………………………………………….........250

8.4.1. Specification Procedures 8.4.2. Examples

8.5. ANOVA Hypothesis Testing ..…………………………………………………………258 8.5.1. Format of F-Test

8.5.1.1. Theoretical F 8.5.1.2. Hypotheses 8.5.1.3. Complete Format

8.5.2. Assumptions of the F-Test 8.5.3. Two-Level Design

8.5.3.1. Components of Deviation Score 8.5.3.2. Estimation of Population Variance 8.5.3.3. Hypothesis Test

8.6. Summary…………………………………………………………………………………272 8.7. Supplemental Readings …………..……………………………………………………273 Topic 9. ANOVA Summary Table Components 9.1. Introduction ……………………………………………………………………………275 9.2. Sources of Variation ……………………………………………………………………276 9.3. Degrees of Freedom (df) ……………………………………………………………….280

9.3.1. Rules for Determining df 9.3.2. df Examples

9.4. Sum of Squares (SS) …………………………………………………………………..285 9.5. Mean Squares (MS) …………………………………………………………………….286

9.5.1. Expected Mean Squares, E(MS) 9.5.2. Algorithm for Stating E(MS) 9.5.3. E(MS) Examples


vi

9.6. F-Ratios ………………………………………………………………………………….297 9.6.1. Rules for Determining F-Ratios 9.6.2. F-Ratio Examples

9.7. Complete ANOVA Summary Table …………………………………………………...303 9.7.1. Summary Table Components 9.7.2. Summary Table Examples

9.8. ANOVA Design Construction ………………………………………………………….308 9.9. Summary ………………………………………………………………………………309 9.10. Supplemental Readings ……………………………………………………………..310 Topic 10. Between-Subjects ANOVA Designs 10.1. One-Factor, Between-Subjects Design………………………………………….....312

10.1.1. One-Factor Design Example 10.1.2. Sum of Squares Calculations

10.1.2.1. Simplified Design Notation 10.1.2.2. SS Computational Formulae Algorithm 10.1.2.3. SS Numerical Computations

10.1.3. Summary Table and Test Format 10.2. Two-Factor, Between-Subjects Design …………………………………………....324

10.2.1. Design Configuration 10.2.2. AxB Interaction 10.2.3. Calculations 10.2.4. Two-Factor Design Example

10.3. n-Factor, Between-Subjects Design ………………………………………………..336 10.3.1. Three-Factor Design 10.3.2. Generalizations

10.4. Summary ………………………………………………………………………………340 10.5. Supplemental Readings ……………………………………………………………..341 Topic 11. Analysis of Comparisons and Interactions 11.1. Multiple Comparisons………………………………………………………………...343

11.1.1. Linear Comparisons 11.1.2. Inflated Type I Error 11.1.3. Planned Comparisons

11.1.3.1. Planned F-Test 11.1.3.2. Critical Difference 11.1.3.3. Planned Bonferroni t Test (Dunn Test)

11.1.4. Unplanned Comparisons 11.1.4.1. Least Significant Difference Test (LSD) 11.1.4.2. Bonferroni t Test (Dunn Test) 11.1.4.3. Scheffé Multiple Contrast Procedure 11.1.4.4. Tukey’s Honestly Significant Difference Test (HSD) 11.1.4.5. Dunnett Test 11.1.4.6. Newman-Keuls Sequential Range Test 11.1.4.7. Choice of Procedure

11.2. Evaluating Interactions……………………………………………………………….376


vii

11.2.1. Example Problem 11.2.2. Graphing Procedures 11.2.3. Simple Effects Test 11.2.4. Trend Analysis 11.2.5. Paired Comparisons

11.2.5.1. Sequential Range Test 11.2.5.2. Unconfounded Comparisons

11.2.6. Interaction Evaluation Process 11.3. Summary ………………………………………………………………………………399 11.4. Supplemental Readings ……………………………………………………….........400 Topic 12. Within-Subjects ANOVA Designs 12.1. Within-Subjects Design Configurations…………………………………………….402

12.1.1. Single-Factor Design 12.1.2. Two-Factor Design 12.1.3. n-Factor Design

12.2. Homogeneity of Covariance ………………………..……………………………….417 12.3. Balancing Order of Treatments ……………………………………………………..423

12.3.1. Balancing Alternatives 12.3.2. Balanced Latin Square 12.3.3. Testing Order Effects

12.4. Differential Transfer…………………………………………………………………..437 12.5. Within-Subjects Design Advantages ……………………………………………...438 12.6. Summary……………………………………………………………………………....440 12.7. Supplemental Readings ……………………………………………………………..441 Topic 13. Mixed-Factors ANOVA Designs 13.1. Mixed-Factors Design Configurations ……………………………………………...443

13.1.1. Two-Factor Design 13.1.2. Two-Factor Design Example 13.1.3. Three-Factor Design 13.1.4. n-Factor Design

13.2. Mixed-Factors Design Considerations ……………………………………………456 13.3. Summary……………………………………………………………………………....457 13.4. Supplemental Readings ……………………………………………………………..458 Topic 14. Summary of Basic ANOVA 14.1. Basic Considerations……………………………………………….........................460 14.2. ANOVA Rules and Algorithms ..……………………………………………............461

14.2.1. Specification of Statistical Models 14.2.2. Rules for Degrees of Freedom 14.2.3. SS Computational Formulae Algorithm 14.2.4. Algorithm for Stating E(MS) 14.2.5. Steps for Determining F-Ratios

14.3. Design Classification…... ……………………………………………………………467 14.4. n-Factor Design Generalizations……………………………………………………471


viii

14.5. ANOVA Design Process……………………………………………………………..472 14.6. Summary……………………………………………………………………………....475 14.7. Supplemental Readings ……………………………………………………………..476 Section 4. Advanced ANOVA Designs Topic 15. Introduction to Advanced ANOVA 15.1. Basic ANOVA Extensions …………………………………………………………...479

15.1.1. Quasi-F Ratios 15.1.2. Randomized Blocks Design

15.1.2.1. Constructing Randomized Blocks 15.1.2.2. Design Comparison 15.1.2.3. Extensions of Randomized Blocks

15.2. Advanced ANOVA Design and Analysis…………………………………………...489 15.3. Summary ………………………………………………………………………………490 15.4. Supplemental Readings ……………………………………………………………..491 Topic 16. Hierarchical ANOVA Designs 16.1. Basic Hierarchical Designs ………………………………………………………….493

16.1.1. Between-Subjects Design 16.1.2. Within-Subjects Design 16.1.3. Mixed-Factors Design

16.2. Hierarchical Design Examples ……………………………………………………...503 16.2.1. Complete Hierarchical Design 16.2.2. Partial Hierarchical Design

16.3. Summary ………………………………………………………………………………516 16.4. Supplemental Readings ……………………………………………………………..517 Topic 17. Blocking ANOVA Designs 17.1. Modular Representation ……………………………………………………………..519

17.1.1. Modular Arithmetic 17.1.2. Balanced Sets of Treatments 17.1.3. Component SS Formulae 17.1.4. Generalizations

17.2. Blocking 2k Designs ………………………………………………………………….533 17.2.1. Simple Blocking of 2k Design 17.2.2. Complex Blocking of 2k Design 17.2.3. Computational Considerations

17.2.3.1. Simple Blocking Example 17.2.3.2. Complex Blocking Example

17.3. Pseudo-Factor Blocking ……………………………………………………………..554 17.4. Summary ………………………………………………………………………………558 17.5. Supplemental Readings ……………………………………………………………..559 Topic 18. Fractional-Factorial ANOVA Designs 18.1. 2k-p Fractional Replicates …………………………………………………………..561


ix

18.1.1. Design Construction 18.1.1.1. One-Half Replicate 18.1.1.2. One-Fourth Replicate

18.1.2. Computational Considerations 18.1.3. Design Resolution

18.1.3.1. Design Effects 18.1.3.2. Definitions of Resolution 18.1.3.3. Identity Relationship 18.1.3.4. Resolution III Design 18.1.3.5. Resolution IV Design 18.1.3.6. Resolution V Design 18.1.3.7. Uses of Design Resolution

18.2. Latin Square ANOVA Designs ……………………………………………………...592 18.2.1. Design Construction

18.2.1.1. Standard Latin Square 18.2.1.2. Balanced Latin Square 18.2.1.3. Relationship to Fractional Replicates

18.2.2. Computational Considerations 18.2.2.1. Additivity Assumption 18.2.2.2. Between-Subjects Design 18.2.2.3. Within-Subjects Design 18.2.2.4. Latin Square Examples

18.2.3. Design Constraints 18.3. Summary ………………………………………………………………………………611 18.4. Supplemental Readings ……………………………………………………………..612 Topic 19. Analysis of Covariance (ANCOVA) 19.1. Introduction to ANCOVA……………………………………………………………..614 19.2. Linear Correlation, r12 ………………………………………………………………..616

19.2.1. Correlation Coefficient 19.2.1.1. Computational Formulae 19.2.1.2. Tests of Significance

19.2.2. Alternative Correlations 19.2.2.1. Point Biserial Correlation, rpbi 19.2.2.2. Phi Correlation, rφ 19.2.2.3. Spearman Correlation, rρ 19.2.2.4. Partial Correlation, r(1.3)(2.3) 19.2.2.5. Semipartial Correlation, r1(2.3)

19.3. Simple Linear Regression …………………………………………………………...654 19.3.1. Line of Best Fit

19.3.1.1. Method of Least Squares 19.3.1.2. Calculation Example 19.3.1.3. Standardized Regression

19.3.2. Goodness of Fit 19.3.2.1. Partitioning Variance 19.3.2.2. Tests of Significance


x

19.3.2.3. Coefficient of Determination 19.4. ANCOVA Computations ……………………………………………………………..670

19.4.1. Basic ANCOVA Design 19.4.2. Advanced ANCOVA 19.4.3. Interpreting ANCOVA

19.5. Summary ………………………………………………………………………………685 19.6. Supplemental Readings ……………………………………………………………..686 Topic 20. Summary of Advanced ANOVA 20.1. ANOVA Design Constraints …………………………………………………………688

20.1.1. Random-Effects Factors 20.1.2. Nested Factors 20.1.3. Control of Nuisance Factors 20.1.4. Data Collection Limitations 20.1.5. Control of Subject Variability

20.2. Advanced ANOVA Design Process………………………………………………...696 20.3. ANOVA of Regression Analysis …………………………………………………….697 20.4. Summary ………………………………………………………………………………698 20.5. Supplemental Readings ……………………………………………………………..699 Section 5. Empirical Model Building Topic 21. Introduction to Empirical Models 21.1. Quantitative Models…………………………………………………………………..702

21.1.1. Mechanistic Models 21.1.2. Empirical Models

21.2. Model Building Experiments ………………………………………………………...708 21.3. Models in Human Factors …………………………………………………………...710 21.4. Summary ………………………………………………………………………………713 21.5. Supplemental Readings ……………………………………………………………..714 Topic 22. Multiple Regression 22.1. Multiple Regression Procedures ……………………………………………………716 22.2. Multiple Linear Regression…………………………………………………………..721

22.2.1. Line of Best Fit 22.2.2. Goodness of Fit 22.2.3. Multiple Regression Example 22.2.4. Best Regression Equation

22.2.4.1. Backward Selection 22.2.4.2. Forward Selection 22.2.4.3. Stepwise Selection 22.2.4.4. All Possible Regressions

22.2.5. Best Equation Example 22.3. Second-Order Polynomial Regression……………………………………………..753

22.3.1. Polynomial Regression Calculations 22.3.2. Polynomial Regression Example


xi

22.4. Summary ………………………………………………………………………………764 22.5. Supplemental Readings……………………………………………………………...765 Topic 23. Central-Composite Designs (CCD) 23.1. CCD Introduction ……………………………………………………………………..767 23.2. CCD Specification ……………………………………………………………………771

23.2.1. CCD Configuration 23.2.2. Replication 23.2.3. Value of α

23.3. CCD Analysis………………………………………………………………………….785 23.3.1. Between-Subjects CCD 23.3.2. Within-Subjects CCD 23.3.3. Mixed-Factors CCD

23.4. CCD Examples………………………………………………………………………..794 23.4.1. Between-Subjects Example 23.4.2. Within-Subjects Example

23.5. Alternative Second-Order Designs …………………………………………………806 23.6. Summary ………………………………………………………………………………809 23.7. Supplemental Readings ……………………………………………………………..810 Topic 24. Sequential Experimentation 24.1. Strategies for Experimentation ……………………………………………………...812 24.2. Response Surface Methodology (RSM)……………………………………………816

24.2.1. Steps in RSM 24.2.2. Method of Steepest Ascent

24.3. Sequential Research …………………………………………………………………822 24.3.1. Sequential Research Paradigm 24.3.2. Sequential Research Example 24.3.3. Guidelines for Sequential Research

24.4. Integrated Research Database……………………………………………………...849 24.5. Summary ………………………………………………………………………………852 24.6. Supplemental Readings ……………………………………………………………..853 Topic 25. Summary of Empirical Models 25.1. Model Building Experiments ………………………………………………………...855 25.2. Components of Empirical Models …………………………………………………..856 25.3. Sequential Experimentation Process ………………………………………………858 25.4. Overall Conclusions ………………………………………………………………….863 25.5. Summary ………………………………………………………………………………864 25.6. Supplemental Readings ……………………………………………………………..865 Acknowledgements…………………………………………………………………………866 References …………………………………………………………………………………...867 Subject index………………………………………………………………………………...874


xii

Notes on Computational Examples Examples Page Example 1. Interval Estimation ……………………………………………………………..107 Example 2. Single-Sample t-Test …………………………………………………………..117 Example 3. Between-Subjects t-Test …………………………………...…………………130 Example 4. Within-Subject t-Test …………………………………………………….....…136 Example 5. Chi-Square Goodness of Fit Test ………………………………………...….184 Example 6. Chi-Square Test of Independence (2x2 Contingency Table) ………….….189 Example 7. Chi-Square Test of Independence (RxC Contingency Table)……………..192 Example 8. Chi-Square Test of Independence (Two Additive 2x2 Partitions) ………...193 Example 9. McNemar Change Test ………………………………………………………..198 Example 10. Cochran Q Test ……………………………………………………………....200 Example 11. Kolmogorov-Smirnov Test……………………………………………………210 Example 12. Kruskal-Wallis One-Way ANOVA …………………………………………...215 Example 13. Wilcoxon Signed Ranks Test ………………………………………………..220 Example 14. Friedman Two-Way ANOVA…………………………………………………225 Example 15. One-Factor, Between-Subjects ANOVA …………………………………...312 Example 16. Two-Factor, Between-Subjects ANOVA …………………………………...330 Example 17. Planned Comparisons ……………………………………………………..…353 Example 18. Unplanned Comparisons ………………………………………………….....362 Example 19. Analysis of Interactions ……………………………………………………....377 Example 20. One-Factor, Within-Subjects ANOVA ……………………………………....405 Example 21. Two-Factor, Within-Subjects ANOVA ……………………………………....413 Example 22. Geisser-Greenhouse and Huynh-Feldt Corrections ……………………....421 Example 23. Testing Order Effects in Balanced Latin Squares………………………....432 Example 24. Within-Subjects and Between-Subjects Design Comparison …………....439 Example 25. Two-Way, Mixed-Factors ANOVA …………………………………………..448 Example 26. Complete Hierarchical Between-Subjects Design ………………………...504 Example 27. Partial Hierarchical Between-Subjects Design …………………………….509 Example 28. Simple Blocking of 2k Within-Subjects Design …………………………….546 Example 29. Complex Blocking of 2k Within-Subjects Design …………………………..550 Example 30. One-Half Replicate of 24 Between-Subjects Design ………………………557 Example 31. 4x4 Latin Square Designs ……………………………………………………605 Example 32. Linear Correlation Coefficient …………………………………………….....623 Example 33. Alternative Linear Correlations ………………………………………………636 Example 34. Simple Linear Regression ……………………………………………………659 Example 35. One-Way Analysis of Covariance (ANCOVA) ……………………………..673 Example 36. Multiple Linear Regression …………………………………………………..732 Example 37. Best Regression Equation …………………………………………………...745 Example 38. Polynomial Regression……………………………………………………….756 Example 39. Orthogonal, Between-Subjects, Central-Composite Design ……………..796 Example 40. Blocked, Within-Subjects, Central-Composite Design ……………………802


1

Human Factors ExperimentalDesign and Analysis ReferenceHuman Factors ExperimentalHuman Factors Experimental

Design and Analysis ReferenceDesign and Analysis Reference

Dr. Robert C. WilligesDr. Robert C. WilligesRalph H. Ralph H. BogleBogle Professor EmeritusProfessor Emeritus

GradoGrado Department of Industrial and Systems EngineeringDepartment of Industrial and Systems EngineeringVirginia Polytechnic Institute and State UniversityVirginia Polytechnic Institute and State University

December 2006December 2006

This set of slides on Human Factors Experimental Design and Analysis is designed to provide reference material to human factors engineers on research design and analysis techniques. This material is organized around the concept of a researcher’s handbook that is available on a desktop computer and can provide an overview of critical experimental design concepts and methods for the human factors engineer as well as provide key references to the scientific literature related to these techniques.

It is assumed that users of this material are researchers in human factors engineering and ergonomics who have background in statistics andexperimental design. These slides and accompanying notes providereference material to help the researcher choose the appropriateexperimental design and analysis. This reference material is not designed as a simple look-up for statistical procedures. Rather, it is designed to providean overview and roadmap to techniques with reference to the statistical literature that provides details of procedures that should be reviewed before using them.


2

This is the outline of topics covered in the Overview to the reference material on applied experimental design. The purpose, presentation style, and organization of the topics are discussed in this overview.

Each of the subsequent major topic presentations in this reference material begins with an numbered outline of the subtopics covered. The detailed information content for every major topic follows this numbering system to facilitate user reference.

OverviewOverviewOverview

0.1. Purpose of Reference Material0.1. Purpose of Reference Material0.1.1. Applied Experimental Design0.1.1. Applied Experimental Design0.1.2. Human Factors Engineering Methods0.1.2. Human Factors Engineering Methods

0.2. Presentation Approach0.2. Presentation Approach0.2.1. Format of Reference Material0.2.1. Format of Reference Material0.2.2. Experimental Design References0.2.2. Experimental Design References

0.3. Organization of Reference Topics0.3. Organization of Reference Topics0.3.1. Introduction to Experimental Design0.3.1. Introduction to Experimental Design0.3.2. Supplemental Data Collection and Analysis0.3.2. Supplemental Data Collection and Analysis0.3.3. Basic Analysis of Variance Designs0.3.3. Basic Analysis of Variance Designs0.3.4. 0.3.4. Advanced Experimental DesignsAdvanced Experimental Designs0.3.5. Empirical Model Building0.3.5. Empirical Model Building


3

0.1. Purpose of Reference Material0.1. Purpose of Reference Material0.1. Purpose of Reference Material

•• 0.1.1. Applied Experimental Design0.1.1. Applied Experimental Design•• 0.1.2. Human Factors Engineering Methods0.1.2. Human Factors Engineering Methods

This is an example of the outline slide that introduces each topic subsection. As this outline suggests, the purpose of this reference material is to provide an overview of various applied experimental design procedures that are useful in human factors engineering. The implications of appliedexperimental design and its relationship to human factors methods are described in this subsection.


4

0.1.1. Applied Experimental Design0.1.1. Applied Experimental Design0.1.1. Applied Experimental Design

•• Research Methods Research Methods NOTNOT StatisticsStatistics–– No Statistical DerivationsNo Statistical Derivations–– Use of Algorithms and ProceduresUse of Algorithms and Procedures–– Basic and Advanced Design AlternativesBasic and Advanced Design Alternatives

•• Emphasis on Research DesignEmphasis on Research Design–– Choosing the Most Efficient AlternativeChoosing the Most Efficient Alternative–– Design Implications and TradeoffsDesign Implications and Tradeoffs

•• Statistical AnalysisStatistical Analysis–– Show only Underlying AnalysisShow only Underlying Analysis–– Assume Use of Statistical PackagesAssume Use of Statistical Packages–– Examples of SAS ApplicationsExamples of SAS Applications

The emphasis of this reference material is on applied experimental design research methods and not on mathematical statistics. In lieu of statistical derivations, procedural steps and algorithms are presented for various experimental design calculations and representations such as statistical models, expected mean square, computational formulae, etc.

The reference material is designed to aid the human factors researcher in choosing the most efficient experimental design among a variety of available alternatives. Consequently, the various alternatives are outlined, and the tradeoffs among these alternatives are presented.

Examples of statistical analyses are provided for only the major procedures. It is assumed that most researchers will use a statistical analysis package to analyze their data. Consequently, most analyses shown in the reference material are presented in an appendix report by Slater and Williges (2006) that provides the program statements and output pages from the SAS application package.


5

0.1.2. Human Factors Engineering Methods0.1.2. Human Factors Engineering Methods0.1.2. Human Factors Engineering Methods

Human Factors EngineeringInterface DesignTraining Design

Design Methods Evaluation Methods

Research Methods

Human factors engineering involves both the human interface design of complex systems and the complimentary training of users of those systems. Successful human interface and training design requires understanding and mastery of various research, design, and evaluation methods. Applied experimental design is useful in each of these three major categories of methods.


6

0.1.2.1. Research Methods0.1.2.1. Research Methods0.1.2.1. Research Methods

•• Human Factors Engineering ResearchHuman Factors Engineering Research–– Human Performance ResearchHuman Performance Research–– Knowledge Base for Human Interface DesignKnowledge Base for Human Interface Design

•• Key Components of Behavioral ResearchKey Components of Behavioral Research–– Data Collected from Human SubjectsData Collected from Human Subjects–– Same or Different Subjects ObservedSame or Different Subjects Observed–– Capabilities and Limitations of Human OperatorCapabilities and Limitations of Human Operator

Experimental designs are central to human factors engineering research. This research deals primarily with human performance research that focuses on cognitive, motor, and biomechanical aspects of the human. Human factors engineering research provides the scientific knowledge base for human interface design in complex systems, and this research is based largely on experimental designs.

Human factors engineering research is characterized by three keycomponents. First, the data are related to aspects of human performance and are collected from human subjects. Second, either the same sample of subjects is observed in a variety of treatment combinations or an independent sample of subjects is observed in each treatment combination. And, third, the research is focused on developing a scientific database of human operator capabilities and limitations in complex systems.


7

0.1.2.2. User Interface Design Methods0.1.2.2. User Interface Design Methods0.1.2.2. User Interface Design Methods

Initial Design

ConceptualDesign

DesignSpecifications

Prototype Design

PrototypeInterface

FormativeEvaluation

Final Design

OperationalInterface

SummativeEvaluation

Adapted from Kies, Williges, and Rosson (1998) by Permission

User-centered interface design is an iterative design process that is focused on the user of the system as shown by the two-headed arrows and the feedback loop in this figure that was modified from Kies, Williges, and Rosson (1998). In their article, they discuss appropriate ethnographic and experimental design methods for iterative design in each of three major phases of design of computer-supported cooperative work systems.

A variety of methods have been developed to support this design process. Essentially, these methods deal first with initial interface design to provide a conceptual design and specific design specifications. Next a prototype design of the interface is developed and actual users are tested somewhat informally though formative evaluation procedures in an iterative fashion. Following successful prototype design, the final operational interface design is developed and tested through a final, summative evaluation. Additional design iterations and major design revisions could be conducted as shown by the feedback loops in the figure. Rigorous experimental design procedures are most often used during summative evaluation in the user-centered design process.


8

0.1.2.3. Training System Design Methods0.1.2.3. Training System Design Methods0.1.2.3. Training System Design Methods

Specification ofTraining Requirement

Development ofTraining Program

Evaluation ofTraining Effectiveness

TrainingObjectives

GraduatesTrainingTrainees

TrainingNeeds

TrainingMethod

TrainingContent

FormativeEvaluation

SummativeEvaluation

Most complex systems require human operator training in order to achieve the best system performance. The design of these training systems is also an iterative process that involves user testing. As shown in this figure, the three major stages of training system design include specification of training requirements, development of the training program, and evaluation of training effectiveness as discussed by Goldstein and Ford (2002). Experimental designs are used primarily in the summative evaluations of graduates of the resulting training system in order to evaluate the efficacy of training.


9

0.1.2.4. Usability Evaluation Methods0.1.2.4. Usability Evaluation Methods0.1.2.4. Usability Evaluation Methods

Ethnographic MethodsStudy of Work

Contextual InquiryScenario Design

Interaction Analysis

End-User MethodsVerbal ProtocolsCritical Incidents

Participatory DesignUsability Testing

Controlled Testing MethodsPsychophysical Scaling

Efficient Experimental DesignsEmpirical Model Building

Sequential Experimentation

A variety of methods are available to support human factors engineering evaluation activities. As shown on this slide, these methods can be grouped into end-user, ethnographic, and controlled testing methods.

End-user methods involve the user of the system in the evaluation process. Verbal protocols and critical incidents are discussed in more detail as techniques to support supplemental data in experimental design. Participatory design involves end-user participation and evaluation throughout the design process (Schuler and Namioka, 1993). Usability testing methods are focused specifically on issues related to improving user performance of the system primarily during formative evaluation in the iterative design process. Hartson, Andre, and Williges (2003) provide a detailed breakdown of usability testing methods into expert, user, model, and location of usability evaluation methods across a variety of criteria.

Kies, Williges, and Rosson (1998) discuss appropriate ethnographic and experimental design methods for formative and summative evaluation of socio-technical systems. Experimental designs are examples of controlled testing methods. Efficient experimental designs, empirical model building, and sequential experimentation are most useful in complex system research and design.


10

0.2. Presentation Approach0.2. Presentation Approach0.2. Presentation Approach

•• 0.2.1. Format of Reference Material0.2.1. Format of Reference Material•• 0.2.2. Experimental Design References0.2.2. Experimental Design References

The presentation used in this reference material is focused on a researcher’s handbook approach. Both the format of the material and the scientific references are directed toward material to support the human factors engineer who is planning, conducting, analyzing, and reporting results of experiments.


11

0.2.1 Format of Reference Material0.2.1 Format of Reference Material0.2.1 Format of Reference Material

•• PowerPoint Slides FormatPowerPoint Slides Format–– Outline Format of Topic CoverageOutline Format of Topic Coverage–– Brief Description of Key Points in NotesBrief Description of Key Points in Notes

•• PDF Document PresentationPDF Document Presentation–– Interactive Desktop UseInteractive Desktop Use–– Index of TopicsIndex of Topics

•• Examples of SAS Statistical AnalysesExamples of SAS Statistical Analyses–– Keyed to Examples in SlidesKeyed to Examples in Slides–– Program and Analysis OutputProgram and Analysis Output

•• References to Extended CoverageReferences to Extended Coverage–– Emphasis on Behavioral Research TextbooksEmphasis on Behavioral Research Textbooks

The reference material was prepared in a PowerPoint slide format. Each page of the reference shows a slide with the material presented in an outline format. Notes are provided under each slide to provide a brief description of the outline and to emphasize the major points of each slide. All of the reference material is delivered in PDF format to facilitate cross-platform, desktop computer use by the human factors engineer. Bookmarks are provided to a subject index in the PDF file.

Throughout the reference material, formulae are presented for statistical computations and examples are provided for the major computations. Additionally, these examples were calculated on a statistical package using SAS as an example. The data inputs, procedures statements, and computational outputs of SAS are provided in an appendix (Slater and Williges 2006) that is hyper-linked to the reference slides so that researchers can view detailed examples of using a statistical package for computations. Each example in the Slater and Williges (2006) appendix is also linked directly to the SAS editor.

References to the scientific literature are provided throughout. References are also provided for behavioral science textbooks on experimental design that can be used for supplemental reading on a more detailed coverage of each topic covered. The complete citation for each reference is listed at the end of the PDF file.


12

0.2.2 Experimental Design References0.2.2 Experimental Design References0.2.2 Experimental Design References

•• Supplemental ReadingsSupplemental Readings–– Organized by TopicsOrganized by Topics

•• References to JournalsReferences to Journals–– Human Factors Engineering MethodsHuman Factors Engineering Methods

•• References to TextbooksReferences to Textbooks–– Behavioral Research MethodsBehavioral Research Methods–– Basic Statistical AnalysesBasic Statistical Analyses–– General Experimental DesignGeneral Experimental Design–– Advanced Experimental DesignAdvanced Experimental Design

Each topic in the reference material lists supplemental readings. These supplemental readings provide more detailed coverage for a better understanding of each topic. Reference to key methodological articles in the human factors journals and textbooks are provided. In addition, references on behavioral research methods, basic statistical analyses, and experimental design textbooks are provided as appropriate.


13

0.3. Organization of Reference Topics0.3. Organization of Reference Topics0.3. Organization of Reference Topics

•• 0.3.1. Introduction to Experimental Design0.3.1. Introduction to Experimental Design•• 0.3.2. Supplemental Data Collection and Analysis0.3.2. Supplemental Data Collection and Analysis•• 0.3.3. Basic Analysis of Variance Designs0.3.3. Basic Analysis of Variance Designs•• 0.3.4. Advanced ANOVA Designs0.3.4. Advanced ANOVA Designs•• 0.3.5. Empirical Model Building0.3.5. Empirical Model Building

The reference material is organized around five major sections. These sections cover general considerations in experimental design, supplemental data collection and analysis, basic analysis of variance experimental design, advanced experimental design, and empirical model building.

Section 1 covers topics related to critical aspects of the experimental design process used by human factors engineers. Section 2 covers methods of data collection and analysis of supplemental data that are often collected in addition to the major data collected through experimental designs. Section 3 addresses concepts of basic analysis of variance (ANOVA) designs used by human factors researchers for collecting data on human subjects performing tasks in complex systems environments. Section 4 covers advancedexperimental design topics that are useful to human factors engineers who must deal with procedural constraints in data collection and large-scale data collection efforts. Finally, Section 5 describes empirical model building procedures used to predict human performance in complex systems.


14

0.3.1. Introduction to Experimental Design0.3.1. Introduction to Experimental Design0.3.1. Introduction to Experimental Design

•• Research Design ProcessResearch Design Process–– Stages of ResearchStages of Research–– Critical Research MethodsCritical Research Methods–– Research ReportsResearch Reports

•• Experimental Design AlternativesExperimental Design Alternatives–– Threats to ValidityThreats to Validity–– Types of Experimental DesignsTypes of Experimental Designs

•• Basic Statistical Concepts and AnalysesBasic Statistical Concepts and Analyses–– ProbabilityProbability–– Sampling DistributionsSampling Distributions–– Statistical EstimationStatistical Estimation–– Hypothesis TestingHypothesis Testing

The introduction section to experimental design covers three major topics. These topics include the research design process used by the human factors engineer, experimental design alternatives (i.e., quasi-, and randomized experimental designs), and basic statistical concepts and analyses needed for experimental design. These three topics are covered in Section 1 of the reference material.


15

0.3.2. Supplemental Data Collection and Analysis0.3.2. Supplemental Data Collection and Analysis0.3.2. Supplemental Data Collection and Analysis

•• Supplemental Data Collection MethodsSupplemental Data Collection Methods–– Self ReportsSelf Reports–– QuestionnaireQuestionnaire–– Rating ScalesRating Scales

•• Nonparametric AnalysisNonparametric Analysis–– Frequency Data AnalysisFrequency Data Analysis–– Ordinal Data AnalysisOrdinal Data Analysis

Supplemental data collection and analysis involves additional data collected on human subjects to aid in the understanding of the results obtained from the experimental design. Two topics are covered in this section. First, an overview of supplemental data collection methods is discussed with an emphasis on rating scales. Second, a summary of the most common data analysis procedures for supplemental data consisting of frequencies and rank orders are presented. Both of these topics are also covered in Section 2 of the reference material.


16

0.3.3 Basic Analysis of Variance Designs0.3.3 Basic Analysis of Variance Designs0.3.3 Basic Analysis of Variance Designs

•• Analysis of Variance (ANOVA) ClassificationAnalysis of Variance (ANOVA) Classification–– Basic TermsBasic Terms–– Design AlternativesDesign Alternatives–– ANOVA Summary Table ComponentsANOVA Summary Table Components

•• BetweenBetween--Subjects DesignSubjects Design–– One, TwoOne, Two--, and n, and n--Factor DesignsFactor Designs

•• Analysis of Comparisons and InteractionsAnalysis of Comparisons and Interactions–– PairedPaired--ComparisonsComparisons–– Evaluating InteractionsEvaluating Interactions

•• WithinWithin--Subjects DesignSubjects Design•• MixedMixed--Factors DesignFactors Design

Factorial analysis of variance designs are the major experimental designs used by human factors engineers. The reference section on basic ANOVA covers five major topics including analysis of variance design classification, between-subjects or completely randomized designs in which a different group of subjects is used in each treatment condition, post hoc analysis of paired comparisons and interactions, within-subjects or repeated measures designs in which the same subject is used in all treatment conditions, and mixed-factors or split-plot designs in which some treatment conditions are between-subjects conditions and some are within-subjects conditions. Each of these topics is covered in Section 3 of the reference material.


17

0.3.4. Advanced ANOVA Designs0.3.4. Advanced ANOVA Designs0.3.4. Advanced ANOVA Designs

•• Basic ANOVA ExtensionsBasic ANOVA Extensions•• Hierarchical DesignsHierarchical Designs•• Blocking DesignsBlocking Designs

–– Modular RepresentationModular Representation–– Blocking 2Blocking 2kk DesignsDesigns

•• FractionalFractional--Factorial DesignsFactorial Designs–– 22kk--pp Fractional ReplicatesFractional Replicates–– Latin Square DesignsLatin Square Designs

•• Analysis of Covariance (ANCOVA)Analysis of Covariance (ANCOVA)–– Correlation and Simple RegressionCorrelation and Simple Regression–– ANCOVA ComputationsANCOVA Computations

This section of the reference material covers major advanced experimental design and analysis procedures used by human factors engineers to handle certain experimental constraints encountered in research. These advanced designs are built on basic ANOVA and regression analysis. Topics covered in the advanced experimental design section include extensions of basic ANOVA, hierarchical or nested designs, blocking designs, fractional-factorial designs, and fundamentals of simple regression analysis used in the analysis of covariance. These five topics are covered in Section 4 of the reference material.


18

0.3.5 Empirical Model Building0.3.5 Empirical Model Building0.3.5 Empirical Model Building

•• Quantitative ModelsQuantitative Models•• Multiple RegressionMultiple Regression

–– Multiple Linear RegressionMultiple Linear Regression–– SecondSecond--Order Polynomial RegressionOrder Polynomial Regression

•• CentralCentral--Composite Designs (CCD)Composite Designs (CCD)–– CCD SpecificationsCCD Specifications–– CCD AnalysesCCD Analyses

•• Sequential ExperimentationSequential Experimentation–– Response Surface MethodologyResponse Surface Methodology–– Sequential Research Paradigm and GuidelinesSequential Research Paradigm and Guidelines

The final section of the reference material covers empirical model building procedures. Four major topics are covered. The section begins with a discussion of quantitative models in research that are used to predict human performance. Next empirical model building using polynomial regression with central-composite designs are described. Finally, sequential experimentation that involves a series of small related experiments covering an extremely large data space are described as a paradigm for conducting systematic research on complex human factors problems. All of these topics are covered in Section 5 of the reference material.

Due to the building block approach used in presenting the topics covered in this reference material, some questions raised in earlier sections are answered in later sections. The user should user the interactive aspects of this reference to locate expanded discussion of some topics throughout the presentation.


19

By way of introduction, Section 1 summarizes some major components that are fundamental to experimental design and analysis. This section covers:

Topic 1 - the research design process;Topic 2 - major categories of experimental design alternatives; andTopic 3 - a brief review of basic statistical concepts and analyses used in experimental design.

Section 1.Introduction to Experimental Design

Section 1.Section 1.Introduction to Experimental DesignIntroduction to Experimental Design

Topic 1. Research Design ProcessTopic 1. Research Design ProcessTopic 2. Experimental DesignsTopic 2. Experimental DesignsTopic 3. Basic Statistical ConceptsTopic 3. Basic Statistical Concepts


20

This topic is an introduction to experimental design that deals with the overall research design process. First, the various stages of research are presented in a flow diagram. Next six critical aspects of this process are highlighted beginning with the Research Problem through Research Reports.

As with all subsequent topics covered in the reference material, this topic concludes with a summary followed by suggestions for supplemental readings for in-depth coverage of the material covered in this topic. Due to space restrictions, the complete citation for each supplemental reading is not presented on the summary slide. However, the complete citation is presented in the References section that is bookmarked in the PDF file.

Topic 1. Research Design ProcessTopic 1. Research Design ProcessTopic 1. Research Design Process

1.1. Stages of Research1.1. Stages of Research1.2. Research Problem1.2. Research Problem1.3. Research Approach1.3. Research Approach1.4. Critical Research Methods1.4. Critical Research Methods1.5. Research Design Alternatives1.5. Research Design Alternatives1.6. Analyzing Results1.6. Analyzing Results1.7. Research Reports1.7. Research Reports1.8. Summary1.8. Summary1.9. Supplemental Readings1.9. Supplemental Readings


21

1.1. Stages of Research1.1. Stages of Research1.1. Stages of Research

Reprinted from Williges (1995) by Permission

STAGE 1DEFINE

STAGE 2PLAN

STAGE 3CONDUCT

STAGE 5INTERPRET

STAGE 4ANALYZE

DevelopIdea

ReviewLiterature

StateProblem

DevelopHypotheses

DefineVariables

DesignExperiment

DefineControls

DevelopApparatus

DefineProcedures

SelectSubjects

PretestExperiment

CollectData

ReduceData

CalculateStatistics

EstimateParameters

TestHypotheses

DrawInferences

GeneralizeResults

ReportExperiment

Williges (1995) presented a research process with five inter-related stages as depicted in this slide. (This figure is reprinted by permission of Person Education, Inc., Upper Saddle River, New Jersey.) His five stages include defining, planning, conducting, analyzing, and interpreting. Often, an experimenter only thinks of research design and analysis and fails to consider all five stages of the research process. Note that this process is a closed-loop flow of several considerations leading to successful research.

Several important research procedures related to the Williges (1995) five-stage research process are subsequently covered in this topic to highlight major issues that can cause problems in the research enterprise. These procedures begin with the definition stage of research.


22

1.2. Research Problem1.2. Research Problem1.2. Research Problem

•• Research IdeasResearch Ideas–– Martin's PhobiasMartin's Phobias–– ObservationsObservations–– Problem DefinitionProblem Definition–– Research HypothesesResearch Hypotheses

•• Scientific LiteratureScientific Literature–– "Treeing""Treeing"–– SourcesSources

–– Scientific JournalsScientific Journals–– Conference ProceedingsConference Proceedings–– Technical ReportsTechnical Reports–– BooksBooks

•• Abstracts and ReferencesAbstracts and References

Martin (2004) humorously discusses many common apprehensions that new researchers have in conducting research, but remember that the possibility of exactly replicating existing research is quite remote. One should try to state the research problem in one paragraph, and then state the hypothesis to be tested through data collection.

An efficient way of searching the scientific literature is a technique called “treeing”. The researcher reads a recent article related to the research problem and then reviews the articles in its reference list. Always be sure that you read any reference that you cite to insure accuracy. Do not rely on secondary references. Online searches and electronic publishing can facilitate searching the scientific literature.

Two things to consider in reference sources are the scientific rigor and the age of the material. Scientific journals have an editorial review board to enhance rigor, but the review and publishing process may take years. Conference proceedings include the most recent research, but are often only reviewed on the basis of an abstract. Technical reports are reports published by individual laboratories usually without external review. Many books have review chapters that summarize older literature in a research area. A researcher should be compulsive and write notes or an abstract on each article read as well as the complete reference citation.


23

1.3. Research Approach1.3. Research Approach1.3. Research Approach

•• Research ProcessResearch Process–– Systematic ObservationsSystematic Observations

–– Defined CircumstancesDefined Circumstances–– Observable BehaviorsObservable Behaviors

–– Inferred RelationshipsInferred Relationships•• Critical CriteriaCritical Criteria

–– RepeatableRepeatable–– ObjectiveObjective–– QuantitativeQuantitative–– GeneralizableGeneralizable

The scientific method uses experimental designs that require systematic observation during data collection. So, one defines the specificcircumstances under which observations are made. Extraneous variables are controlled to avoid confounding effects and to facilitate interpretation. Human behavior is observed in an unbiased, objective fashion that avoids experimenter opinions. The emphasis is placed on collecting quantified data so that inferential statistical analysis can be conducted on the resulting data set. From these results, one can infer causative relationships.

Besides insuring that the observations are repeatable, objective, and quantitative, the researcher should include as many relevant variables as possible in the investigation so that the results will generalize to real world applications. When all possible variables are operating, there is less control and more random error is added to the experiment. When designing an experiment, one must trade off which variables are controlled and which variables are not controlled to facilitate generalization. This often results in including several variables in one experiment and increases the data collection effort.


24

1.4. Critical Research Methods1.4. Critical Research Methods1.4. Critical Research Methods

•• 1.4.1. Variables1.4.1. Variables•• 1.4.2. Procedures1.4.2. Procedures•• 1.4.3. Protection of Human Subjects1.4.3. Protection of Human Subjects•• 1.4.4. Equipment1.4.4. Equipment•• 1.4.4. 1.4.4. PretestingPretesting

Research methods include topics such as variables, procedures, protecting human subjects, equipment, and pretesting. These four topics are critical because each can often result in major problems in the research process. Each is reviewed separately. Martin (2004) provides a more comprehensive discussion of these topics as well as other methods to consider in designing human subject research.


25

1.4.1. Variables1.4.1. Variables1.4.1. Variables

•• TypesTypes–– UnivariateUnivariate vs. Multivariate Proceduresvs. Multivariate Procedures–– Independent (X) vs. Dependent (Y) VariablesIndependent (X) vs. Dependent (Y) Variables–– Subject VariablesSubject Variables–– Confounding VariablesConfounding Variables

•• Experimental ControlsExperimental Controls–– Control ConditionsControl Conditions–– Experimental DesignsExperimental Designs

There are several types of variables used in discussing experimental designs. Univariate means consideration of one variable while multivariate means consideration of more than one variable. An independent variable (X variable) is a variable that the experimenter manipulates and is independent of the performance of subjects participating in the experiment. A dependent variable (Y variable) is one that depends upon the performance of the subjects in the experiment and constitutes the data collected in the experiment (e.g., errors, completion time, or accuracy). Subject variables are things such as prior experience that one tries to control through randomization or selection. Confounding variables are other variables that occur in the experiment that can affect the experiment but have nothing to do with the focus of the study.

Specific experimental designs are often chosen to control confounding variables. In most human factors research studies, one conducts multivariable experiments involving several independent variables simultaneously. However, human factors researchers usually conduct univariate statistical analyses on each dependent variable separately. Consequently, univariate data analyses rather than multivariate analyses are emphasized in this reference material.


26

1.4.2. Procedures1.4.2. Procedures1.4.2. Procedures

•• CharacteristicsCharacteristics–– StandardizedStandardized–– Constant Across SubjectsConstant Across Subjects

•• Major ConsiderationsMajor Considerations–– Experimental TaskExperimental Task–– Instructions and TrainingInstructions and Training–– Selection of SubjectsSelection of Subjects–– Data CollectionData Collection–– Primary Data AnalysesPrimary Data Analyses–– Treatment of Human SubjectsTreatment of Human Subjects

The keys to setting up procedures in an experiment are standardization and consistency in procedures across subjects. The task to be completed should be the same for each subject. Instructions and training should be written out and recorded for each subject so that everyone gets the same information. For example, recorded instructions should be played while the subjects are reading them so that they are forced to go from the beginning to the end at a constant rate. The selection of subjects should be representative of the subjects in the population of interest. Data collection should be systematically stored for accurate future reference. One should keep back ups for all data collection. And, the primary data analysis should be planned before data collection begins.

Treatment of human subjects is very important, because all human factors experiments are conducted using human subjects. Subjects should not be endangered physically or mentally during their participation. Since subjects are volunteers, they have the right to withdraw from the experiment at any time.


27

1.4.3. Protection of Human Subjects1.4.3. Protection of Human Subjects1.4.3. Protection of Human Subjects

•• Major ConcernsMajor Concerns–– Subject RiskSubject Risk–– Right to WithdrawRight to Withdraw–– PaymentPayment–– ConfidentialityConfidentiality

•• Institutional Review Board (IRB) ApprovalInstitutional Review Board (IRB) Approval–– Expedited vs. Full IRB ReviewExpedited vs. Full IRB Review

•• Components of IRB Review PackageComponents of IRB Review Package–– IRB Submittal FormIRB Submittal Form–– Description of Research ProceduresDescription of Research Procedures–– SubjectSubject’’s Informed Consent Forms Informed Consent Form

Major concerns in the protection of human subjects include subject risk, the right to withdraw, payment plans, and maintenance of confidentiality. For example, refer to subjects by number rather than name in data collection sheets to insure anonymity. Subjects in human factors experiments are often paid for their participation. If so, the researcher should be careful that payment does not interfere with the subject’s right to withdraw.

Often an Institutional Review Board (IRB) assesses the level of subject risk during an experiment. If so, one must have IRB approval to proceed with the experiment. Two types of review are expedited and full IRB review. Most human factors research requires only expedited IRB review, because subjects are at low risk. If, however, minors are used as subjects or invasive procedures such as blood testing is involved in the human factors research, full IRB review is required. The standard IRB review components include an IRB submittal form, description of research procedures, and the subject’s informed consent form.


28

1.4.4. Equipment1.4.4. Equipment1.4.4. Equipment

•• Ordering EquipmentOrdering Equipment•• Types of EquipmentTypes of Equipment

–– Commercial EquipmentCommercial Equipment–– Modified EquipmentModified Equipment

•• Equipment Operation and MaintenanceEquipment Operation and Maintenance•• Equipment ChecklistEquipment Checklist•• Equipment DriftEquipment Drift•• Backup EquipmentBackup Equipment

Sometimes equipment must be ordered and can delay the start of an experiment if ordering time is not considered. Both commercial and modified equipment is used in human factors research. The equipment must be set up the same way each time and must be maintained to avoid failure in the middle of data collection. An equipment checklist should be used to enforce consistency.

One must be careful of equipment drift where equipment settings or resolution could change over time as the equipment is used repeatedly. Analog equipment is more sensitive to equipment drift than digital equipment. So, sufficient warm-up period should be allowed for analog equipment before commencing data collection. If possible, the researcher should have backup equipment to avoid delays resulting from equipment failure.


29

1.4.5. Pretesting1.4.5. 1.4.5. PretestingPretesting

•• Key ActivityKey Activity•• PurposePurpose

–– Level of Independent VariableLevel of Independent Variable–– InstructionsInstructions–– Equipment OperationEquipment Operation–– Completion TimeCompletion Time–– Data RecordingData Recording

•• ProcedureProcedure–– Art vs. ScienceArt vs. Science–– Number of SubjectsNumber of Subjects–– Experimental DesignExperimental Design

•• Discuss Research Plan with ColleaguesDiscuss Research Plan with Colleagues

Pretesting is the most important aspect of setting up an experiment, but it is often overlooked or minimized. The purpose of pretesting is to check the levels of the independent variables to determine if they are appropriate. Instructions must be tested, because subjects may interpret instructions differently from the experimenter’s intention. Equipment operation and completion time should also be pretested, because each subject will work at a different pace. Pretest data recording so that it is reliable and unbiased to insure that no data will be lost.

Pretesting is more of an art than a science. It takes experience and knowledge of the problem area. The pretesting procedure is really not set. The number of pretest subjects varies for each experiment. One subject is definitely not enough, so at least several subjects should be used. There are no formal experimental designs for pretesting. Usually one picks a treatment combination where a large difference is expected to check if these differences exist and if adjustments are needed. Finally, it is quite helpful to discuss plans with research colleagues who have experience with collecting data in a similar environment. They can provide good advice and insights on the pending experiment.


30

1.5. Research Design Alternatives1.5. Research Design Alternatives1.5. Research Design Alternatives

•• Types of Experimental DesignsTypes of Experimental Designs–– QuasiQuasi--Experimental DesignsExperimental Designs–– Randomized Experimental DesignsRandomized Experimental Designs

•• Randomized Experimental Design AlternativesRandomized Experimental Design Alternatives–– TwoTwo--Group DesignsGroup Designs–– Basic ANOVA DesignsBasic ANOVA Designs–– Advanced Experimental DesignAdvanced Experimental Design

Experimental designs provide plans for the systematic collection of data under managed conditions as compared to making only passive observations. Cook and Campbell (1979) describe two general categories of experimental designs, quasi-experimental designs and randomized experimental designs. The distinction between them is determined by the existence of experimental control and random assignment of subjects. Quasi-experimental designs may or may not specify control conditions to manipulate in an experiment and do not have random assignment ofsubjects to treatment conditions. Randomized experimental designs have controls built into the design and also have random assignment of subjects to treatment conditions.

This reference material concentrates on randomized experimental designs, because they provide the most valid data for causative inferences and the most generalizable results. These experimental designs extend from two-group designs, to basic factorial ANOVA designs, to advanced experimental designs. Most human factors researchers use basic factorial experimental designs due to the nature of their research problems.


31

1.6. Analyzing Results1.6. Analyzing Results1.6. Analyzing Results

•• Data CollectionData Collection–– Systematic ProcedureSystematic Procedure–– Check Data RecordingCheck Data Recording

•• Data ReductionData Reduction–– Raw DataRaw Data–– Collapsing DataCollapsing Data

•• Data ObservationData Observation–– Data PlotsData Plots–– Descriptive StatisticsDescriptive Statistics–– OutliersOutliers

•• Statistical AnalysesStatistical Analyses–– Parametric vs. Nonparametric AnalysesParametric vs. Nonparametric Analyses–– Primary vs. Supplemental AnalysesPrimary vs. Supplemental Analyses

The experimenter should think about the major analyses before data collection to help in choosing the most appropriate design. Have some checks and balances built into data collection to insure accuracy. Usually some data reduction is necessary before conducting statistical analyses. Always keep your raw data at least until the report is written. One can always collapse data, but one cannot return to raw data after collapsing if secondary analyses should require using raw data.

Before conducting any statistical analysis, plot the data to determine if the expected differences seem to exist and the data are coded correctly. Looking at the results before analysis helps in interpreting the statistical analysis. Use descriptive statistic like means or variance in data plots. A good rule is never discard a data point unless one has clear documentation that it is an outlier and not a true reflection of subject variability.

Parametric analyses have certain parameters that define the statistical analysis and have certain assumptions about the type and distribution of scores that are not needed in less powerful nonparametric analyses. Primary analyses are the major analyses that were planned before data were collected. Supplemental analyses aid in interpretation and are often based on nonparametric analysis of demographic data or ratings.


32

1.7. Research Reports1.7. Research Reports1.7. Research Reports

•• 1.7.1. Scientific Publications1.7.1. Scientific Publications•• 1.7.2. Major Components of Reports1.7.2. Major Components of Reports•• 1.7.3. Additional Considerations1.7.3. Additional Considerations

No piece of research is really complete until it is reported. Researchers have an obligation to their scientific colleagues to report their findings. A written report is most common, but reporting can also be an oral presentation. Several types of scientific publications are used, but each of them have major components in common while differing in other special sections and considerations.


33

1.7.1. Scientific Publications1.7.1. Scientific Publications1.7.1. Scientific Publications

•• Variety of Human Factors PublicationsVariety of Human Factors Publications–– Technical ReportsTechnical Reports–– Journal ArticlesJournal Articles–– Proceedings PapersProceedings Papers–– Books and Book ChaptersBooks and Book Chapters

•• Publication CharacteristicsPublication Characteristics–– DifferencesDifferences

–– LengthLength–– Editorial ReviewEditorial Review–– Manual of StyleManual of Style

–– SimilaritiesSimilarities–– Scientific PublicationScientific Publication–– Major ComponentsMajor Components

There are a variety of human factors publications. Technical reports are reports that are completed in the individual laboratory and submitted to research sponsors. Journal articles are publications that add to the archival scientific literature either in printed or electronic format. Proceedings papers are presented at scientific meetings like the HFES conference and are commonly published in CD-ROM format. Books and book chapters are part of the basic scientific literature.

Types of publications differ in length. Usually proceedings papers are the shortest in length while technical reports are the longest. Generally, journal articles receive the most rigorous editorial review. Publications also differ in style. A technical report might have an executive summary that is not often seen in a journal article. Style elements depend on the publication. However, all scientific publications usually have four similar major components including an introduction, method, results, and discussion section.


34

1.7.1. Scientific Publications (Cont'd)1.7.1. Scientific Publications (Cont'd)1.7.1. Scientific Publications (Cont'd)

•• General CharacteristicsGeneral Characteristics–– Objective ReportingObjective Reporting–– Researcher OpinionsResearcher Opinions–– Often Restricted LengthOften Restricted Length–– Active VoiceActive Voice–– Third Person NarrativeThird Person Narrative

•• General Flow of Scientific ReportsGeneral Flow of Scientific Reports–– Story MetaphorStory Metaphor–– Major ComponentsMajor Components

–– IntroductionIntroduction–– MethodMethod–– ResultsResults–– DiscussionDiscussion

Scientific publication is characterized by objective reporting, and researchers’ opinions are restricted to designated sections. The results section is the objective reporting of results and data analysis. The discussion section presents the researcher’s opinions and interpretation of the results. There is often a restriction on length of the publication. Active voice is used to make it more interesting instead of a passive voice. Historically, the third person is used as opposed to first person. However, some journals are now allowing the use of first person narrative.

The general flow of the scientific report follows a story metaphor. Each section of the report helps tell the scientific story. There are four major components. Each section has a unique purpose, but these sections are integrated. The introduction tells readers the purpose of the research, its context in the scientific literature, and why they should read the report. The method tells readers how the data were c

Documents

Human Factors Experimental Design and Analysis Reference · 2011. 5. 14. · i Army Research Laboratory Aberdeen Proving Ground, MD 21005-5425 ARL-RP-0186 July 2007 Human Factors