Research Experience for Teachers (RET)

Research Methods

Dr. Randa L. Shehab

Dr. Chen Ling

RET 2006 Summer 2

Overview Research Methodology

Problem Formulation Development and Measurement of

Performance Criteria Experimental Design Data Collection and Analysis Presentation of Results

Technical Writing and Presentation

RET 2006 Summer 3

The Scientific Method A systematic approach to answering

questions using empirical investigation Steps

Form a hypothesis - an idea that can be tested through observation or experimentation

Collect data through observation Analyze the data - test the hypothesis “Do not reject” or “reject” the hypothesis Confirm results by repeating the experiment

RET 2006 Summer 4

The Research Environment Naturalistic Observation

Systematic, detailed observation of behaviorPurely descriptive, no causality

Laboratory ResearchArbitrary environment where elements in

environment can be controlledControl allows statements about causality

RET 2006 Summer 5

Basic Research Acquisition of knowledge for developing

or validating a theory Not necessarily for practical application Used to set forth general design

principles Examples

Nature of human performance?How is performance affected by variable(s)?Relationship between variables? Impact on system performance?

RET 2006 Summer 6

Applied Research Finding answers to practical problems Highly specific hypothesis Absence of general knowledge or theory

advancement Used to evaluate specific design alternatives Examples

Effect of technology on human performance? Effect of behavioral variables on use of technology? Most effective behavioral/technological variable to

achieve desired performance?

RET 2006 Summer 7

Why Research? To understand behavior and answer

questions about people, environments, systems

To evaluate the research of othersWe should change the system to make it more

effective…Determine if research is done correctly and if it

applies To conduct research of your own

Research is a tool for answering questions

RET 2006 Summer 8

Learning to Perform Scientific Research

Scientific Research is Unique Acquire Research Experience Perform Solid Literature Review Use Proper Tactics and Strategy Avoid Mistakes of Other Investigators

RET 2006 Summer 9

General Scientific Method: Steps

Development of Hypothesis Body of Existing Knowledge Define the problem

Controlled Experimentation Observation of Phenomena

Scientific curiosity

Quantification of Observations Analyze the problem

Logical Analysis Test of Hypothesis

RET 2006 Summer 10

Research Experiments

Experiment: A series of controlled observations taken in an artificial situation with deliberate manipulation of variables to answer one or more specific hypotheses.

Controlled ObservationsConditions and eventsMust be repeatable!

Artificial SituationAllows accurate observations, but might affect

behavior

RET 2006 Summer 11

Research Experiments

Manipulation of VariablesSystematically vary conditions to see effectsEliminate extraneous factors to determine

causes Specific Hypotheses

Allows clear definition of the experimental plan Ideas can be obtained from observation

“Effects of (independent variable) on (measured variable)”

“Comparison of the effects of (different technologies / products) on (performance measure)”

RET 2006 Summer 12

Developing a Hypothesis A hypothesis states an idea that can be tested

through experimentation or observation The hypothesis proposes an explanation for

some observed phenomenon

Initial Interest Working

Hypothesis

LiteratureReview Formal

Hypothesis

RET 2006 Summer 13

Types of Hypotheses Working Hypothesis

Preliminary statement based on limited information Informal statement derived from initial observation Subject to modification

Exploration Obtain additional information on the topic Books, journals, related knowledge, collaboration

Formal Hypothesis Precise expression of a predicted relationship between

or among events Capable of being tested Does not change

RET 2006 Summer 14

Key Principles of Experimental Design

Provide a Measure of Random Error Design must contain a measurable estimate of

variation to determine statistical significance Provided through replication

Avoid Systematic Bias Effects due to time-ordering such as learning Counterbalance - perform half the trials using one

sequence and the other half using another Randomize - assign units randomly

Do Not Confound Variables Change only one variable at a time

RET 2006 Summer 15

Defining the Problem Easiest way is to define the variables A variable is a quantity that may

assume any one of a set of values Independent Variable

Variable that is deliberately manipulatedMust have at least 2 levelsForms the basis of the hypothesis

RET 2006 Summer 16

Independent Variables The factor or treatment Presence/Absence

Does the variable impact behavior?

QuantitativeContinuous along a single dimensionMust have enough levels to describe expected

relationshipMust consider range and spacing of levels

RET 2006 Summer 17

Independent Variables Qualitative

Combination of multiple factors not described with a single dimension

Vary one dimension and control the others Select “optimal” values for all variables held

constant Test at range of values to identify the optimal

points

RET 2006 Summer 18

Quantitative Variables Number of Levels

Choose enough levels to get an accurate view of the relationship between the variables

Cost tradeoff when adding more levels Total Trials = (# IV's) (IV levels) (repetitions)

Dep

ende

nt V

aria

ble

Independent Variable

Level 1 Level 2 Level 3

SuspectedRelationship

Actual Relationship

RET 2006 Summer 19

Quantitative Variables Range of Values

Wide enough to cover region of interestStart large and refine (pilot study)

SpacingShould match phenomenon being investigatedEqual: Better for statistical analysisUnequal: Often more representative of

system response (e.g., decibels)

RET 2006 Summer 20

Qualitative Variables Fundamental Principle

Select the best levels of all other variables (usually quantitative) to be held constant, then use the best combination at each qualitative level

Alternatively, test across a broad range of values to locate optimum levels

Example When comparing

control shapes, select the best size, texture, etc. for each shape

Compare Round A with Square C

De

pe

nd

en

t V

aria

ble

Knob Size

A B C

Round Square

RET 2006 Summer 21

Dependent Variables The criterion measure Variable that is measured Value “depends” on the value of the

independent variable Selection of the criterion determines the

outcomeDoes it answer the question? Is it relevant to the task?

RET 2006 Summer 22

Control Variables Variables that are held constant because they

may affect the results Avoid Confounding

Uncontrolled influences on experimental results

Interpret results within a narrow limit to determine causality

Examples:Time of dayAttitudes/expectation Instruction

RET 2006 Summer 23

Variable Examples Effect of Alcohol on Reaction Time

Independent Variables Alcohol Level / Consumption (0%, .08%, 1.2% BAC) Gender (Female, Male)

Dependent Variables Errors Processing Time Motor Time

Control Variables Food Intake Environmental

RET 2006 Summer 24

Characteristics of Good Performance Measures

Appropriate Level of Detail Measure appropriately reflects differences to detect Adequate precision and range to detect differences

Reliability Degree of repeatability (can be expressed as a

correlation coefficient) Interrater reliability – the degree to which multiple

observers agree when scoring the same event

Validity Degree to which a measure actually measures what it

is supposed to measure Ensures a measure tells you “what” your data mean

and that you have selected an appropriate response variable

RET 2006 Summer 25

Characteristics of Good Performance Measures

Sensitivity Can detect changes in the behavior of interest

Accuracy More precise than the phenomena being measured Minimize measurement error Non-Intrusiveness

Collection method does not affect performance Observer should not be a distraction Data collection should not interrupt the task (i.e., filling out

forms) Measuring device should not attract participant's

attention Implementation Requirements

Practical with respect to time, budget personnel requirements, ease and quality of collection, and measurement robustness

RET 2006 Summer 26

Populations and Samples

Sampling Methods Random Sampling Stratified Sampling Matched Sampling

Population: allrelevant cases

Sample: subsetof the population

Fortuitous Sampling Proportionate Stratified

RET 2006 Summer 27

Random Sampling Randomly select members of population

IndependentRepresentative of the population for

large sample sizesReduces systematic bias

Randomly assign subjects to conditions How would you randomly sample n=20

to study ability of drivers to localize warning sirens?

RET 2006 Summer 28

Fortuitous SamplingUse whatever is availableDoes not guarantee independenceNot representative of the

population How would you fortuitously sample

n=20 to study ability of drivers to localize warning sirens?

RET 2006 Summer 29

Stratified Sampling Identify critical subgroups of the

population Randomly select members for subgroup

samples based on similar critical characteristics

Mirrors population characteristics Accurate information is not always

available How would you develop a stratified

sample of n=20 to study ability of drivers to localize warning sirens?

RET 2006 Summer 30

Proportionate Stratified Sampling Identify critical subgroups of the

population Randomly select members for subgroup

samples until proportionate to the population characteristics

Mirrors population characteristics and count

How would you develop a proportionately stratified sample of n=20 to study ability of drivers to localize warning sirens?

RET 2006 Summer 31

Matched Sampling Experimental samples are identical with the exception of

the independent variable Controls for extraneous variables Allows strong conclusions to be made about any significant

differences Equating groups

Precision matching - match identical participants Range matching – match participants within a range Average matching - match average group score

How would you develop a matched sample of n=20 to study the effects of different work hardening programs on material handling endurance?