View
226
Download
1
Tags:
Embed Size (px)
Citation preview
Effect of motion on Effect of motion on direction discriminationdirection discrimination
• Small error bars. Error variance is really low.Small error bars. Error variance is really low.
• Scientists of such disciplines can explain Scientists of such disciplines can explain most of their variancemost of their variance
0
0.5
1
1.5
2
2.5
3
3.5
4
Low motion condition High motion condition
Effect of similarity of Effect of similarity of objects on categorizationobjects on categorization
• Larger error bars. Error variance is still low.Larger error bars. Error variance is still low.
• Scientists of such disciplines can explain a Scientists of such disciplines can explain a lot of their variancelot of their variance
Low similarity High similarity0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
Effect of proximity on group Effect of proximity on group performanceperformance
• Very large error bars. Error variance is high.Very large error bars. Error variance is high.
• Scientists of such disciplines can explain only a small Scientists of such disciplines can explain only a small percentage of variancepercentage of variance
• Note. Mean remains the sameNote. Mean remains the same
Low Proximity High proximity0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Percentage of variance in Percentage of variance in usability?usability?
LargeLarge
MediumMedium
SmallSmall
VariableVariable
Functions of experimental Functions of experimental DesignDesign
• Reduce the error variance, so that you Reduce the error variance, so that you can get focus on the question that you can get focus on the question that you are interested inare interested in
• Experimental statistics make Experimental statistics make assumptions about the data, assumptions about the data, experimental design helps you keep experimental design helps you keep statistically hones.statistically hones.
First rule of statisticsFirst rule of statistics
•Garbage in, garbage out!Garbage in, garbage out!
The most sophisticated statistical The most sophisticated statistical techniques cannot save you if your techniques cannot save you if your design in flaweddesign in flawed
What is experimental What is experimental designdesign
• The design is the general structure of The design is the general structure of the experiment. the experiment.
• Experimental design and the research Experimental design and the research problem are really two separate problem are really two separate things. things.
• Experiment design is not inevitable, Experiment design is not inevitable, i.e., every research problem can be i.e., every research problem can be investigated in many ways.investigated in many ways.
An example An example
• Research question: Can people create better Research question: Can people create better presentations with MultiPoint than presentations with MultiPoint than PowerPointPowerPoint
• Independent Variable: Presentation tool.Independent Variable: Presentation tool. 2 Levels: MultiPoint and PowerPoint2 Levels: MultiPoint and PowerPoint
• Dependent Variable: Quality of presentation.Dependent Variable: Quality of presentation. Quality defined by independent judgment of Quality defined by independent judgment of
presentationpresentation
Between Subjects DesignsBetween Subjects Designs
• Two group, between subjects designTwo group, between subjects design
• Method: Call two groups into the lab, Method: Call two groups into the lab, randomly assign them to groups. Some randomly assign them to groups. Some use PowerPoint, some use MultiPoint. use PowerPoint, some use MultiPoint.
• Results:Results: PowerpointMultipoint2 43 51 65 36 75 8
Within Subjects DesignsWithin Subjects Designs• One group, within subjects designOne group, within subjects design
• Method: Call one group into the lab, randomly Method: Call one group into the lab, randomly assign them to different orders of using assign them to different orders of using PPoint and MPoint PPoint and MPoint
• Results:Results: PowerpointMultipoint2 43 51 65 36 75 8
Design: AB design
AB DesignAB Design
• Takes care of Fatigue and Learning Takes care of Fatigue and Learning EffectsEffects
• Variations: ABBA designVariations: ABBA design
Group 1 Group2Order 1: MultiPoint PowerPoint
Order 2: PowerPointMultiPoint
• The main function of experimental design The main function of experimental design isis to maximize the effect of systematic variance to maximize the effect of systematic variance
(the independent variable)(the independent variable) to minimize the error variance (often to minimize the error variance (often
individual differences)individual differences)
• Error Variance can be of different kinds: Error Variance can be of different kinds: measurement errormeasurement error random differences between experimental random differences between experimental
groups (do your best to eliminate this)groups (do your best to eliminate this) Individual differences within group (not much Individual differences within group (not much
you can do)you can do)
Review: The Design of Review: The Design of ExperimentsExperiments
• Sources of variances in an Sources of variances in an experimentexperiment
• NotationNotationExperimental ManipulationSystematic Variance
Error Variance: Due to Individual Differences
Error Variance:Due to random factors
Between Subject Design: Between Subject Design: Sources of varianceSources of variance
Experimental Manipulation:
Individual Differences between users
Random Variance
Within subject designWithin subject design
Experimental Manipulation:Effect of device
Random Variance
Methods to control Error Methods to control Error VarianceVariance
• RandomizationRandomization
• EliminationElimination
• MatchingMatching
• Additional Independent VariableAdditional Independent Variable
• Statistical ControlStatistical Control
Most of these methods address individual differences variance
RandomizationRandomization• Most effective way to control error variance. If Most effective way to control error variance. If
thorough randomization has been achieved thorough randomization has been achieved then experimental groups can be considered then experimental groups can be considered equal in every way, and you are justified in equal in every way, and you are justified in comparing them. comparing them.
• Random SelectionRandom Selection: random selection of units : random selection of units from the populationfrom the population
• Random AssignmentRandom Assignment: Every experimental unit : Every experimental unit (subject or material) has an equal chance of (subject or material) has an equal chance of being in every condition.being in every condition.
Random Assignment with Random Assignment with controlcontrol
• Sometime you cannot totally Sometime you cannot totally randomly assignrandomly assign
• Example: you need to control for Example: you need to control for gender in the PPoint / MPoint studygender in the PPoint / MPoint study Randomly assign from within the Randomly assign from within the
gender groups to the Multipoint / gender groups to the Multipoint / PowerPoint groupsPowerPoint groups
Elimination or ConstancyElimination or Constancy• Sometimes you can control a possible confounding Sometimes you can control a possible confounding
by eliminating it (or eliminating its effect). by eliminating it (or eliminating its effect). Choose the subjects so that they are equal on the Choose the subjects so that they are equal on the
confounding variable.confounding variable.
• Example: PPoint and MPoint experimentExample: PPoint and MPoint experiment Important confounding variable: previous experience in Important confounding variable: previous experience in
using Powerpoint. using Powerpoint. Solution: You could decide that you are only interested in Solution: You could decide that you are only interested in
people who have done between ten and 20 PPoint people who have done between ten and 20 PPoint presentations before and eliminate everyone else from the presentations before and eliminate everyone else from the sample.sample.
Elimination contd.Elimination contd.
• By reducing the difference between By reducing the difference between participants, we reduce the size of participants, we reduce the size of random variance and increase random variance and increase chances of finding meaningful chances of finding meaningful differences.differences.
• Problems with method: Loss of Problems with method: Loss of generalizabilitygeneralizability
MatchingMatching• Match subjects to variable which is substantially Match subjects to variable which is substantially
related to the dependent variable. related to the dependent variable.
• Example: Previous multimedia experience can act Example: Previous multimedia experience can act as a confounding variable in the study. as a confounding variable in the study. People who have multimedia experience will tend to do People who have multimedia experience will tend to do
better presentations, and use Mpoint more effectivelybetter presentations, and use Mpoint more effectively Solution: Match subjects. Recruit 10 subjects, get a Solution: Match subjects. Recruit 10 subjects, get a
baseline measure of their multimedia skills. Try to make baseline measure of their multimedia skills. Try to make sure that mean multimedia skill in equal in both groups. sure that mean multimedia skill in equal in both groups.
Problems with Matching: reduces availability of subjectsProblems with Matching: reduces availability of subjects
Note. The above example pertains to a between subjects design
Confounding variables that Confounding variables that act differently at different act differently at different
levels of the IVlevels of the IV
• Pose a much bigger problemPose a much bigger problem
• Example: Example: Multimedia experience will benefit Multimedia experience will benefit
MPoint usage more than PPoint usageMPoint usage more than PPoint usage
Such a variable can lead to statistically Such a variable can lead to statistically uninterpretable resultsuninterpretable results
Additional Independent Additional Independent VariableVariable
• If the confounding variable actually interests If the confounding variable actually interests you, you can add it into your designyou, you can add it into your design
• Example: Can people with Multimedia Example: Can people with Multimedia experience use Mpoint better than people experience use Mpoint better than people without multimedia experience.without multimedia experience. 22ndnd Independent Variable: Previous experience Independent Variable: Previous experience
with Multimedia.with Multimedia.
Design: 2 x 2 design.Design: 2 x 2 design.