Methodology Matters: Doing Research in the Behavioral and Social Sciences ICS 205 Ha Nguyen Chad Ata

Methodology Matters:Doing Research in the Behavioral and Social Sciences

ICS 205

Ha Nguyen

Chad Ata

The way we do research matters Because the meaning of the discovered evidence is

based on the way it is obtained 3 domains are always involved when doing

research– Substantive content of interest– Conceptual ideas that give meaning to the content– Methodological techniques for conducting research

3 important keywords– Relations which are between elements within a context

or embedding system

The Substantive & Conceptual Domains

Substantive Domain– Elements = Phenomena– Relations = Patterns of phenomena– Studying the “actors behaving towards objects

in context”

Conceptual Domain– Elements = Properties of the actors’ behaviors...– Relations = Interconnections between elements

The Methodological Domain

Elements = methods = Modes of Treatment– Includes techniques for

• Measuring• Manipulating

– Giving instruction to participants– Imposing constraints on the environment– Selecting materials for use– Giving feedback about prior performances– Using experimental confederates

• Controlling• Distributing impact

… Methodological Domain…

Relations = Comparison Techniques– Involve 3 features of the system under study

• Dependent variables or features– These are measured

• Independent variables or features– These are manipulated

• All of the rest of the system’s features– These are relevant but not the focus of the study

– Comparison performed to assess covariation or association between dependents and independents

Levels Substantive Conceptual Methodological

Elements Phenomena Properties Modes of Treatment

Relations Patterns Relations Comparison Techniques

Contexts Ongoing Systems

Conceptual Systems

Research Strategies

The Domains

Research Methods Are Like Tools

Each method offers opportunities not available by other means

Each method also has inherent limitations– Therefore you should use multiple methods in

order to counterbalance the flaws in each individual method

• Make sure to chose methods such that one strength can compensate for another’s weakness

The Research Methods

Experimental Simulation

FieldExperiment

FieldStudy

ComputerSimulation

FormalTheory

SampleSurvey

JudgmentStudy

LaboratoryExperiment

Abstract

UnobtrusiveObtrusive

Concrete

I

II

III

IV

GeneralizabilityMaximum

PrecisionMaximum

RealismMaximum

Quadrant I: Field Strategies


FieldExperiment

FieldStudy

ComputerSimulation

FormalTheory

SampleSurvey

JudgmentStudy


Abstract


Concrete

II

III

IV


PrecisionMaximum

RealismMaximum

Quadrant I: Field Strategies

Field Study– Natural & unobtrusive

Field Experiment– Manipulating one variable in an otherwise

natural system decrease in realism– A bit obtrusive

Quadrant II: Experimental Strategies


FieldExperiment

FieldStudy

ComputerSimulation

FormalTheory

SampleSurvey

JudgmentStudy


Abstract


Concrete

IIII

IV


PrecisionMaximum

RealismMaximum

Quadrant II: Experimental Strategies

Laboratory Experiment– Concocted system

• Increased precision

• Decreased realism & generalizability

Experimental Simulation– Similar to lab experiment, but

• strives to increase realism

Quadrant III: Respondent Strategies


FieldExperiment

FieldStudy

ComputerSimulation

FormalTheory

SampleSurvey

JudgmentStudy


Abstract


Concrete

I

II

IV


PrecisionMaximum

RealismMaximum

Quadrant III: Respondent Strategies

Sample Survey– Sampling of the population high generalizability

– Low precision

Judgment Study– Here the focus is on the stimulus material, rather than

respondents’ attributes High precision

– “actors of convenience” Low generalizability over the population

Both often done in neutral settings low realism

Quadrant IV: Theoretical Strategies


FieldExperiment

FieldStudy

ComputerSimulation

FormalTheory

SampleSurvey

JudgmentStudy


Abstract


Concrete

I

II

III


PrecisionMaximum

RealismMaximum

Quadrant IV: Theoretical Strategies

Formal Theory– Formulating general relations between variables of

interest high generalizability– No concrete system involved low in realism– No observation low precision

Computer Simulation– Like experimental simulation, except human

participants are not included– Modeling the real world high realism– Outcomes are predictable low precision– Modeling a particular system low generalizability

Questions to keep in mind…

Does the presented material coincide with the strengths & weaknesses of the research strategies that were used?

Is the research evidence well supported by a single or by multiple research strategies?– Are the strategies’ weaknesses and strengths

offset such that they counterbalance one another?

Study Design For every empirical study, observations

have to be gathered, combined, and compared– Ex: Given a pair of properties, X and Y, can

we tell if X caused Y?• X is independent variable, Y is dependent variable

– What techniques are available for comparison?• Baserates

• Correlation

• Difference

Comparison Technique #1

Baserates – How often (at what rate, or what proportion of

time) does Y occur?• If I do not know how often Y occurs in the general

case, then I cannot decide whether the rate of Y in the study is significant (particularly high or low).


Difference Question– Is Y present under conditions where X is

present? Is Y absent when X is absent?


Correlational Question– Do the values of property X covary with the

values of property Y?• If X is high, is it likely that Y will also be high?• If X is low, is it likely that Y will also be low?

– High positive correlation, high negative correlation, little or no correlation

This technique tells you whether X and Y go together but it does not help in determining causality.

Randomization

Randomization – using a random assignment procedure to allocate “cases” to “conditions”– Each case must be equally likely to end up in

any given condition– This procedure does not guarantee to rule out

all other factors that could affect the results.

Validity of Findings

4 types of validity– Internal validity

• How close can you come to asserting that the presence of X caused the altered level of Y?

– Statistical conclusion validity• Is the given result due to chance?

– Construct validity• How well defined are the theoretical ideas in the study?

– External validity• How well will the results hold up for replication and how

generalizable are the results?

Measure Techniques

Categorized by two questions– Who makes the record of the behavior?– Is the participant aware that his/her behavior

is being recorded for research purposes?

6 Major Types of Measures

Participant Is Aware Participant Is Not Aware

Self-Report Trace Measure

Observation w/Visible

Observer

Observation w/Hidden Observer

Records of Public Behavior

Records of Private Behavior

Strengths and Weaknesses of the Measures Self-reports

– Strength - Low costs, does not require a lot of time to create and use

– Weakness - Reactive

Trace measures– Strength – unobtrusive– Weakness – not very versatile, time consuming

to gather

Strengths and Weaknesses of the Measures Observations

– Strength – relatively versatile

– Weakness – can only be used on overt behavior, vulnerable to observer errors, can be reactive for visible observers, possible ethical issues for hidden observers

Archival records– Strength – good for researching the past, studies for

extensive periods of time

– Weakness – public records can be reactive, versatility is limited

Techniques for Manipulating Variables Selection – selecting cases with desired

values and assigning them to appropriate conditions

Direct Intervention – manipulating the specific variable directly

Techniques for Manipulating Variables Inductions

– 3 types• Misleading instructions

• False feedback

• Experimental confederates

– Ethical issues

Concluding Comments

Results depend on methods. All methods have limitations. Therefore, any set of results is limited.

It is not possible to maximize all desirable features of different methods in any one study.

Each set of results must be interpreted in relation to other evidence relating to the same questions.

Documents

Methodology Matters: Doing Research in the Behavioral and Social Sciences ICS 205 Ha Nguyen Chad Ata