Experimental methods in empirical research The Rolls-Royce of scientific research? Daniel Gile

D.Gile Experimental res 1

Experimental methods in empirical researchThe Rolls-Royce of scientific research?

Daniel Gile

[email protected]

www.cirinandgile.com

mailto:[email protected]

http://www.cirinandgile.com/


What is good scientific research?

(In one easy lesson – satisfied or no money back)

Many possible definitions, but even the most demanding researchers would probably accept the following:

Good scientific research consists in exploring reality rigorously,

cautiously, in full awareness of human limitations inherent to such exploration, in compliance with social, editorial and

institutional norms of science

taking care to avoid and/or limit and/or measure the probability and/or magnitude of possible errors


Is a specific research method or paradigm part of that definition of good scientific research?


Is experimental research a prerequisite for

scientific research?

Japanese psychologist

Bears in the mountain

Tuna fish migration

Astronomy

Paleontology…


Theoretical vs. Empirical methods

Theoretical research → ideas:

Analysis of facts

Analysis of theories

Empirical research is centered → data:

Collection

Analysis

(Most often for the sake of developing or testing theories)


Naturalistic vs. Experimental research

Naturalistic research:

collecting data

about phenomena as they occur ‘naturally’

Experimental research:

Creating ‘controlled’ environments

to collect data about phenomena in these environments


Why experimental research?

Phenomena which occur naturally are influenced by many factorsso

It is difficult to trace the influence of one particular factor

For example :

The success of a learning processMay depend on a teaching method

(in which the investigator is interested)

But could also be influenced

By the teacher’s personalityBy environmental conditionsBy the student’s motivationBy the time available to study…


Why experimental research? The principles (1)

Something happens over time (from t1 to t2) to the object of study O, which turns into O’

We want to know why and how

We think factor F1is the cause of this change

But perhaps it is F2? Or F3?

How do we find out?

t1 t2

F1

F2

F3

O O’



We think factor F1 is the cause of this changeBut perhaps it is F2? Or F3?

One possibility is to create an environmentWhere O is shielded from F2 and F3

So if O changes, it can only be due to F1This is one type of experimental setup

t1 t2

F1

F2

F3

O O’



Another possibility is to keep F2 and F3 constantand suppress (or change) F1

If nothing happens to O, then it must be F1 which was causing the change

F1

F2

F3

O O’

?F2

F3

O


Why experimental research? The principle (4)

In all these (and other) scenarios, We create a ‘controlled’ environment

in which we shield the object from outside factors or ‘control’ these factors

So as to be able to isolate the influence of whatever factor(a ‘variable’) in which we are interested

from the influence of ‘confounding’ factors

F1O O’


Experimental research - An example

Is translator training method A an efficient booster of translation skills?

If we ‘just’ compare the work of 40 translators, 20 of whom have been trained with A and 20 without it,

Differences in translation performance may be due to A, but also to:

- characteristics of the texts, - age, - experience, - languages involved, etc.

If we experiment with the same textand two groups with similar characteristics

(age, experience, languages etc.)

If differences found, probably due to A


Experiments for hypothesis-testing

This is a prototypical model of hypothesis-testing experiment

The behavior of 2 or more groups is observed

Each is made to be similar to the others except for the value of the variable to be examined

Comparisons of output variables (‘dependent variables’) are made

with a view to determine whether differences are likely to be due to chance or not

If not, they are called ‘significant’.


‘What if’ experiments

Hypothesis-testing experiments are the best knownBut not the only ones

Exploratory experiments are also used:What if…?

(No special hypothesis)

What if I gave a translator this type of reference document?What if I made a translator work in a team with an expert without

any knowledge of the source language?What if I doubled his/her pay when the customer is happy with

the product?

What if…


‘What if’ examples

Caveman – biological clock

Isolated Community of volunteers

Big Brother

Agriculture

Children in a room

Sometimes with hypotheses or theories (formalized)

Sometimes without


Advantages of experiments

Eliminate the ‘confounding’ influence of ‘parasitic’ variables

Often allow to measure some indicative value more accurately than in naturalistic studies

(precise corrections made during translation, pauses etc.)

Allow to create interesting situations which do not occur or occur very seldom in reality

and therefore could not be studied naturalistically

Make it possible to study situations with small investments as opposed to huge investments

which studying the same phenomena with naturalistic methods would entail


Why do experiments have such a high status in science?

- Because of the advantages outlined in the previous slide

- Because often used to test theoriesand therefore take place at an advanced phase of the research

process on a given phenomenon

- Because they often involve sophisticated thinking and design

- Because they most often involve statistics, which is a highly sophisticated tool

BUT DOES ALL THIS MAKE THEM MORE ‘SCIENTIFIC’THAN NATURALISTIC STUDIES?


Limitations of experiments: ecological validity

Most often cited limitation of experiments in the social sciences has to do with ecological validity:

Is the task and are the measurements a valid model for real life?

For example: A translator who translates in a particular room under the

observation of cameras or an experimenter or on a specific computer with a specific type of software or under specific

time constraints or with specific instructions as to the use of data or with the instruction to think aloud while translating

…Does s/he perform the same translation task as in real life?

If not or if the answer is uncertain, can conclusions on real life be drawn from the experiments with the caution prescribed by

science?


Limitations: neglected confounding variables

In experiments, confounding variables are controlled.But are all of them controlled?

- Potentially relevant of which the investigator is not aware?

- Variables which cannot be controlled because there is no way to control them or controlling them is too expensive/impractical?(family traditions, some genetic factors, some personality traits,

economic situation and history, history of social interaction…)

- Variables which cannot be controlled without reducing the sample to an insignificant size?

(only subjects aged 20 to 25, right-handed, with a given personality profile, with parents from a particular background and having a specific economic status, who attended particular schools with particular curricula, who read at least 20 books a year…)


Neglected confounding variables

If we control F2 and F3 but not F4 and F5

Can we safely assume that the effect measured

in the experiment is only due to F1?

The answer is clearly No

t1 t2

F5

F4

F1

F2

F3

O O’


Limitations: experiments as a case study

By ‘controlling’ relevant variables and parameters,For instance all subjects must be of the same age

Or right-handed, or work on one particular source textOr have 5 to 10 years of professional experience

Experimenters deliberately exclude some naturally occurring variability

And it is difficult to say with the appropriate caution prescribed by science

Whether the results would also be replicated with other values

If there are many replications of the experiments with different values (and without bias), fine

But if there aren’t?


Limitations: statistical significance (1)

Inferential statistics (statistical tests) are used to determine whether differences are ‘significant’ or not

Variables in a population have certain values and a ‘distribution’(How tall, how many times go to church, how much money

spend on Belgian chocolate…)

In order to check whether some factor makes a difference(more training, more work, better food…)

Check whether the population with the ‘extra’ has the same distribution of values than the population without the ‘extra’

(or with a certain value in that variable and another value)



How will a statistical test check this?

You will measure the values of a relevant output variable(for instance some metric of translation quality)

On samples from the two populations(the two conditions)

You will find some difference between the mean values

The test will tell you how likely it is that the difference is compatible with the idea that the two populations have the

same distribution for the relevant variable

If it tells you this is unlikely, you will say the difference is ‘significant’



Actually, it will determine, on the basis of measurements made on the sample means

A certain distribution

And the proportion of values which lie between a lower and an upper limit if the two populations are not different

If your value is below a threshold or above another threshold,It will tell you that it corresponds to only 5% or 1% of the cases

still compatible with the idea that the two populations are the same

“significant at .05”“significant at .01”



)



‘Significance’ is determined at a certain level: the experimenter allows him/herself a certain probability of ‘false

positive’ results (of wrongly concluding that the difference is not produced by chance)

Significance at 5% (.05) means that the experimenter allows him/herself to draw the wrong conclusion that the difference is

‘real’ once every 20 times

If you were to decide that something is true or not, would you say that it is true if you have one chance out of 20 to be wrong?

(Significance at 1% means that you could be wrong once in 100 times)



Statistical significance tests rely on a number which is calculated from values obtained from the samples being examined,

with a threshold For instance, if you get 5,62 the difference is significant, if you get

5,63 it isn’t.How do you like this transition from ‘yes’ to ‘no’?

Does it make sense to you?

Statistical significance says the difference is likely to be due (or not to be due) to chance.

It does not say how large the difference is on average.In TS, how useful is knowing that there is some difference, but without

knowing how large it is?


So are experiments not a good research paradigm?

Experimental research can be very useful:

It is potentially powerful in eliminating (some) confounding variables

It does allow observation of situations which would not occur naturally

(but in the social sciences, what about ecological validity?)

It does allow precise measurements which would be difficult or impossible to carry out in real life

It can take on board variability, measure it and overcome it

But it has its limitations, and works best under certain conditions


Conditions for powerful experimental research

Proper design, and in particular proper sampling and piloting

Cautious inferencing from the results

Correct statistics (test that conditions for the use of tests are met)

Many replications with various values of the independent variable

(so that you really cover a range of values and move away from the case-study scenario)

How often are such conditions met in your field?


If conditions are not met

Still useful, but for ideas and tentative results

Experimental results are by no means the final ‘scientific criterion’ for a decision

In particular, ‘significance’ is just a tentative direction

Naturalistic findings can be more powerful

Especially with the use of corporaBecause they allow the use of much authentic data

Can even test hypotheses, more powerfully than experiments!


Conclusion

Experiments are a tool

A powerful tool under certain conditions(and in particular in cognitive psychology)

But only a tool

Not very powerful under other conditions

NOT THE ULTIMATE IN SCIENCE

Don’t let scientific status symbols fool you

Science is in the human mindWhen you explore reality with a rigorous, skeptical, cautious,

systematic mindset,You are ‘scientific’ whatever the tool

Documents

Experimental methods in empirical research The Rolls-Royce of scientific research? Daniel Gile