Methoden wetenschappelijk onderzoek

Setting up research

This lecture

•  What are experiments? •  How do we prepare an experiment?

•  How do we propose it? •  How do we design it?

•  Stuff to do before you start

Experimental Science (1)

•  Science in which the subject is small and simple enough to be manipulated directly – Physics – Psychology/cognitive science –  (Experimental) biology – Human-computer interaction – Machine learning, pattern recognition

Experimental Science (2)

•  Create an experimental setting •  Manipulate one input variable •  Control “all” other input variables •  Measure one or more output variables

•  How to set this up is the topic of experimental design

Experimental Science (3)

•  Physics: materials testing •  Psychology: relation between age and

memory •  Biology: testing the effect of fertilizer •  Human-computer interaction: testing the

ease-of-use of different interface designs •  Pattern recognition: testing the success

with different parameter settings

Observational Science

•  Sometimes a subject is too big to control – Astronomy – Website traffic

•  We can still investigate this – Descriptive/classification studies – Observational studies: find similar cases – Simplification to an experiment

Qualitative or quantitative?

•  Quantitative analysis is prototypical science –  Many repeated measurements –  Making graphs –  Doing statistics –  Etc.

Quantitative analysis •  This is where you investigate a well-defined topic

precisely (quantitatively) –  Quantitative description (think demography) –  Hypothesis testing

•  This is generally specialized and detailed –  In the sense that there tend to be established ways of

doing it

•  A qualitative analysis made beforehand prevents you from measuring nonsense

Towards quantitative analysis

•  Before one can do a quantitative analysis, one needs to know what is going on –  This is the domain of qualitative analysis

Qualitative analysis (1)

•  Qualitative analysis is an important first step in experimental design – Often, researchers use existing literature

•  What can I control? •  What should I measure? •  How should I measure it? •  Which values can I expect?

Qualitative analysis (2) •  Sometimes (and perhaps more often in practical

situations) there is no ready-made information •  In that case finding answers can be a difficult creative

process –  Familiarize yourself with your subject –  Play with it –  Let others (your students!) play with it

•  This may be part of the pilot experiment(s) •  Or you may do case studies

•  Such work is often descriptive in nature

Quantifying your subject •  In order to make a quantitative analysis, you

need to quantify your subject –  You need to convert aspects of interests to

measurable quantities

•  Categorical – just categories (e. g. man/woman) •  Ordinal – categories with an order (e. g. ranks) •  Interval – distance is meaningful (e. g. position) •  Ratio – distance and true zero (e. g. mass)

Measurements

•  Knowing what to measure is part of how you operationalize your research question – ~How to make it precise

•  But the measurements may be an indirect way of finding out what you want to know! – Temperature ~ energy – Reaction time ~ ease of use – Questionnaire ~ ease of use

Range and distribution

•  Try to get an idea of realistic values •  And how these values are distributed

–  Which ones are frequent or rare

•  Makes it easier to spot errors and bias –  Or to make exciting new discoveries!

•  Range and value are also important for input parameters

Bias (1) •  A bias is a systematic error

–  Results are changed in a predictable direction –  But you may not be aware of this –  Example: thermometer in the sun –  Example: using computer scientists as testers

•  Biases are bad for your research, so you should detect and remedy them! –  Or compensate them, if at all possible –  But this may not always be practical… –  In which case you must be honest about them

•  Sources of bias can be detected in qualitative analysis

Bias (example)

-50 -45 -40 -35 -30 -25 -20 -15 -10 0

1

2

3

4

5

6

7

minimum temperature (degrees centigrade)

num

ber o

f day

s

Minimal temperature in Yellowknife on February days

Bias (2) •  Bias may also be in your mind

–  Observer bias •  You know what you want to find, so you tune your

experiment or your analysis until you find what you want to find –  This happens subconsciously –  Hire assistants to help you –  Let your colleagues monitor you

•  Performance measurements of software are prone to this –  You tune your preferred model precisely –  But not the competition’s model

Noise (1)

•  Noise is random variation of your measurements – No systematic relation between real value and

measured value – Often assumed that the noise is the same for

all underlying real values

Noise (2)

•  What does noise mean? – Often noise models lack of knowledge

•  If we would know initial conditions of a coin flip perfectly, we could predict the outcome

– Especially true in biological/psychological experiments

•  Variability in outcome is deterministic, but we don’t know the relevant parameters.

•  Or we do not have the resources to measure/control them

Noise (3)

•  What about randomness in simulation? –  Randomness is often used to model noise in the real

world –  Or to test sensitivity to initial conditions –  In theory we could derive the relation between the

initial conditions and the outcomes, but in practice this is often intractable

•  This is of course artificially introduced –  But can be treated the same as natural noise

Noise (4)

•  Because noise is independent of the underlying real value, it can be compensated by repeating a measurement sufficiently often

Noise (5)

•  Noise is much less of a problem for experiments than bias –  But it should always be reported –  And therefore measured –  More about this when we talk about statistics and

reporting

•  Important message: just reporting a value or an average is not enough –  One also needs to report about spread

What to do with measurements

•  Always keep track of what and how you measure things – Keep a scientific journal! – Use version control for software – Never throw away or overwrite data files

•  Rather buy a new hard disk

– Store your data in a systematic way •  A data base, or something else that is easy to search

What to do with data

•  Data about which you publish should be public property – Not the same as accessible to anyone – But should be shared with other scientists

•  Some scientists have become famous by building publicly available corpora of data

Corpora of data

•  Existence of corpora means that you do not always have to gather data – Reference data sets make comparison easier – But may limit scope

•  E.g. everybody parses the Wall Street Journal corpus, but how representative is that?

•  You can also use existing corpora to see if the data you gather makes sense

How to set up an experiment

How is science funded? (1)

•  The university pays professors to teach and do research –  But this happens less and less often

•  In order to do research one has to apply for external funding –  From the university –  From national/local grant agencies –  From European grant agencies –  From dedicated funds –  From businesses

How is science funded? (2)

•  Even in companies with research departments, one often needs to apply for funding

•  You need to make a case why your research is to be funded – And given that there are generally more

applications than funds, why not someone else’s

Proposal review (1)

•  Proposals are reviewed like articles – By peer review – But additionally, by a committee of the

granting agency •  In addition to the quality of the proposal,

the quality of the researcher is important – Your CV, your publication record – Success increases success in this game!

Proposal review (2)

•  Funding agencies generally have goals – Promoting a field – Supporting promising young scientists – Supporting innovation –  (sometimes) fair distribution among institutes

•  Your proposal is evaluated for these goals – This is usually the job of the committee

What is in a proposal? (1)

•  Your CV, presented in a way that stresses the elements that are relevant – But this is no different from applying for a job

•  An outline of the research you want to do – The research question – Why this is relevant – The methods you are going to use – A description of the expected outcome – The impact of this outcome

What is in a proposal? (2)

•  A proposal is not much different from a normal research plan – Although usually with somewhat less detail

•  However, it is important to keep in mind the objectives of the funding agency – Try to think how you could do research that

fits their bill

Assignment 2

•  I will ask you to write a proposal summary – The scientific part

•  More about this towards the end of the lecture

Research project setup

•  Without the sales pitch, and with more detail, a proposal becomes the setup of a research project

•  Making this is an important part of science – Prevents you from acting before thinking – Especially in artificial intelligence, there is too

much gung-ho research

Steps in making a setup (1) 1.  Familiarize yourself with the field

–  Normally this has already happened 2.  Define a research question 3.  Operationalize the question 4.  Figure out how to best analyze your data 5.  Get permission to do it 6.  Run a pilot experiment 7.  Fine-tune the setup 8.  Do the experiment

Familiarity with the field

•  Identify open, interesting questions – You want to do something new – But nevertheless connected to what others do

•  Know about standard methods, test sets and analysis methods

•  Know about ways to report your research

•  This you acquire by studying literature – And talking to other researchers!

The research question (1)

•  This is an important creative step –  I cannot give you a recipe

•  Usually starts big – “How can a computer learn language?” – But this is infeasible

•  You have to narrow it down – “How can I use support vector machines to

learn irregular verbs?”

The research question (2)

•  Look at what others use as questions –  But with only small variations, you won’t get famous

•  Keep the underlying question in mind –  Otherwise you may end up investigating an

abstraction, rather than reality •  E.g. formal languages rather than natural language

–  This is less of an issue in application-oriented research

Operationalization (1)

•  Transform your scientific question into an experimental or observational program – What are you going to do? – What are you going to manipulate? – What are you going to measure? – How much data are you going to collect?

Operationalization (2)

•  Again: look carefully at how others have operationalized their research – And build your own design accordingly

•  Finding really new ways to do research is really difficult – This often leads to new branches of science

•  e. g. Brain imaging techniques •  e. g. Artificial life

Operationalization (3)

•  Behavior is variable and measurements contain noise – Therefore, one wants to do repeated

experiments – But how best to do this? – Resources are limited

•  This is a whole field of study!

Suggestions (1)

•  There is an important difference between experiments with (computer) models and humans – Humans have many more unknown

behaviors, so must be more carefully controlled

– E.g. humans learn or get tired in sequences of experiments: first may be different from last

Suggestions (2)

•  Performance comparison of systems – “Benchmark” – Use standard problems/data sets – Repeat the experiment to establish variation

•  With different random seeds/initial states

•  But what do we learn? – Only which system is best – Not necessarily how to improve them further

Suggestions (3) •  We can change a parameter

–  And see how this influences behavior –  Parameters can be continuous or discrete

•  Terminology –  What you control = independent variable –  What you measure = dependent variable

•  To find out how the system works, and how it can be improved

•  Examples –  Biology: add fertilizer –  Psychology: background music and reading speed –  Computers: number of neurons and generalization

•  Compare groups with different values of the independent variable

Suggestions (4)

•  We can change multiple parameters – Parameters may be independent (total effect

is sum of individual effects) – Usually, parameters interact

•  Combinations grow exponentially: kN

– k = number of steps – N = number of parameters

•  This may be a practical problem

Suggestions (5) •  If you do experiments with humans, other factors

play a role –  Randomization becomes important –  Because it is expected that systematic differences

between humans will have systematic effects •  You have to assign you subjects randomly

•  Example: interface 1 vs. interface 2 –  Don’t first test interface 1 and then interface 2 –  But randomly one or the other

Suggestions (6)

•  Also it is better to do within subject comparisons than between subject comparisons – Between subjects there are all kinds of

unknown differences – Within a subject there is only the controlled

variable •  Therefore you do multiple tests on one

subject

Suggestions (7)

•  In order to reduce variation, you may let one subject do a task multiple times –  But these are not independent data points –  Therefore one preferably averages over data points

•  Example –  Time to find items using a library catalog –  Each subject looks for a list of comparable items

Analysis (1)

•  Ideally, you should decide how to analyze your data beforehand –  Qualitative, explorative research may be problematic –  But in standard experimental designs, it should be

straightforward – look what others have done

•  Deciding on analysis methods beforehand helps you to prevent you fooling yourself –  Try analyses until you find something significant

Analysis (2)

•  Parts of the analysis – Which filtering/smoothing to use – How to summarize your data (averages etc.) – What outcomes to compare – Which statistical tests to use – etc.

•  More about statistics in a later course – Descriptive and inferential

Analysis (3)

•  Ideally the analysis is blind – The person doing the analysis does not know

what the outcome is expected to be

•  But in practice you usually do it yourself

Permission •  Almost any experiment involving humans requires permission

–  To prevent unethical research –  For this purpose, universities have ethical committees

•  Ethical committees evaluate your research –  How do you recruit? –  Who do you recruit? –  What do your subjects do? –  How long does it take? –  Where do they do it? –  Do you record personal information? –  Do you inform your subjects about the purpose of your experiment? –  Do you get their informed consent?

•  They do not check the quality of your design, only if your experiment follows ethical rules.

The pilot experiment

•  Before running large numbers of (expensive) tests, try out your design – Do your measures work? – Do my subjects act as expected? – How large is the effect? – Does my analysis work?

Fine-tuning

•  After the pilot experiment you can fine-tune your experiment – Change details of design and analysis – Estimate the required number of subjects

•  You may potentially need to re-pilot – Especially if you make important changes

Assignment •  Write a proposal summary •  On the topic you chose for your literature study

–  Should contain experiments •  Answer questions:

–  What is your research question? –  Why is this important?

•  With references to related work –  What method will you use? –  What results do you expect?

•  Try to make it convincing! •  Note that it is not read by a specialist in your area! •  http://arti.vub.ac.be/cursus/2012-2013/mwo/form.doc •  http://arti.vub.ac.be/cursus/2012-2013mwo/form.pdf

Deadline November 8 midnight