Methoden wetenschappelijk onderzoek
Setting up research
This lecture
• What are experiments? • How do we prepare an experiment?
• How do we propose it? • How do we design it?
• Stuff to do before you start
Experimental Science (1)
• Science in which the subject is small and simple enough to be manipulated directly – Physics – Psychology/cognitive science – (Experimental) biology – Human-computer interaction – Machine learning, pattern recognition
Experimental Science (2)
• Create an experimental setting • Manipulate one input variable • Control “all” other input variables • Measure one or more output variables
• How to set this up is the topic of experimental design
Experimental Science (3)
• Physics: materials testing • Psychology: relation between age and
memory • Biology: testing the effect of fertilizer • Human-computer interaction: testing the
ease-of-use of different interface designs • Pattern recognition: testing the success
with different parameter settings
Observational Science
• Sometimes a subject is too big to control – Astronomy – Website traffic
• We can still investigate this – Descriptive/classification studies – Observational studies: find similar cases – Simplification to an experiment
Qualitative or quantitative?
• Quantitative analysis is prototypical science – Many repeated measurements – Making graphs – Doing statistics – Etc.
Quantitative analysis • This is where you investigate a well-defined topic
precisely (quantitatively) – Quantitative description (think demography) – Hypothesis testing
• This is generally specialized and detailed – In the sense that there tend to be established ways of
doing it
• A qualitative analysis made beforehand prevents you from measuring nonsense
Towards quantitative analysis
• Before one can do a quantitative analysis, one needs to know what is going on – This is the domain of qualitative analysis
Qualitative analysis (1)
• Qualitative analysis is an important first step in experimental design – Often, researchers use existing literature
• What can I control? • What should I measure? • How should I measure it? • Which values can I expect?
Qualitative analysis (2) • Sometimes (and perhaps more often in practical
situations) there is no ready-made information • In that case finding answers can be a difficult creative
process – Familiarize yourself with your subject – Play with it – Let others (your students!) play with it
• This may be part of the pilot experiment(s) • Or you may do case studies
• Such work is often descriptive in nature
Quantifying your subject • In order to make a quantitative analysis, you
need to quantify your subject – You need to convert aspects of interests to
measurable quantities
• Categorical – just categories (e. g. man/woman) • Ordinal – categories with an order (e. g. ranks) • Interval – distance is meaningful (e. g. position) • Ratio – distance and true zero (e. g. mass)
Measurements
• Knowing what to measure is part of how you operationalize your research question – ~How to make it precise
• But the measurements may be an indirect way of finding out what you want to know! – Temperature ~ energy – Reaction time ~ ease of use – Questionnaire ~ ease of use
Range and distribution
• Try to get an idea of realistic values • And how these values are distributed
– Which ones are frequent or rare
• Makes it easier to spot errors and bias – Or to make exciting new discoveries!
• Range and value are also important for input parameters
Bias (1) • A bias is a systematic error
– Results are changed in a predictable direction – But you may not be aware of this – Example: thermometer in the sun – Example: using computer scientists as testers
• Biases are bad for your research, so you should detect and remedy them! – Or compensate them, if at all possible – But this may not always be practical… – In which case you must be honest about them
• Sources of bias can be detected in qualitative analysis
Bias (example)
-50 -45 -40 -35 -30 -25 -20 -15 -10 0
1
2
3
4
5
6
7
minimum temperature (degrees centigrade)
num
ber o
f day
s
Minimal temperature in Yellowknife on February days
Bias (2) • Bias may also be in your mind
– Observer bias • You know what you want to find, so you tune your
experiment or your analysis until you find what you want to find – This happens subconsciously – Hire assistants to help you – Let your colleagues monitor you
• Performance measurements of software are prone to this – You tune your preferred model precisely – But not the competition’s model
Noise (1)
• Noise is random variation of your measurements – No systematic relation between real value and
measured value – Often assumed that the noise is the same for
all underlying real values
Noise (2)
• What does noise mean? – Often noise models lack of knowledge
• If we would know initial conditions of a coin flip perfectly, we could predict the outcome
– Especially true in biological/psychological experiments
• Variability in outcome is deterministic, but we don’t know the relevant parameters.
• Or we do not have the resources to measure/control them
Noise (3)
• What about randomness in simulation? – Randomness is often used to model noise in the real
world – Or to test sensitivity to initial conditions – In theory we could derive the relation between the
initial conditions and the outcomes, but in practice this is often intractable
• This is of course artificially introduced – But can be treated the same as natural noise
Noise (4)
• Because noise is independent of the underlying real value, it can be compensated by repeating a measurement sufficiently often
Noise (5)
• Noise is much less of a problem for experiments than bias – But it should always be reported – And therefore measured – More about this when we talk about statistics and
reporting
• Important message: just reporting a value or an average is not enough – One also needs to report about spread
What to do with measurements
• Always keep track of what and how you measure things – Keep a scientific journal! – Use version control for software – Never throw away or overwrite data files
• Rather buy a new hard disk
– Store your data in a systematic way • A data base, or something else that is easy to search
What to do with data
• Data about which you publish should be public property – Not the same as accessible to anyone – But should be shared with other scientists
• Some scientists have become famous by building publicly available corpora of data
Corpora of data
• Existence of corpora means that you do not always have to gather data – Reference data sets make comparison easier – But may limit scope
• E.g. everybody parses the Wall Street Journal corpus, but how representative is that?
• You can also use existing corpora to see if the data you gather makes sense
How to set up an experiment
How is science funded? (1)
• The university pays professors to teach and do research – But this happens less and less often
• In order to do research one has to apply for external funding – From the university – From national/local grant agencies – From European grant agencies – From dedicated funds – From businesses
How is science funded? (2)
• Even in companies with research departments, one often needs to apply for funding
• You need to make a case why your research is to be funded – And given that there are generally more
applications than funds, why not someone else’s
Proposal review (1)
• Proposals are reviewed like articles – By peer review – But additionally, by a committee of the
granting agency • In addition to the quality of the proposal,
the quality of the researcher is important – Your CV, your publication record – Success increases success in this game!
Proposal review (2)
• Funding agencies generally have goals – Promoting a field – Supporting promising young scientists – Supporting innovation – (sometimes) fair distribution among institutes
• Your proposal is evaluated for these goals – This is usually the job of the committee
What is in a proposal? (1)
• Your CV, presented in a way that stresses the elements that are relevant – But this is no different from applying for a job
• An outline of the research you want to do – The research question – Why this is relevant – The methods you are going to use – A description of the expected outcome – The impact of this outcome
What is in a proposal? (2)
• A proposal is not much different from a normal research plan – Although usually with somewhat less detail
• However, it is important to keep in mind the objectives of the funding agency – Try to think how you could do research that
fits their bill
Assignment 2
• I will ask you to write a proposal summary – The scientific part
• More about this towards the end of the lecture
Research project setup
• Without the sales pitch, and with more detail, a proposal becomes the setup of a research project
• Making this is an important part of science – Prevents you from acting before thinking – Especially in artificial intelligence, there is too
much gung-ho research
Steps in making a setup (1) 1. Familiarize yourself with the field
– Normally this has already happened 2. Define a research question 3. Operationalize the question 4. Figure out how to best analyze your data 5. Get permission to do it 6. Run a pilot experiment 7. Fine-tune the setup 8. Do the experiment
Familiarity with the field
• Identify open, interesting questions – You want to do something new – But nevertheless connected to what others do
• Know about standard methods, test sets and analysis methods
• Know about ways to report your research
• This you acquire by studying literature – And talking to other researchers!
The research question (1)
• This is an important creative step – I cannot give you a recipe
• Usually starts big – “How can a computer learn language?” – But this is infeasible
• You have to narrow it down – “How can I use support vector machines to
learn irregular verbs?”
The research question (2)
• Look at what others use as questions – But with only small variations, you won’t get famous
• Keep the underlying question in mind – Otherwise you may end up investigating an
abstraction, rather than reality • E.g. formal languages rather than natural language
– This is less of an issue in application-oriented research
Operationalization (1)
• Transform your scientific question into an experimental or observational program – What are you going to do? – What are you going to manipulate? – What are you going to measure? – How much data are you going to collect?
Operationalization (2)
• Again: look carefully at how others have operationalized their research – And build your own design accordingly
• Finding really new ways to do research is really difficult – This often leads to new branches of science
• e. g. Brain imaging techniques • e. g. Artificial life
Operationalization (3)
• Behavior is variable and measurements contain noise – Therefore, one wants to do repeated
experiments – But how best to do this? – Resources are limited
• This is a whole field of study!
Suggestions (1)
• There is an important difference between experiments with (computer) models and humans – Humans have many more unknown
behaviors, so must be more carefully controlled
– E.g. humans learn or get tired in sequences of experiments: first may be different from last
Suggestions (2)
• Performance comparison of systems – “Benchmark” – Use standard problems/data sets – Repeat the experiment to establish variation
• With different random seeds/initial states
• But what do we learn? – Only which system is best – Not necessarily how to improve them further
Suggestions (3) • We can change a parameter
– And see how this influences behavior – Parameters can be continuous or discrete
• Terminology – What you control = independent variable – What you measure = dependent variable
• To find out how the system works, and how it can be improved
• Examples – Biology: add fertilizer – Psychology: background music and reading speed – Computers: number of neurons and generalization
• Compare groups with different values of the independent variable
Suggestions (4)
• We can change multiple parameters – Parameters may be independent (total effect
is sum of individual effects) – Usually, parameters interact
• Combinations grow exponentially: kN
– k = number of steps – N = number of parameters
• This may be a practical problem
Suggestions (5) • If you do experiments with humans, other factors
play a role – Randomization becomes important – Because it is expected that systematic differences
between humans will have systematic effects • You have to assign you subjects randomly
• Example: interface 1 vs. interface 2 – Don’t first test interface 1 and then interface 2 – But randomly one or the other
Suggestions (6)
• Also it is better to do within subject comparisons than between subject comparisons – Between subjects there are all kinds of
unknown differences – Within a subject there is only the controlled
variable • Therefore you do multiple tests on one
subject
Suggestions (7)
• In order to reduce variation, you may let one subject do a task multiple times – But these are not independent data points – Therefore one preferably averages over data points
• Example – Time to find items using a library catalog – Each subject looks for a list of comparable items
Analysis (1)
• Ideally, you should decide how to analyze your data beforehand – Qualitative, explorative research may be problematic – But in standard experimental designs, it should be
straightforward – look what others have done
• Deciding on analysis methods beforehand helps you to prevent you fooling yourself – Try analyses until you find something significant
Analysis (2)
• Parts of the analysis – Which filtering/smoothing to use – How to summarize your data (averages etc.) – What outcomes to compare – Which statistical tests to use – etc.
• More about statistics in a later course – Descriptive and inferential
Analysis (3)
• Ideally the analysis is blind – The person doing the analysis does not know
what the outcome is expected to be
• But in practice you usually do it yourself
Permission • Almost any experiment involving humans requires permission
– To prevent unethical research – For this purpose, universities have ethical committees
• Ethical committees evaluate your research – How do you recruit? – Who do you recruit? – What do your subjects do? – How long does it take? – Where do they do it? – Do you record personal information? – Do you inform your subjects about the purpose of your experiment? – Do you get their informed consent?
• They do not check the quality of your design, only if your experiment follows ethical rules.
The pilot experiment
• Before running large numbers of (expensive) tests, try out your design – Do your measures work? – Do my subjects act as expected? – How large is the effect? – Does my analysis work?
Fine-tuning
• After the pilot experiment you can fine-tune your experiment – Change details of design and analysis – Estimate the required number of subjects
• You may potentially need to re-pilot – Especially if you make important changes
Assignment • Write a proposal summary • On the topic you chose for your literature study
– Should contain experiments • Answer questions:
– What is your research question? – Why is this important?
• With references to related work – What method will you use? – What results do you expect?
• Try to make it convincing! • Note that it is not read by a specialist in your area! • http://arti.vub.ac.be/cursus/2012-2013/mwo/form.doc • http://arti.vub.ac.be/cursus/2012-2013mwo/form.pdf
Deadline November 8 midnight