Embed Size (px)
Citation preview
• Course teachers
Ole K Andersen, dr scient ([email protected])
NN (theory of science)
• Course secretary
Jette Damkjær (common secretary for 1st semester M.Sc., [email protected])
Vita Kammersgaard ([email protected])
• Course literature (OKA part)
– Leedy & Ormrod. Practical Research: Planning and Design, 8th edition. Merrill, Prentice Hall, 2004.
– Overheads: Available after each lecture
– Course homepage
www.hst.aau.dk/~oka/SMAC
Videnskabsteori, -kommunikation og
etik
(SMAC) fall 2008
Course Plan
• Lecture 1 Introduction to the SMAC course, research hypothesis
• Lecture 2 Scientific methods in science and engineering (OKA)
• Lecture 3 Theory (and history) of science – only S-SN students (PKT)
• Lecture 4 Theory (and history) of science – only S-SN students (PKT)
• Lecture 5 Ethics in science, with emphasis on biomedical sciences (OKA)
• Lecture 6 Theory (and history) of science – only S-SN students (PKT)
• Lecture 7 Theory (and history) of science – only S-SN students (PKT)
• Lecture 8 Scientific communication, examples on oral presentation
and poster presentation. Details on SEMCON (OKA)
See homepage!
PKT, Patrik Kjærsdam Telleus
SMAC-lecture 1 - exercise
• Written response from 4-6 groups….
• The documents indicated a description of the problem, a tentative solution, goals, factors governing the problem, and metrics of the studies and of course the hypothesis
• One group experienced the traditional ‟development project‟ challenges
• Splitting the project in sub-problems
• Strategy for data collection
R&D methods in engineering
science• 4 sequential steps:
– (Overview)Analysis Hypothesis Synthesis Validation (communication/dissemination)
• Synthesis phase:
– Establish project task hierachy
– Design and implement every task
– Document proces (protocol and history)
– Run experiments: evaluate how well the task objective is achieved
• Confirm validity of computations (e.g. simulations)
• Design verification
• Effectiveness of solution
Lecture 6: Research
methodology (in engineering
science)• Basic vs applied research+examples
• Logic reasoning
• Quantitative vs qualitative measures
• Experimental designs and protocol
• Research design/methods at 7. sem
• Research– no specific device/product for general use – „negative results‟ are acceptable– descriptive (not directly applicable) – a tool for later developmental projects
• Development– product is well-defined– theory and main materials are known– external regulations (e.g. standards, patents)
Basic
research
Applied
research
Development
Scientific methods
- in Engineering
• Examples of basic science (at AAU)– acoustical perception– electromagnetic properties of different materials– mathematical theories for robust regulation– speech recognition based on physiological modeling
• Examples of applied science– telemedicine– design of ASICs using standard SW, FPGA research– satellite attitude control– mobile phone systems - e.g. protocols– Applications of GPS
Basic vs applied science
Engineering methods in basic
science
• Experiments
– Design of experimental setup based on literature and research hypothesis. This typically involves engineering disciplines outside research field + development
– Pilot studies leading to research protocol
– Data acquisition, data analysis
– Both methods and results are published, often independently
– Often descriptive results - increases our knowledge– Example:
• Design of mechanical stimulator for stretching a joint to study motor reflexes
• physical system
• control of DC motor
• Safety
• Angle and forces (kinematics and kinetics)
• Data analysis in perspective of previous methods in the literature
• Modeling (of physical systems)– System too complex/difficult for theoretical/experimental analysis
– Assumptions handled efficiently
– Theoretical work
– Simulations
– Validation of model is critical (an experiment)
– Properties of model are studied
– Methods and results of simulations are published
Example:
– Electrical properties of nerves, e.g. selective electrical stimulation
- too difficult to measure/test
- model: single fibre, infinite system, isotrophic, and homogenous
Engineering methods in basic
science
Engineering methods
in applied science• The theoretical methods are well-known
• Application area seeks new technologies
• Analytical work/system identification/modeling design and implementation
• Implemented according to needs/ specifications
• Results:
– Failure/success to apply these methods for solving the specific problem
– Performance of methods
– Strategy of design
• Example: telemedicine
– ECG recordings in hospital are trivial
– RF transmission of data is trivial
– Research project: combine the two areas does it save life?
• which data protocol is most efficient
• electrode/amplifier design so patient can handle the instrumentation
• preprocessing to alert physician?
• range of motion
Another example of applied
science• The electronic patient record
– databases are around/paper records are frequently lost/misplaced/standardization missing
– electronic results are present (images, critical data etc)
– research project: design database and access systems
– safety is a big issue
– open for modification/adding new features
– network bandwidth and storage formats
• Results:
– design and implementation
– performance of methods
Topic close to development!
• Research project:
– will it improve quality of treatment?
– Organisational issues
Logic reasoning
• Research planning begins with an
understanding of the way knowledge is
gained
• Two universal principles in understanding
the unknown
– Deductive logic
– Inductive logic
Deductive logic• Leading from a general assumption to a specific
phenomenon through logical reasoning (Aristotle)
– Starting with a major premise or statement (similar to axiom) which seems (seemed) self-evident and accepted truth e.g. “Humans are mortal” or “The earth is flat”
– Chain of reasoning illustrated with Columbus´s situation:
• The earth is flat
• Flat surface must have boundaries
• Boundaries would imply edges
• Exceeding the edges would mean falling of
– Logic is sound - the reasoning is accurate - the conclusion is valid BUT is based on a false premise
– Watch out for dogmatic premises in research projects !
Inductive logic (1)• Leading from a single observation to the
understanding of a more general phenomenon -this has been termed “the scientific method” (after Galileo)
• A typical scientific process– Positing a hypothesis as a logical means of locating the data and as
an aim to resolve the problem
– Empirically testing the hypothesis by acquiring, processing and interpreting the data
– Confirmation or rejection of the hypotheses
– The data implies…..(inductive logic)
• „Bottom up‟ construction of knowledge
• The paper is typically concluded by deduction of implications of the hypothesis
• Note: The hypothesis is typically based on few observations (e.g. pilot studies) that is inductive logic or alternatively an established theory using deductive logic
Inductive logic (2)• Example of inductive thinking
– “How long can a person have a flat EEG (an isoelectric
brain tracing indicating cerebral death) and still
recover?”
– Observation of 2,650 cases
– All cases with flat EEG for more than 24 h did not
recover
– Lead to the conclusion that recovery is impossible
when EEG is flat for > 24 h (a hypothesis is not proved)
Lead to one
conclusion
Separate and individual facts observed by the researcher
Qualitative vs quantitative
methods• “Effectiveness of lectures to teach scientific methods?”
– Qualitative approach
• Participates in lectures
• Talking with teacher
• Interviews with students
• Understanding the perspectives
• Finding themes and patterns in answers
• In-depth description and interpretation of observations
– Quantitative approach
• Impact on the achievement
• Comparison of two groups
– A = followed lectures
– B = not followed lectures
• Scores after the examination
• Test hypotheses: A = B
• Interpretation and summary
Qualitative vs Quantitative
Question Quantitative Qualitative
Purpose To explain and predict
Confirmation
Validation
Theory testing
Outcome oriented
To describe and explain
Exploration
Interpretation
Theory building
Process-oriented
Nature of research
process
Focused
Known variables
Established guidelines
Static design
Detached view
(objective)
Holistic
Unknown variables
Flexible guidelines
Emergent design
Personal view
(subjective)
Methods of data
collection
Representative, large
samples
Standardized
instruments
Informative, small
samples
Observations,
interviews
Communication of
findings
Numbers
Statistics
Aggregated data
Formal voice, scientific
style
Words
Narratives, individual
quotes
Personal voice, literary
style
Quantitative research
• Non-experimental
– Methodologies to describe a specific situation or phenomenon, as it is with no attempts to manipulate variables
• Experimental
– Methodologies to study the influence of a factor or factors conditioning the situation or phenomenon
Non-experimental research
• Descriptive surveys
– Measures the characteristics at one time point
• Longitudinal surveys
– Measures changes in a sample over time, e.g. effect of a certain therapy
• Correlation studies
– Knowledge from descriptive studies used to explore potential relationships, e.g. grades and parental income
• Describe homogeneity/heterogeneity, correlation coef‟s
• Interpret relationships
• Ex post facto research
– Describes relationships between something in the past (post facto) and present responses, e.g. influence of personality traits on present responses to a certain question
Descriptive surveys
• Observations as a means to collect data
– Questionnaires
– Structured interviews
– Rating scale checklists
• Selection of population
– Non-probability sampling
– Probability sampling
• Bias
– Distortion of data
• Randomisation
– From population to primary sampling unit
– Example: a Gallup Poll
Example of samplingPopulation level
Stratification level (proportional)
Randomisation level (fraction)
Data level
Use of non-experimental
research
• Necessary to describe the phenomenon in order to understand it
– E.g. the human genome
• Establish prevalence and incidence– Prevalence: how many is currently ill at a certain time point
– Incidence: how many new onsets of the phenomenon (disease) out of the total ensemble is expected within a certain time period
• Seeking cause-effect relationships
7. Semester project example
Non-experimental
• Group CN-780/2004, ”Reliability in single,
double and N2R ring network strutures”
• Goal: Improve reliability of access networks
• Simulations of the network and induction of
errors (cause and effect)
• Estimation of maximum delay and other
properties
• Critical assumptions (trafic distribution and
no of nodes in network)
Experimental research• “Attempt to account for the influence of a factor or factors
conditioning the situation”
• General concept
– Control is essential
– Experimental group = Control group (matched groups)
• Characteristics
– Cause-effect can be established under certain conditions
• Classification
– Independent variables
• Investigator has control over the variable, can change it
– Dependent variables
• Investigator has no control over the variable and it occurs under the influence of the independent variable
• Types (important to know limitations)
– Pre-experimental designs
– True experimental designs
– Quasi-experimental designs
Pre-experimental design
One-shot case study
• Experimental (X) treatment/processing/modulation procedure followed by an observation (O)
• Simplest design
• Lack of control, low internal validity (potentially biased)
• Single observations
• Confounding factors
– Time
– Situational factors
• Example: Cold feet day 1, running nose day 2. Is that sufficient to conclude cold is related to footwear?
X O
• One experimental procedure is applied in one group following a pretest observation (O1) and followed by a posttest observation (O2)
• Still lacks control
• Confounding factors
– Time
• Results suggestive, but not conclusive
Pre-experimental design
One-group pretest-posttest design
X O2O1
• Determine the influence of an experimental
variable on one group and not on another
• Weakness is no comparisons of pretest
observations
• Groups may or may not be different
• Example: descriptive study in a patient group
compared with healthy volunteers
Pre-experimental design
Static group comparison
X O1
O2
Group 1:
Group 2:
• Study the effect of an experimental influence on a carefully
controlled sample (control - experimental groups) by means
of randomisation (R)
• Variations can be obtained
– Open study
– Single-blind
– Double-blind
• Solid, reliable design with high validity
True experimental design
Pretest-posttest control group design
X O2
O4
Group 1:
Group 2:
RO1
O3
• Extended design
• Removes an eventual effect of “pretesting”
• Disadvantage is more groups = more samples = more
work
True experimental design
Four group design
X O2
O4
Group 1:
Group 2:R
O1
O3
X O5
O6
Group 3:
Group 4:
• When pretest is not possible
• Least stringent experimental design
• Randomisation is important
True experimental design
Posttest only design
RX O1
O2
Group 1:
Group 2:
• Investigate a situation in which random selection and
assignment are not possible
• Pretest observations not equalised by randomisation,
but the pre-tests ensure that groups are not different
• Analysis of co-variance may, in part, compensate for
initial differences in dependent variables (ANCOVA
or partial correlation analysis)
Quasi experimental design
Non-randomised control
X O2
O4
Group 1:
Group 2:
O1
O3
• Determine the influence of one experimental variable
(X) introduced only after a series of initial
observations and only where one group is available
• If a substantial change occurs in O5-O6 (O7), then
the experimental variable can be suspect to cause the
change
• External validity (can it be generalised, extrapolated)
can be increased by repeating the experiment under
different conditions
Quasi experimental design
Time-series experiments X O5 O6O1 O2 O3 O4 O7 O8
Experimental designs
• Many different types and variants available
• Choose the optimal for the specific purpose!
• Learn from papers and pilot-tests
• Consider feasibility
7 sem projects example• ST700/2005. Assessment of the effect of functional
trimming on the dynamic pressure....
• A new method to treat hoofs in cattle hoofs.....
• Experimental test on cattles on four days, two before trimming and two after
• Control group missing
• No blinding
X O3 O4O1 O2
7 sem projects example• SB740/2004. Noise robust automatic speech recognition....
• Two existing methods that works individually were combined
• Experimental test on existing database (training and testing on two groups)
• Static group comparison
• (Difficult to do blind due to time limit)
• Training set and test set must be carefully randomised
X O1
O2
O3
C1
C2
Two important aspects of
measurements
• Validity
– Soundness, effectiveness of the measuring instrument
– What does the test measure?
– Does it measure what it is supposed to measure?
– How well, how accurately does it measure?
• Reliability
– Consistency of the measurement
– How well can you measure something again and again?
Validity of measurements
• For the availability scale:
– What does “always” mean?
– What then about “generally”?
– What is “never” equal to?
• At first, the interval scale appears easy to use
• The validity of this scale as a measuring
instrument is questionable
0 10 20 30 40 50 60 70 80 90 100
Never
available
Seldom
available
Available by
appointment
Generally
available
Always
available
• Face validity
– Is the instrument measuring what it is supposed to measure?
– Is the sample being measured representative of the behaviour or trait being measured?
• Internal validity
– Freedom from bias in forming conclusion of the data. So the observations are due to the actual parameters being modulated.
• External validity
– Generalisation from a sample to other cases
Validity of measurements
Reliability of measurements
• Test-retest
– Compares the results of two administrations of the same test separated by some time interval
• Interrater reliability
– Two or more observers of the same situation/phenomenon record the measure (>0.85 is OK)
– Training can improve reliability
Note
– Measures with low validity can have high reliability
Sensitivity/specificity/validity
TN
FN FP
TP
Example
speech detection
iveFalseNegatveTruePositi
veTruePositiySensitivit
iveFalsePositveTrueNegati
veTrueNegatiySpecificit
iveFalsePositveTruePositi
veTruePositiValidity
How good is it in detecting speech
How good is it in detecting non-speech
How reliable is the speech detection
Recognition of the word ‟1‟
TP=hears ‟1‟ when ‟1‟ is said
FN=hears ‟x‟when ‟1‟ is said
FP=hears ‟1‟ when ‟x‟ is said
TN=hears ‟x‟ when ‟x‟ is said
Research designs
• Research planning
– To be compared to the architectural plan before the construction of a building
– Keywords are “meticulous” and “accurate”
– Inductive vs deductive logic
• Research protocol
– A baking recipe
– The engineering plan following the architectural plan
– The research protocol and research proposal are essential instruments in the scientific process
Research protocol
General outline
• Introduction
• Specific aims
• Materials and methods (list)
– Instruments
– Procedures
– Data analysis
• Expected results
• Discussion
• Conclusion
SMAC-mini-module 6
Exercise
• Exercise: What kind of method should ideally be used for evaluating your system/project?
– Quantitative/qualitative methods?
– Experimental
• If experimental then propose a design
– Non-experimental
• Descriptive surveys
• Longitudinal surveys
• Correlation studies
• Evaluate the measures you will use regarding validity, reliability, sensitivity etc.
Send your replies (*.pdf) to [email protected] including project title, group ID
