Upload
gilda
View
39
Download
0
Embed Size (px)
DESCRIPTION
Generalisability in Science Education Research. FontD Seminar 2004 in Mälardalens Högskola. Hans Niedderer Mälardalens Högskola Institutionen för Matematik och Fysik. Part 1. Theoretical aspects about generalisability. Aspects of generalisability. - PowerPoint PPT Presentation
Citation preview
Generalisability Generalisability in Science Education Researchin Science Education Research
FontD Seminar 2004 in Mälardalens Högskola
Hans Niedderer
Mälardalens HögskolaInstitutionen för Matematik och Fysik
Part 1
Theoretical aspects about generalisability
Aspects of generalisability
Aspect 1: Statistics and classical quantitative design Aspect 2: Generalisability as essential feature of theory Aspect 3: Generalisability and paradigm (Kuhn)
Aspect 1: Statistics and classical quantitative design
Petri 1996: ONE Student Hake 1998: 5000 students
Aspect 1: Generalisability guaranteed by design?
Weaknesses of case studies (Wirth & Leutner 2004)
… No discovery of a universal, generalizable truth … No discovery of a cause-effect relationship … Not appropriate for testing hypotheses
I could think of cases: with/without computer, with/without electronium
General empirical approach (Wirth & Leutner 2004)
GeneralizationInduction
Modification
Deduction of hypothesis
Scientific observationTheory
QuestionProblem
Internal Validity
General empirical approach (Wirth & Leutner 2004)
GeneralizationInduction
Modification
Deduction of hypothesis
Scientific observationTheory
QuestionProblem
Internal Validity
=> "External validity"
External validity
Extent to which conclusions drawn from a scientific observation can be generalized to other persons, situations or points of time.
Control environmental conditions, real life setting, representative sample, replication (in different contexts), theory use
Possibilities in science education
Where we have strong hypotheses from previous qualitative research ... Investigate the effect of using an electronium model of the
atom compared to a course using a propability model of the atom
Doing similar courses in optics with and without using different kinds of computer software
Example: In Roger's dissertation project at KAU, we try to develop a classical design, using different treatments with different use of ICT in optics
Aspect 1: Statistics and classical quantitative design
Generalisability ...seen as a problem of design and statistical evidence.
Aspect 2: Generalisability as essential feature of theory
Reif, F., Larkin, J. H. (1991) Cognition in Scientific and Everyday Domains: Comparison and Learning Implications, JRST Vol. 28, NO. 9
Aspect 2: Generalisability as essential feature of theory
Reif, F., Larkin, J. H. (1991) Cognition in Scientific and Everyday Domains: Comparison and Learning Implications, JRST Vol. 28, NO. 9
Little generalisability in EDL thinking ("cluster concepts")"Generalisability as definition of science" (Reif, Larkin, Schecker, Niedderer)
Incl. Science education !
Example of working for more generalisability
In Margareta's work about ownership we are working hard on a better theoretical definition of the concept of ownership, with maximal Generality Parsimony Precision Consistency
Furthermore we try to define in such a way that it has predictive power.
No statistics will be needed in this task.
Aspect 2: Generalisability as essential feature of theory
Generalisability ...seen as amount and quality of use of theory.
Theory with general concepts
Try to come to general definitions of concepts: use the same definition for every case, not one definition in one case and a different definition in an other case (as we would all do it in everyday life context!)
Similar: work on general claims or hypotheses Try to build up confidence by telling frequencies -
how often you were able to apply this concept - in qualitative work - and how often you were unsure about it. A negative statement - category does clearly NOT fit - is a positive statement in this sense!
Aspect 3: Generalisability as acceptance in the scientific community
T. S. Kuhn distinguishes three phases of development of a scientific theory:
the pre-paradigmatic phase: Many different questions, many "theories", little generalisability
the paradigmatic phase, high generalisability of paradigmatic research
the revolutionary phase: generalisability of the new paradigm takes time to build up.
Aspect 3: Generalisability as acceptance in the scientific community
In the paradigmatic phase of research, there are agreed questions concepts repeated and agreed results meanings
In SER we have at least one such field:alternative conceptions of learners Many Swedes contributed to it (B. Andersson, ..., F. Marton)
This gives a body of agreed knowledge which gives the highest amount of generalisability(Generalisability by cummulation in the scientific community)
Number of investigations in Pfund&Duit [email protected]
1998
> 4000 entriesalltogether
TopicMechanics 281
Electricity 146
Heat 68
Optics
69Particles60
Energy 69
Astronomy 36
Quantum Physics 11
421
218
111
35
1991 1994 1998687
379
159
190162
151
57
R. Duit:Bibliography "Students' Alternative Frameworks and Science Education"Aug. 2002
Part 2
Cases
Cases
Case 1: 6000 students study by Hake (1998) Case 2: Doctoral study of Bethge (1988) Case 3: Doctoral study of Petri (1996)
Case 1: Empirical Results Hake 1998
Interactive-engagement vs traditional methodsA six-thousand-student survey of mechanics test data for introductory physics coursesR. Hake, Am.J.Phys. 1998 (1)
Traditional methods
"Traditional" (T) courses as those reported by instructors to make little or no use of IE methods, relying primarily on
passive-student lectures recipe labs, and algorithmic-problem exams
“Interactive Engagement” (IE)
Methods as those designed at least in part to promote conceptual understanding through interactive
engagement of students in "heads-on" (always) and "hands-on" (usually) activities which yield immediate feedback through discussion
with peers and/or instructors
Interactive-engagement vs traditional methods
gain vs pretest - universities
4832 studentsg < 1
G in %
Generalisability in case 1
Aspect 1: Statistics and design 6000 students is very impressive Control of variables?
"interactive teaching as reported by the teachers" and "better understanding as measured by the FCI"?More qualitative tasks related to alternative conceptions about force?FCI measures only one aspect of competence, namely qualitative, multiple choice tasks related to many different alternative conceptions (not only Newton's force)
Generalisability in case 1
Aspect 2: Generalisability as essential feature of theory General theoretical claim Overgeneralised? Other factors not taken into account?
Aspect 3: Generalisability as acceptance in the scientific community
Case 2: Students' alternative conceptions in atomic physics (Bethge 1988)
Case 2: Students' alternative conceptions in atomic physics (Bethge 1988)
Methods 1) Audio recordings of current physics lessons were our main
data source. 2) A pair-relation questionnaire with associative elements. In this
type of questionnaire students were asked to make statements using two given concepts, for example: wave - energy level wave function - trajectory trajectory - energy level position - wave function electron - wave trajectory - probability
3) A questionnaire with seven "thinking type" tasks 4) Interviews with nine pairs of students
Conceptions related to orbits (trajectories) in quantum physics after teaching
(O1) Classical orbits (about 50%)(O2) Only special orbits allowed
(O3) Smeared orbits
The concept of "trajectory" is combined with notions of "probability" and "wave function" from wave mechanics in several ways to form a new "intermediate" conception:
- the orbits are "smeared", not exactly determined, "fuzzy"
- the probability for a special orbit is given- the probability of parts of the orbit is given
(O4) Trajectories do not exist in quantum physics (about 25%)
about 25%
Generalisability in case 2
Aspect 1: Statistics and design Criteria of Wirth & Leutner
external validity, generalisability to other persons, situations or points of time, representative sample: This research was with data from many students and classes,
real life setting: with data from real teaching The results seem thus be generalisable from a
statistical view with respect to 17 to 19 age students, after teaching in atomic physics with more than Bohr's model
Generalisability in case 2
Aspect 2: Generalisability as essential feature of theory General theoretical claim: students also in quantum atomic
physics show a limited number of alternative conceptions. Some of these are ...
Aspect 3: Generalisability as acceptance in the scientific community Replication in different contexts (Aspect 1) and
cummulation (Aspect 3) … was done to some extent later by dissertations (Lichtfeldt
1992, Mashhadi 1996, Deylitz 1999) and other research (Harrison et al. 1999, Müller et al. 2002)
So the results are - today ! - partially generalisable
Case 3: Learning pathways in atomic physics (Petri 1996)
Building on theoretical ideas in the community Driver 1989, Scott et al. 1991: conceptual pathways Bremen workshop 1991: need to describe learning pathways
"(1) Need to document learning pathways for different content areas in physics""(2) Need to construct ways of describing cognitive systems that are useful to researchers in physics education"
Niedderer (1991 to 1996): Prior example electric circuits Psillos 1999: Conceptual evolution
Giving ONE example in great detail (analysing huge amount of data from ONE student)
Carls Learning Pathway “Modell of the Atom"Petri 1996; Petri&Niedderer 1998
Example of working for more generalisability
In developping a new dissertation project at MdH, Peter Gustavsson and I are planning to have a new doctoral student working on a learning pathway for a whole class in distant education.
An advanced model of the learning process
Planetary view
Quantum cloud view
Smeared orbits view
Quantum particle view
Example: conceptions of an atom
time
strength
Generalisability in case 3
Aspect 1: Statistics and design 1 student: No generalisability to other students
Aspect 2: Generalisability as essential feature of theory Showing new, general model of a "learning pathway" as a
theoretical idea (GENERALISABILITY-claim!) in ONE example in great detail, thus making it work!Holzkamp: An experiment is a realisation of theory.
Aspect 3: Generalisability as acceptance in the scientific community The theoretical ideas were later used - and cited - by other
researchers(Psillos 1999 "conceptual evolution"; Taber (2000) "manifold conceptions in cognitive structure")
Part 3
Conclusions
Final conclusion
Try to come to theoretical definitions of concepts and claims, thus building up potential generalisability
Generalisability is a decision of the community of researchers, developping a paradigm (Kuhn), coming to paradigmatic research
This decision is based on empirical research in combination with normative decisions about relevant questions and theoretical approaches
To formulate it the other way round: I do not believe in generalisability from one study.
Final conclusion (ctd.)
In this view, generalisability means similar questions are asked with similar theory with similar results by (many) other researchers
This is why literature search and writing a "state of the art" in a doctoral dissertation and relating this to own results is so important!
Aspect 3: Generalisability as acceptance in the scientific community
In one field we are in a revolutionary phase:the basic understanding of teaching and learning
Old paradigm about teaching and learning
Paradigm shift about teaching and learning
Paradigm shift about teaching and learning
Acceptance in the community of practioners: still high
Acceptance in the scientific community: high,but consequences are still under development
Paradigm shift about teaching and learning
Acceptance in the community of practioners: still high
Example of a generalisable statement, not yet fully accepted:A learner's constructions are different from the teaching input - as a rule, not as an accident
Case 4: Mixed methods approach (Deylitz 1999, Budde 2003)
Pre- and post test results of special items
Results
Learning environment Cognitive system of student
Problem with quantum interpretation
Uli: Well, actually I completely put aside the concept of trajectory in the area of atomic physics. The function you have is nothing but the probability of presence of an electron. ... you can't say it moves on an orbit. To explain - well, the motion cannot really be explained anymore ... It's however, not an orbit any more. .. The electron must move somehow - very strange, it is now here and then there .. That gets crazy ..
Damned, it could theoretically move in between. Just that it moves in a strange zigzag, but that would mean again something like an orbit. And that's crazy again. Well, somehow I can't get that clear.
Diagram of probability distribution shown during
interview
Elke: If they didn’t move, a probability distribution would make no sense at all . The electron would stay in one position - that’s all ... As it shows up at different places it must move! Otherwise, it would be present always at only one point.
Summary of findings
25% of students use conceptions near to modern physics
25% use some typical intermediate conceptions such as "smeared orbits"
50% stick to classical orbits ("Bohr model")
... even after teaching
Part 2
Jürgen Petri (1996)
Research on learning processes
Research questions
How can we describe a learning process from a cognitive point of view?
Related theory International workshop on physics learning in 1991 Learning processes studies Cognitive modelling of physics learning Conceptual or learning pathways (Driver, Scott;
Niedderer, Petri; Psillos et al.) Conceptual change (Hewson, Duit, ...)
Data
1 student, 80 lessons on QAP, grade 13 gymnasium 6 interviews at different points in time Audio tapes from group work of students, partially
transcribed Written artefacts (reports etc.) from 1 student
Methods: Scheme for interpretive analysis of qualitative and quantitative data
xxx
Carls LearningPathway “Modell of the Atom"Petri&Niedderer 1998
Carls initial conception about an atom
C: The electrons, negatively charged tiny balls, move around the nucleus in definite orbits or circles, even several electrons in one orbit or shell.
orbits
Planetary model
Carls second conception about an atom
C: If I take the nucleus and think of a field around, a wave-like probability-field, then I think of drawing a psi-function as a wave that spreads out equally in all directions. And everywhere, where the probability is higher, there is an orbit. I don't know, I can't get rid of these orbits, though I don't know, where I got them.
Smeared orbits
IC smeared orbit in an atom ("cognitive attractor")
Found by different authors, with different teaching approaches:Bayer 1985, Bethge 1988, Petri 1996
Final state “model of an atom“ (Petri&Niedderer 1998)
Final state of Carl‘s cognitive system “atom"
An advanced model of the learning process
Planetary conception
Quantum cloud conception
Smeared orbits conception
Quantum particle conception
Example: conceptions of an atom
time
strength
Part 3
Stefan Deylitz (1999)
Research about evaluation
Erg
Testaufgaben Einzelne Items
Research questions
Evaluation of a new approach to quantum atomic physics (QAP)
Related theory/literature: different types of evaluation Comparative E. Formative E. Summative E.
==> How far were our own main goals achieved?
Data
26 students, 3 courses, grade 12 gymnasium New test constructed according to goals Pre and post test Post interviews with all students
Selected questions of the questionnaire
1a Draw a picture of an atom and label it!
1b Describe your model with a few sentences.
1e Can you determine the size of an atom in your model of an atom?
3b Given are three drawings. Describe commonalties and differences between these three atomic models and use the notions "electron orbit", "probability density", and "charge cloud".
Methods
Performance levels 0, 1, 2, 3Because of open-ended questions , we had to take answers to different questions related to the same knowledge domain. So, item combinations were defined to determine performance levels.
Evaluation: To what extent have the main goals been achieved by the students?
A quantum mechanical model is dominating
A classical model is
dominating
The psi-function is the main component of the model of an atom
No connection between psi-function and
model of an atom
The concept of state has a new meaning with relation to quantum physics
Students formulate the process of application without help
Equation is hidden behind
STELLA models
Observed phenomena are not related to a
quantum model of atom
A qualitative understanding of shielding; used in own STELLA models
No understanding of shielding effects
to be seen
Total performance level 0
Model of an atom
Psi-function
Notion of state
Schrödinger equation
Relating measurements to theory
Higher order atoms
0 1 2 3
50% -
50% -
50% -
50% -
50% -
50% -
Total performance level 3
The concept of state has no meaning with relation
to quantum physics
Observed phenomena are explained by relating them to a quantum model of atom
The first three goals
Deylitz 1999
Evaluation: To what extent have the main goals been achieved by the students?
The second three goals
A quantum mechanical model is dominating
A classical model is
dominating
The psi-function is the main component of the model of an atom
No connection between psi-function and
model of an atom
The concept of state has a new meaning with relation to quantum physics
Students formulate the process of application without help
Equation is hidden behind
STELLA models
Observed phenomena are not related to a
quantum model of atom
A qualitative understanding of shielding; used in own STELLA models
No understanding of shielding effects
to be seen
Total performance level 0
Model of an atom
Psi-function
Notion of state
Schrödinger equation
Relating measurements to theory
Higher order atoms
0 1 2 3
50% -
50% -
50% -
50% -
50% -
50% -
Total performance level 3
The concept of state has no meaning with relation
to quantum physics
Observed phenomena are explained by relating them to a quantum model of atom
A quantum mechanical model is dominating
A classical model is
dominating
The psi-function is the main component of the model of an atom
No connection between psi-function and
model of an atom
The concept of state has a new meaning with relation to quantum physics
Students formulate the process of application without help
Equation is hidden behind
STELLA models
Observed phenomena are not related to a
quantum model of atom
A qualitative understanding of shielding; used in own STELLA models
No understanding of shielding effects
to be seen
Total performance level 0
Model of an atom
Psi-function
Notion of state
Schrödinger equation
Relating measurements to theory
Higher order atoms
0 1 2 3
50% -
50% -
50% -
50% -
50% -
50% -
Total performance level 3
The concept of state has no meaning with relation
to quantum physics
Observed phenomena are explained by relating them to a quantum model of atom
Deylitz 1999
Part 4
Marion Budde (2003)
Research about learning effects of
content specific teaching inputs
Research questions
What effects has a certain teaching input on the learning process of an individual student?
Theoretical perspective:
==>Resonance
Detail of teaching approach ("teaching input")
The electronium modelThe probability model
Data
2 students: Audio tapes from teaching (five lessons of 45 minutes per week, for approximately18 weeks)
1 student attended private lessons (approximately 50 lessons), which were also audio-recorded.
Pre and post test for understanding of atomic models.
Methods
Criteria for statements showing a learning effect: Statements made spontaneously Statement with different content Contradicting statement Consequent statement in a complex discussion Use of own formulation or in a new context
Development of different types of resonance No resonance Congruent/non-congruent resonance
Overview of the observed resonances
Continuum /Continuousdensity
No motion
Student “Klaus” Student “Thomas”
Not intended non-resonanceParticle conception is toostrong: even liquids are seenas consisting of particles
Slightly delayedcongruent intendedresonance
Not intended non-resonanceDistance conception of density
Spontaneous congruentintended resonance
Spontaneous congruentintended resonance
Liquid Slightly delayed congruentintended resonance
Research about resonances during teaching
Trial of summary 1
Expect strong mechanistic thinking Particle Orbits, trajectories Planetary model Stability from equal forces: F-coulomb = F-centrifugal
Trial of summary 2
Expect intermediate conceptions(self constructed in students heads,not intended by the teacher)
Trial of summary 3
The "electronium" interpretation of psi in atoms seems ... to be intuitive ... to develop a meaning of a distribution ... to help to replace the particle conception ("cognitive tool")
by a similar powerful conception of a compressible gas or liquid
Cognitive attractor "smeared orbits"
Bayer (1986) describes it as a spherically smeared shell with a certain thickness, coming from a kind of wave orbits.
Bethge (1988) describes it as a new and broader understanding of orbits, taking into the notion of orbits some elements of smeared or most probable orbits or probabilities along the orbit.
Kuehnen (1994) describes conceptions which are no more purely mechanistic seeing the electron with some probability in a special space region, where they move along orbits. But they still are particles.
These different investigations show that the conception of ”smeared orbits” probably is what could be called a cognitive attractor in this content area, which describes a cognitive construction using a conception of orbits to make sense of a new teaching input with the idea of probability.