Upload
shamus
View
44
Download
3
Tags:
Embed Size (px)
DESCRIPTION
Design -Based Research for Advancing Educational Technology. EDIT 9990. Goals. Critique the state- of-the-art of educational research. Describe applications of “design-based research.” Encourage new thinking about why and how we do research. - PowerPoint PPT Presentation
Citation preview
DesignDesign-Based -Based Research for Research for Advancing Educational TechnologyAdvancing Educational Technology
EDIT 9990EDIT 9990
Goals Critique the state-
of-the-art of educational research.
Describe applications of “design-based research.”
Encourage new thinking about why and how we do research.
Improving the quality of teaching and learning through educational research is critical to our survival.
Global Warming
Dutch Floating Homes
The lack of scientific literacy is appalling in even the most developed countries.
Bad News
Most educational research has little impact on practitioners and yields few discernable benefits.
Oh...no!
The Failure of Educational Research
– Vast resources going into education research are wasted.
– They [educational researchers] employ weak research methods, write turgid prose, and issue contradictory findings.
The Failure of Educational Research
– Too much useless work is done under the banner of qualitative research.
– Qualitative research…. [yields] ….little that can be generalized beyond the classrooms in which it is conducted.
College of EducationThe University of Georgia
Ranked 27th of 187 education colleges in the USA
240 faculty members in 9 departments
5,000 students in 33,000+ student university
Research Productivity 1997-2001Refereed Journal Articles (in-cites.com)
U. of Wisconsin - 202 U. of Georgia - 201 U. of Michigan - 164 Indiana U. - 161 U. of Maryland - 146
Georgia vs. Wisconsin
$7,824
44,073
51%
49th
$8,604
42,232
78%
7th
Per pupil
Salary
HS Grad.
Ranking
It is time we put the PUBLIC back in publication!
Bush Administration Position “There’s been no
improvement in education over the last 30 years, despite a 90 percent increase in real public spending per pupil.”
Promotes randomized controlled trials as used in medical research.
Accountability: Guaranteeing Results
Flexibility: Local Control for Local Challenges
Research-Based Reforms: Proven Methods with Proven Results
Parental Options: Choices for Parents, Hope for Kids
Four Reform Four Reform PrinciplesPrinciples
The What Works Clearinghouse (WWC) has been established by the U.S. Department of Education’s Institute of Education Sciences to provide educators, policymakers, researchers, and the public with a central and trusted source of scientific evidence of what works in education.
What Works What Works ClearinghouseClearinghouse
Slavin’s 5 Questions for Valid Educational Research Is there a control group? Are the control and
experimental groups assigned randomly?
If a matched study, are the groups extremely similar?
Is the sample size large enough?
Are the results statistically significant? Robert Slavin
“What Works” Position “Once we have dozens or
hundreds of randomized or carefully matched experiments going on each year on all aspects of educational practice, we will begin to make steady, irreversible progress.”
NCLB funds “scientifically based research.” Robert Slavin
“It Won’t Work” Position Double blind experiments
impossible in education Implementation variance
reduces treatment differences
Causal agents are under-specified in education
Goals, beliefs, and intentions of students and teacher affect treatments David R. Olson
Medical and health knowledge is rarely applied sufficiently.
Another “It Won’t Work” Position The What Works Clearinghouse
(WWC) standards “ignore the critical realities about social, organizational, and policy environments in which educational programs and interventions reside.”
Advocates “decision-oriented” evaluation research over “conclusion-oriented” academic research.
Recommends extended-term mixed-method (ETMM) designs as a viable alternative.
Madhabi Chatterji
American Evaluation Association
The priority given to randomized controlled trials “manifests fundamental misunderstandings about 1) the types of studies capable of determining causality, 2) the methods capable of achieving scientific rigor, and 3) the types of studies that support policy and program decisions. We would like to help avoid the political, ethical, and financial disaster that could well attend implementation of the proposed priority.”
Randomized controlled trials are the only way
we’ll ever be able to prove “what works” in education!
Randomized controlled trials promotes
pseudoscience and will limit effective change!
Educational researchers have failed to make a clear appeal to the public for their support.
People learn when…..
Ellen Lageman argues that educational researchers, in a misguided effort to be “scientific,” have turned away from the pragmatic vision of John Dewey.
She attacks the excessive emphasis on quantitative measurement.
Kieran Egan argues that progressive ideas from Herbert Spencer, John Dewey, and Jean Piaget are responsible for the “general ineffectiveness” of our schools.
He also assails the notion that education can be improved through research as traditionally conceived.
Thomas Kuhn The Structure of Scientific Revolutions
"I'm not sure that there can now be such a thing as really productive educational research. It is not clear that one yet has the conceptual research categories, research tools, and properly selected problems that will lead to increased understanding of the educational process. There is a general assumption that if you've got a big problem, the way to solve it is by the application of science. All you have to do is call on the right people and put enough money in and in a matter of a few years, you will have it. But it doesn't work that way, and it never will."
Complexity of Interactions We cannot store up
generalizations and constructs for ultimate assembly into a network.
When we give proper weight to local conditions, any generalization is a working hypothesis, not a conclusion.
Lee Cronbach
Learning Styles“Research into learning styles can, in the main, be characterised as small-scale, non-cumulative, uncritical and inward-looking. It has been carried out largely by cognitive and educational psychologists, and by researchers in business schools and has not benefited from much interdisciplinary research.”
Dichotomies convergers versus divergers verbalisers versus imagers holists versus serialists deep versus surface learning activists versus reflectors pragmatists versus theorists adaptors versus innovators assimilators versus explorers field dependent versus field
independent globalists versus analysts assimilators versus
accommodators imaginative versus analytic
learners non-committers versus plungers common-sense versus dynamic
learners concrete versus abstract learners random versus sequential learners
initiators versus reasoners intuitionists versus analysts extroverts versus introverts sensing versus intuition thinking versus feeling judging versus perceiving left brainers versus right brainers meaning-directed versus undirected theorists versus humanitarians activists versus theorists pragmatists versus reflectors organisers versus innovators lefts/analytics/inductives/successive
processors versus rights/globals/deductives/ simultaneous processors executive, hierarchic, conservative
versus legislative, anarchic, liberal.
If Sisyphus were a scholar, his field would be educational research.
- David Laberee
Educational Technology Research
Pseudoscience Results
Insert cone of experience example
Pseudoscience Results
Insert cone of experience example
Educational technology researchers are not doing much better than other educational researchers.
NCLB Requirements "every student is
technologically literate by the time the student finishes the eighth grade," and
"that technology will be fully integrated into the curricula and instruction of the schools by December 31, 2006."
Abundant technology has not led to extensive use of computers for “tradition-altering classroom instruction.”
The small percentage of computer-using instructors only use it to maintain existing classroom practices.
Teachers have legitimate concerns. Is it simple enough for
me to learn quickly? It it versatile? Will it motivate students? Is it aligned with skills I’m
expected to teach. Is it reliable? It it breaks, who will
help? Will it weaken my
classroom authority?
Ed. Tech Research Reality Isolated researchers conduct
individual studies rarely linked to a research agenda or concerned with any relationship to practice.
Studies are presented at conferences attended by other researchers and published in journals few people read.
Occasional literature reviews and meta-analyses are published.
Ed. Tech Research Reality Many educational technology
studies claim to have predictive goals (testing theories) and use quasi-experimental designs with quantitative measures.
Research reviewers usually must reject 75 percent or more of the published studies to find the few worthy of further review or inclusion in meta-analyses.
Ed. Tech Research Reality Dillon & Gabbard’s 1998 literature
review of “Hypermedia as an Educational Technology” highlights problems with IT research.
Major conclusion: “Clearly, the benefits gained from the use of hypermedia technology in learning scenarios appear to be very limited and not in keeping with the generally euphoric reaction to this technology in the professional arena.”
Ed. Tech Research Reality Fabos & Young 1999 literature review
of “Telecommunications in the Classroom: Rhetoric Versus Reality” is another bad sign.
Major conclusion: “…many of the expected benefits of telecommunications [enhancing writing, multicultural awareness, and economic possibilities] are inconclusive, optimistic, and even contradictory.”
Bernard et al. (2004) Meta-analysis: “How Does Distance Education Compare to Classroom Instruction?”
a very small but positive mean effect size for interactive distance education over traditional classroom instruction on student achievement
small negative effect for retention rate
DE Research from 1985-2002
1,010 potential “studies” retrieved
232 studies met all criteria
599 independent effect sizes
47,341 students (achievement)
Results: Overall Effects 325 independent outcomes (total
achievement)
Hedges’ g = +0.0122, p < .001
Range of findings from –2.17 to +2.66
177 outcomes with low methodology removed
Hedges’ g = +0.017, p > .05
Significantly heterogeneous
Distribution of Effect
Sizes
-3
-2
-1
0
1
2
3
1 21 41 61 81 101 121 141 161 181 201 221 241 261 281 301 321
Distribution of Effect Sizes for Achievement Outcomes
Magnit
ude o
f Eff
ect
Siz
e
Hedges’g
Effect Sizes Ordered by Magnitude
325 independent outcomes (achievement)Hedges’ g = +0.0122, p < .001
Sir John Daniel - UNESCO… the futile tradition of comparing test performances of students using new learning technologies with those who study in more conventional ways…is a pointless endeavor because any teaching and learning system, old or new, is a complex reality. Comparing the impact of changes to small parts of the system is unlikely to reveal much effect and indeed, “no significant difference” is the usual result of such research.
Chewing Gum More Effective than Interactive Multimedia CD-ROM
Dr. Ken Allen at NYU wanted to compare CD-ROM with lectures
Wrigley’s wanted to fund chewing gum study
Combined study Gum chewers = B-
Abstainers = C+ CD-ROM no better
Media comparison studies are akin to comparing copper bracelets with voodoo dolls as medical cures.
Experimental approaches to educational technology research won’t work in the way that they do in medical research!
MedicalCures
GeneticsResearch
Basic Applied
“Pasteur’s Quadrant” approach to research is needed (Stokes, 1997).
Considerations of use?
No Yes
Quest for fundamental understanding?
No
Yes
Bohr
Research is inspired by:
Edison
Pasteur
Good News There are new strategies for conducting “design-based research” that can improve our research so that it can become a socially responsible enterprise.
Thankgoodness!
Educational researchers often fail to distinguish between research goals and methods.
Six Ed. Tech research goals:
TheoreticalPredictiveInterpretivistPostmodernDesign/DevelopmentAction/Evaluation
Theoretical Goals
Focus on explaining phenomena through logical analysis and synthesis of principles and results from other studies
EXAMPLE: Gagne’s theory of the conditions of learning
Predictive Goals Focus on determining
how education works by testing hypotheses related to theories of learning, teaching, performance, etc.
EXAMPLE: cooperative learning and control studies by Hooper, Temiyakarn, and Williams Simon Hooper
Interpretivist Goals Focus on determining
how education works by describing and interpreting phenomena related to learning, teaching, performance, etc.
EXAMPLE: Delia Neuman’s observations of disabled children using commercial software Delia Neuman
Postmodern Goals
Focus on examining the assumptions underlying educational programs with the goal of revealing hidden agendas and empowering disenfranchised minorities
EXAMPLE: Ann DeVaney’s analysis of IT in relation to race, gender, and power
Design/Development Goals Focus on dual
objectives of developing creative approaches to solving problems and constructing reusable design principles
EXAMPLE: Sasha Barab’s “Quest Atlantis” project and “Learning Engagement Theory”
Action/Evaluation Goals
Focus on describing, improving, or estimating the effectiveness and worth of a particular program
EXAMPLE: Hill and Reeves four-year evaluation of ubiquitous computing initiative.
Methods should not be selected until goals & research questions are clear:
Quantitative
Qualitative
Critical Theory
Historical
Literature Review
Mixed-methods
So what does design-based research look like in the real world?
Design-Based Research Collective
Goals of designing learning environments and theories are intertwined
Development and research occur in continuous cycles
Research on designs leads to sharable theories relevant to practitioners
Research must account for how designs function in authentic settings
Development of accounts relies on methods that connect actions to outcomes
Design-Based Research Strategies Define a pedagogical outcome and create
learning environments that address it. Emphasize content and
pedagogy rather than technology.
Give special attention to supporting human interactions.
Modify learning environments until outcome is reached.
Chris Dede – Harvard University
River City Curriculum & Situated Learning Theory
Yasmin Kafai - UCLA
van den Akker, Nieveen, McKenney – University of Twente
Design-Based Research Example “Authentic Learning in Interactive
Multimedia Environments.” Ph.D. dissertation by Jan Herrington at
Edith Cowan University in Australia. Supervised by Professor
Ron Oliver. Winner of AECT
Young Researcherof the Year in 1999.
Outcome Practitioners Desired
New teachers will use a wider variety of assessment methods in their student teaching experience and eventual practice.
Learning Environment Design
Identified the critical characteristics of a situated learning model.
Developed an interactive multimedia learning environment based on those characteristics.
Situated Learning Model Herrington 1997
Provide an authentic context reflecting the way the knowledge will be used in real-life
Provide authentic activities Provide access to expert
performances and the modeling of processes
Provide multiple roles and perspectives
Support collaborative construction of knowledge
Situated Learning Model Herrington 1997
Promote reflection to enable abstractions to be formed
Promote articulation to enable tacit knowledge to be made explicit
Provide coaching and scaffolding at critical times
Provide for integrated assessment of learning within the tasks.
Mixed Methods Design Videotaped
preservice teachers using program
Interviewed teachers and their supervisors in schools during student teaching practicum
Logged usage data
Findings Problem Solved:
– Novice teachers acquired advanced knowledge while engaging in higher order thinking
– New knowledge and skills applied in practicum
Design Principles:– Situated learning model is
a successful design model for interactive learning
What challenges do we face in adopting design-based research approaches?
Design-Based Research Challenges
–Sampling bias
–Response bias
–Researcher bias
–Overwhelming data
–Confounded variables
–Dissemination
–Scaling up Ann Brown
Design-Based Research CollectiveWe suggest that the value of design-based research should be measured by its ability to improve educational practice.
Because we are a design profession (not a discipline), educational technologists should pursue design-based research that integrates the desire to solve problems with the search for knowledge.
“The status of research deemed educational would have to be judged, first in terms of its disciplined quality and secondly in terms of its impact. Poor discipline is no discipline. And excellent research without impact is not educational.” - Charles W. Desforges (2000)