1

What Can We Hope To Achieve with eLearning Research?

The Educational Gold Standard

Dr. Iain Doherty

Director, eLearning Pedagogical Support Unit

Centre for the Enhancement of Teaching and Learning

The University of Hong Kong

December 6, 2013

2

GOLD STANDARDTHE GOLD STANDARD IN EDUCATIONAL RESEARCH

3

WHY QUESTION?Good question . . .

HKU is developing three Massive Open Online Courses (MOOCS), one of which is being offered by the Li Ka Shing Faculty of Medicine.

MOOCS are throwing up a lot of questions and whilst past research offers some answers there is clearly a need for MOOC specific research.

I sit in lots of meetings discussing research and I ask myself . . .

What kind of research should we be conducting?

4

GOLD STANDARDThe idea of a gold standard brings to mind notions of clarity of hypotheses, adequacy and rigor of research methods, robustness of conclusions etc.

These are not the issues that are addressed in this presentation . . .At least not directly.

The issue for this presentation really concerns the goals of eLearning / educational research.

But we should note that quality of research has been an issue in eLearning . . .

5

SAD TRUTHSSOME SAD TRUTHS ABOUTELEARNING / EDUCATIONAL RESEARCH

6

Hypermedia and Learning

500 papers

Effectiveness of Hypermedia

118 studies

Minimal Criteria - Scientific Merit

30 studies

“After reviewing nearly 500 papers related to hypermedia and learning, Dillon and Gabbard (1998) identified 118 studies that appeared to meet their criteria for quantitative studies examining the effectiveness of hypermedia in education . . . only 30 studies published between 1990 and 1996 met the minimal criteria of scientific merit for inclusion in the literature review”.(Reeves, 2000)

Review showed little impact from use of hypermedia

Reeves (2000) talks about the generally shoddy nature of eLearning research and cites substantial evidence for poor methodology and lack of impact, for example:

7

Reeves says that eLearning / educational research tends to occupy the bottom left quadrant:.

So, we need to remain cognizant of the quality of eLearning research.

Assuming quality – for now at least – we can take a straightforward perspective on the gold standard for educational research.

Even Worse…

Research inspired by questfor fundamental understanding

Research inspired by considerations of use

No Yes

No

Yes

Yikes

Basic Research Design Research

Action Research

8 The Easy Answer

The research has to provide “theoretical” insights as a basis for further research?????

There is a straightforward answer to the question

ANSWER TWO

This has to be true not just in a local context but in multiple contexts

ANSWERTHREE

The improvements also have to be sustained over time and they have to be scalable

ANSWER ONE

The gold standard is research that improves the teaching and learning experience

9 The Easy Answer made Difficult

Theory– “An idea or set of ideas that is intended to explain

facts or events”– “the general principles or ideas that relate to a

particular subject”

Keep in mind the question whether educational research has to contribute to educational theory.

10 Direction

For Reeves we need to attend both to what we are doing and to why we are doing it . . . (Reeves, 2000)

Assuming clarity of goals, appropriate methodology, rigorous application of method, clarity of findings, clear conclusions

We need research that is immersed in the classroom / comes out of the classroom

Teaching is messy (Brydon-Miller et al, 2003) and if we want to research it we need to get our hands dirty.

11

ACTION RESEARCHSHOULD WE AWARD THE GOLD STANDARDTO ACTION RESEARCH?

12 Action Research

“In general, the demand from academic staff is for help with design –for customisable, re-usable ideas, not fixed, pre-packaged solutions. However, demand is also expressed in ways which emphasise academics’ strong sense of being time-poor. There is no visible demand for complex methodologies, approaches which require substantial revision of existing work practices, or methods which require mastery of complex skills or specialised language”. (Goodyear, 2005, p.2)

“Action research can change current classroom practice. Certainly it is better than merely presenting the research results of professionals, as teachers must become personally involved in changing their classroom behavior, and few teachers become personally involved in an issue of the Review of Educational Research”. (Hodgkinson, 1957)

13

Action Research is in the classroom and involves systematic observation and data collection (quantitative and / or qualitative) that can be used by the practitioner-researcher to improve what is going on in the classroom.

Identifya problem

1

Createa solution

2

Implement

3

Reviseaccordingly

5

Evaluate

4

14

Action Research can be shared quickly and easily through e.g.a database of action research designs. (Laurillard, 2008)

Improvements are sustainable in the local context by the individual teacher and scalable through offering the template designs to other teachers.

University of Wollongong Template Designs

Learning Activity Management System (LAMS)

15

It sometimes theorizes against knowledge base

Unlike “science” this is not controlled experimentation that

takes variables into account(Hodgkinson, 1957)

ACTIONRESEARCH

Unlike “science” work stops at the practical solution with no

attempt to generalize (Hodgkinson, 1957)

Some people claim that action research is not really research but just a form of evaluation.

However, it can be considered to be research if results are shared so that others may draw their own inferences. (Reeves, 2000)

16 EPSU Action Research

FLIPPED CLASSROOM

Faculty of Medicine

ONLINE

CASESDepartment of

Pathology

LANGUAGE LEARNING

School of Chinese

GRADEMARK

Faculty of Social Sciences,

Centre for Applied English Studies

EPSU has engaged in some action research with teachers:

17 Remember the Question Mark

There are many varieties of action research and some do aim for theoretical insights.(Chandler & Torbet 2003)

However, educational action research is not generally concerned with drawing substantially on theory.

It is useful but not the right approach for researching the complexity of MOOCS.

Improve theteaching and

learningexperience

Can be applied inmultiple contexts

Can be sustainedover time and

is scalable

It can providetheoretical insights

as a basis forfurther research

(Friedman & Rogers,2009)

ActionResearch

18

DESIGN RESEARCHSHOULD WE AWARD THE GOLD STANDARDTO DESIGN RESEARCH?

19

Researchers with . . . [design research] goals are focused on the dual objectives of developing creative approaches to solving human teaching, learning, and performance problems while at the same time constructing a body of design principles that can guide future development efforts” .(Reeves, 2000)

Design Research – like action research – is carried out in the classroom and this is important in terms of testing designs with respect to what is really happening.

20 Empirical Versus Design Research

– Design research works off of what Sandoval refers to as design conjectures (2004) = hypotheses.

– Based on familiarity with a body of knowledge + the actual teaching situation researchers, teachers, students etc. come together to create a learning design that is expected to behave in certain ways in the classroom.

– The “conjectures” are tested in the real world to determine if what was conjectured is in fact the case.

21 Example of Design Research– Conjecture that computer mediated situated learning

environment will lead to knowledge that students [pre-service teachers in this case] can apply. (Herrington & Oliver, 2000; Herrington & Oliver, 1999)

– Students develop mathematics assessment strategy.– Herrington and Oliver identify 9 elements to a situated learning

environment and they ground these elements in the relevant research literature.

22 Example of Design Research

– So the first point is that design research utilizes a lot of theory. Very time intensive to put a research based design into practice.

– In this case the design research approach also required the creation of a complex learning environment rich with resources.

– Take a look at the situated learning element again, this time with the resources mapped against the element.

23

Run the design experiment and capture data from observing students interacting with the multimedia environment, from reports produced by the students and from questionnaires.

Carry out data analysis and report findings.

24 Design Research

The clear benefithere is that principles are being developed and tested out in the

environment that matters

The actual situation is

incredibly complex and messy and

multiple variables will impact on the learning design

Researchis complex and intensive witha lot of data gathering

However, there is a positive here because this is “real world” and

unexpected insights occur!

25 Design Research can

Improve the teaching and

learning experience

Can be applied in multiple

contexts [but not generalizable to

multiple contexts]

Provides theoretical

insights as a basis for further

research

Can be sustained over time and is scalable? Requires a huge commitment of time / resources and tends not to be scalable because others do not have the inclination / time to put results into practice / further the research.

26

CONCLUSIONS

27

OK, so the gold standard depends on perspective but there is one key issue – the research has to make a difference in practice.

– Action research suits teachers + instructional designers +

management to some degree.

– Design research suits teaching intensive teachers + instructional designers + management to some degree.

– Basic / theoretical research suits, well, researchers + instructional designers who draw on it + management who have to report research outputs.

28

Q&A

29 References

• Brown, A. L. (1992). Design Experiments: Theoretical and Methodological Challenges in Creating Complex Interventions in Classroom Settings. The Journal of the Learning Sciences, 2(2), 141–148. Retrieved from http://www.jstor.org/stable/1466837

• Brydon-Miller, M., Greenwood, D., & Maguire, P. (2003). Why Action Research? Action Research, 1(1), 9–28. doi:10.1177/14767503030011002

• Chandler, D., & Torbert, B. (2003). Transforming Inquiry and Action: Interweaving 27 Flavors of Action Research. Action Research, 1(2), 133–152. doi:10.1177/14767503030012002

30 References

• Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design Research: Theoretical and Methodological Issues. The Journal of Learning Sciences, 13(1), 15–42. Retrieved from http://www.jstor.org/stable/1466931 .

• Friedman, V. J., & Rogers, T. (2009). There is Nothing so Theoretical as Good Action Research. Action Research, 7(1), 31–47. doi:10.1177/1476750308099596

• Goodyear, P. (2005). Educational Design and Networked Learning: Patterns, Pattern Languages and Design Practice. Australasian Journal of Educational Technology. Retrieved from http://www.ascilite.org.au/ajet/ajet21/goodyear.html

31 References

• Herrington, J., & Oliver, R. (1999). Using Situated Learning and Multimedia to Investigate Higher-Order Thinking. Journal of Educational Multimedia and Hypermedia, 8(4), 401–422. Retrieved from http://www.aace.org/pubs/jemh/v8n4.htm

• Herrington, J., & Oliver, R. (2000). An Instructional Design Framework for Authentic Learning Environments. Educational Technology Research and Development, 48(3), 23–48. doi:10.1007/BF02319856

• Hodgkinson, H. L. (1957). Action Resarch - A Critique. Journal of Educational Sociology, 31(4), 137–153. Retrieved from http://www.jstor.org/stable/2264741

32 References

• Laurillard, D. (2008). The Teacher as Action Researcher: Using Technology to Capture Pedagogic Form. Studies in Higher Education, 33(2), 139–154. Retrieved from http://dx.doi.org/10.1080/03075070801915908

• Oh, E., & Reeves, T. (2008). Design Research vs. Instructional Systems Design: Implications for Educational Technologists. In World Conference on Educational Multimedia, Hypermedia and Telecommunications 2008 (pp. 2119–2127). Vienna, Austria: AACE. Retrieved from http://www.editlib.org/index.cfm?fuseaction=Reader.ViewAbstract&paper_id=28664

33 References

• Reeves, T. C. (2000). Enhancing the Worth of Instructional Technology Research through “Design Experiments” and Other Development Research Strategies. In Annual Meeting of the American Educational Research Association (pp. 1–15). New Orleans, LA, USA. Retrieved from http://it.coe.uga.edu/~treeves/

• Sandoval, W. A. (2004). Developing Learning Theory by Refining Conjectures Embodied in Educational Designs. Educational Psychologist, 39(4), 213–223. doi:10.1207/s15326985ep3904_3