LEARNING ANALYTICS, ACTION SCIENCE AND

CRITICAL REALISM

DR TIM ROGERS, TEACHING INNOVATION UNIT, UNIVERSITY OF SOUTH AUSTRALIA

TOPICS FOR TODAY

OUTLINE

• Short background to my underlying philosophy, critical realism

• Description of an approach to action research, action science

• A look at some practices from action science that are, or could be, applied in educational action research

• An overview of learning analytics tools as methods for educational action research

SOME PHILOSOPHICAL UNDERLABOURING

CRITICAL REALISM

• The best explanation of the progress of the natural sciences suggests underlying essences and causal powers are central, not just observable relationships between variables

• In the social sciences concerned with the causal roles of structure and agency

• So actions may involve unacknowledged conditions, unconscious motivations, tacit skills, and have unintended consequences

• A language user may not know the role of socialisation in language acquisition (unacknowledged condition), may not be aware of reasons for concealing their thoughts (unconscious motivation), may not be able to describe the grammatical rules they evidently know in practice (tacit skills) and may be incorrect in their belief about the effect of their utterance (unintended consequences).

• A trainee teacher may not know the role of their historical experiences as a student (unacknowledged conditions), may not be aware of the reasons for being angry at a student (unconscious motivation), may not be able to describe the rules underlying their explication of a concept (tacit skills), and may be incorrect in their belief about the effect of their interventions (unintended consequences).

MEANING-MAKING AND THEORY BUILDING IN THE SOCIAL WORLD

ACTION SCIENCE

• Original researchers were Chris Argyris and Donald Schön

• A meta-theory of human action: human action is (often) goal driven and involves tacit theories for achieving these aims

• In situation X (conditions), 2) do Z (strategy), 3) achieve Y (goal)

• Two theories at work: espoused theories and theories-in-use

• We are often unaware of the gaps between these two theories

LEARNING ABOUT OUR ‘FRAMES’ FOR LEARNING

META-LEARNING IN ACTION SCIENCE

• ‘Single-loop learning’ occurs when action strategies are changed, but the rest of the theory-in-use remains constant

• ‘Double-loop’ learning involves changes in goals, frames, assumptions, values, and/or standards for performance

AN EXAMPLE OF AN OUTPUT OF ‘THEORY OF ACTION’ RESEARCH-IN-PRACTICE

BLAME CYCLE THEORY-IN-USE

…OR COULD BE READILY APPLIED TO TEACHING

EXAMPLES WITHIN TEACHING…

• Professor Viviane Robinson, University of Auckland:

• Robinson, V. M., Meyer, F., Sinnema, C., & Le Fevre, D. (2016). Truth-seeking or truth-claiming? Leaders' patterns of social problem solving. Paper presented at 2016 American Educational Research Association (AERA) Annual Meeting, Washington, DC, USA. 8 April - 12 April 2016.

• Le Fevre, D. M., Robinson, V. M. J., & Sinnema, C. E. L. (2015). Genuine inquiry: Widely espoused yet rarely enacted. EDUCATIONAL MANAGEMENT ADMINISTRATION & LEADERSHIP, 43 (6), 883-899.

• Le Fevre, D., & Robinson, V. M. J. (2015). The Interpersonal Challenges of Instructional Leadership Principals’ Effectiveness in Conversations About Performance Issues. Educational Administration Quarterly, 51 (1), 58-95.

…OR COULD BE READILY APPLIED TO TEACHING

EXAMPLES WITHIN TEACHING…

• Dr Jenny Rudolph, Harvard Center for Medical Simulation:

• Rudolph, J. W., Simon, R., Rivard, P., Dufresne, R. L., & Raemer, D. B. (2007). Debriefing with good judgment: combining rigorous feedback with genuine inquiry. Anesthesiology clinics, 25(2), 361-376.

• Rudolph, J. W., Simon, R., Dufresne, R. L., & Raemer, D. B. (2006). There's no such thing as “nonjudgmental” debriefing: a theory and method for debriefing with good judgment. Simulation in Healthcare, 1(1), 49-55.

LEARNING ANALYTICSFOR EDUCATIONAL ACTION RESEARCH

DESIGN LEARNING

EXPERIENCE

COLLECT DATA

ANALYSE DATA

REFINE LEARNING THEORY/MODEL

Stanford Graduate School of Education http://analyticsdose.com/big-data-the-science-of-learning-analytics-and-transformation-of-education/

LEARNING ANALYTICS…

…is the collection, collation, analysis and reporting of data about learners and their contexts, for the purposes of understanding and optimizing learning

ASSUMPTIONS AND FOCI

• Provides actionable intelligence: ‘closes the loop’ • Can accommodate ‘big data’ (usually from the LMS e.g.

Moodle) • Visualises data for summary and exception highlighting

OR ‘BRICOLAGE’

SOURCES: A CONFLUENCE DISCIPLINE

Ed theory and practice, social network analysis, data mining, machine learning, natural language processing, data visualisation, sense-making, psychology (social, cognitive, organisational), learning sciences, and more

WHY LA, WHY NOW?

• Volume of students

• Widening participation

• Constrained budgets

• Increased scrutiny on quality

• Do more, absorb greater variety, with less money, and show it works and is improving. Or else…

DESCRIPTIVE ANALYTICS

• What’s going on?

• Who’s doing it? Or not doing it?

• An example workflow

UNISA’s data dashboard

Overview  data:  student  course  site  visits

Overall  engagement

Forum  engagement

Student  forum  engagement

Visualising relationships pt1

http://www.snappvis.org/

Visualising relationships pt2

http://www.snappvis.org/

Visualising relationships pt3

http://www.snappvis.org/

PREDICTIVE ANALYTICS

• Who or what do I need to watch out for?

• Previous cohort student data used to model incoming cohort risk (of attrition or failure)

WHO IS LIKELY TO HAVE LOW ENGAGEMENT?

EARLY WARNING DASHBOARDS

SOME ISSUES WITH PREDICTION

• Self-fulfilling prophecies and ethics • Course level is key to predictive accuracy but

global algorithms often used • Helps or harms student motivation? • Who’s to say what my dependent variable

should be? • Nevertheless, can be very helpful for timely

interventions to know who might be at risk, and when

CAUSAL ANALYTICS - EXPLANATION

• Why is what is happening, happening?

• Sensemaking in the whole learning system…

• …for both educators and students

NEW TECHNOLOGIES + ESTABLISHED THEORIES = NEW INSIGHTS

• What is missing: a focus on learning process

Bringing together:

• Self-Regulated Learning (SRL) proficiency (Gasevic; Winne)

• Natural language processing (NLP) and text mining (Rose)

• Learning design and instructional conditions (Lockyer; Gasevic)

WHY MIGHT STUDENTS NOT USE A TOOL?

And what might we do to rectify this?

• A story of new technology (CLAS) meeting new analytic techniques (NLP) and established learning theory (SRL)

Gašević,  D.,  Dawson,  S.,  &  Mirriahi,  N.  (Revised  &  resubmitted).  What  is  the  role  of  teaching  in  adoption  of  a  learning  tool?  A  natural  experiment  of  video  annotation  tool  use.  

TAKEAWAY

LA will:

• Go hand-in-glove with learning design

• Provide timely process feedback about learning, rather than outcomes

• Encourage educator and student active engagement

• Facilitate educational action research and scholarship of teaching and learning (SOTL)

FIN QUE?