The IRAC Framework: Locating the performance zone for learning analytics

The IRAC FrameworkLocating the Performance Zone for Learning

AnalyticsDavid Jones @djplaner

Colin Beer @beercDamien Clark @damoclarky

http://bit.ly/iracZone

How universitieswill implementanalytics

==


data warehouses “have been around for quite

some time, they have been plagued by high failure rates and limited spread or use”

(Ramamurthy, Sen and Sinha, 2008, p. 976)

the vast majority of big data and magical business

analytics projects fail. Not in a great big system-won’t-work way…They fail because the users don’t use them. (Schiller,

2012)


The IRAC FrameworkLocating the Performance Zone for Learning

Analytics


Learning analytics, the analysis and representationof data about learners in order to improve learning (Clow, 2013, p.

1)

1. the development of new processes and tools aimed

at improving learning and teaching for individual students and instructors

2. The integration of these tools and processes intothe practice of teaching and learning

(Elias, 2011, p. 5)

(Siemens, 2013, p. 13)

(Atif, Richards et al, 2013)

Context: CQUniversity

Purpose: Identify students who may be at risk of failing

A nuanced appreciation of context is at the heartof mindful innovation with Information Technology

(Swanson & Ramiller, 2004)




(Swanson & Ramiller, 2004)

Help




Help academic staff help

Information

(Siemens, 2013, p. 13)

http://en.wikipedia.org/wiki/Law_of_the_instrument

Information - SAO

Source

Analysis

Output

# coursesPass rateGPAMoodle clicks

Single formula

Estimation ofsuccess

Student readiness survey Coming

soon!!

Insight

Information - SAO

Source

Analysis

# coursesPass rateGPAMoodle activity

Single formula

Coming soon!!

Discussion forumsSocial media pages

AWE and PASS

(Atif et al, 2013)

Output

Estimation ofsuccess

Insight

…the hegemonic status clickometry enjoys for analytic purposesin the realm of LMSs is counterproductive if our goal is to facilitateand evaluate the effectiveness of on-line learning

(Dietrichson, 2013, p. 349)

The amount of time students spend on the LMSis not an indication of deep or surface learning.

(Lodge & Lewis, 2012, p. 563)

Any RSS/ATOM generating application

http://bit.ly/bambim

Information - SAO

Source

Analysis

Output

Blog posts

External link analysis

Useful resources

Insight

Blog postsSchool sector

Link analysis

Recommend usefulposts from others

Blog posts

NLP

Detect and scaffoldreflection

Computational linguistics Detect

emotions…… …

…

Predict-o-mania

http://www.slideshare.net/dgasevic/learning-analytics-are-about-learning

..predictive models are one-off and, therefore cannot be extended easily from one context to another

(Essa and Ayad, 2012, p. 2)

Current modeling approaches..tend to be blackboxes from the standpoint of practitioners. Themere generation of a risk signal…does not convey enough information for the purpose of designing meaningful personalized interventions

(Essa and Ayad, 2012, p. 2)

Information Representation

Affordances Change

Clickstream data

Variable models

Scutable✖

✖

✔Demographic data✔

EOS ✔

Information

Representation

(Siemens, 2013, p. 13)

Peoplesoft

Peoplesoft

LMS

N/A


Affordances Change

Clickstream data

Variable models

Scutable✖

✖


IntegratedTabular

✔✔

EOS ✔

Dashboards suck!

Information

Representation

Affordances

(Siemens, 2013, p. 13)

..no surprise that those things that theaffordances make easy are apt to get done, those things that the affordancesmake difficult are not apt to get done

(Norman, 1993, p. 106)

M – mail-mergeE – emailP – phone callF – face-to-face

Hi all,

A quick email to let you know I have now completed all marking, and assessment results and the criteria sheets have been uploaded to Moodle. Again, there was a nice range of concepts explored and differing approaches to the use of the ePortfolio (Mahara).

In terms of our coursework, we are almost to the end of our theme-based activity and it is important to you are now drawing all of the threads together. Don't forget assessment task three is also right around the corner, so I would encourage you to make sure you are checking our Moodle site, contributing to the

forums and participating in either (or both) the scheduled Blackboard Collaborate sessions and the new drop-in sessions.

Please feel free to give me a call if there is anything you would like to discuss.

Regards, {$staff.firstname} {$staff.email}

20-30%


Affordances Change

Clickstream data

Variable models✖


“CRM” functionality ✔

Scutable✖

IntegratedTabular

✔✔

Exaptation

EOS ✔


Affordances Change

Clickstream data

Variable models✖


“CRM” functionality ✔

Scutable✖

IntegratedTabular

✔✔

Exaptation

Checkpoint & Process Analytics

(Lockyer et al, 2013)

✖

EOS ✔

http://www.itap.purdue.edu/studio/passnote/

Less is moreBirds of a featherTake actionDon’t be a threat

http://www.itap.purdue.edu/studio/passnote/


Affordances Change

Clickstream data

Variable models✖


“CRM” functionality ✔Pedagogical scaffolding✖

Scutable✖

IntegratedTabular

✔✔

Exaptation

Checkpoint & Process Analytics✖

EOS ✔

https://labs.desmos.com/teacher/demo#overview

Information

Representation

Affordances

Change

(Siemens, 2013, p. 13)

remainder of negative feedback concernedfaculty use of the tool…over penetration…a desire for more specific information….

demonstrates that instructors and studentsare on the same page about the lack of bestpractice

(Arnold, 2010, n.p.)

(Carroll, Kellog & Rosson, 1991, p. 80)


Affordances Change

Clickstream data

Variable models✖


“CRM” functionality ✔Task-artifact cycle✔

Scutable✖

IntegratedTabular

✔✔

Exaptation


Pedagogical scaffolding✖

EOS ✔

fundamental DSS principles like evolutionary development (Arnot & Pervan,

2005)

development processes are not implemented in a linear or

even in a parallel fashion, but in continuous action cycles that involve significant user participation(Arnot & Pervan,

2005)


Affordances Change

Clickstream data

Variable models✖



Evolutionary development ✔

(Buckingham-Shum, 2012, p. 8)

Scutable✖

IntegratedTabular

✔✔

Exaptation


The risk is that research and development focuses on the data which is simplest to log computationally,

perpetuating the dominant pedagogies and learning outcomes from an industrial era

Pedagogical scaffolding✖

Total clicks on

Moodle site


Affordances Change

Clickstream data

Variable models✖



(Reiser, 2001, p. 63)

In place✔

Scutable✖Tabular ✔

Exaptation

Checkpoint & Process Analytics✖..provide workers with the help they need toperform certain job tasks, at the time they needthat help, and in a form that will be most helpful.Pedagogical scaffolding✖

EOS ✔

IRAC

Greater use

Improved learning

How universitieswill implementanalytics

!=


Analysis of LA literature

Develop IRAC

Use IRACDesign research – BIM, SSI,

MAVEvaluation of institutional projects

Arnold, K. E. (2010). Signals: Applying Academic Analytics. Educause Quarterly, 33(1).

Arnott, D., & Pervan, G. (2005). A critical analysis of decision support systems research. Journal of Information Technology, 20(2), 67–87. doi:10.1057/palgrave.jit.2000035

Buckingham Shum, S. (2012). Learning Analytics. Moscow.

Carroll, J. M., Kellog, W. A., & Rosson, M. B. (1991). The Task-Artifact Cycle. In John Millar Carroll (Ed.), Designing Interaction: Psychology at the Human-Computer Interface (pp. 74–102). Cambridge, UK: Cambridge University Press.

Clow, D. (2013). An overview of learning analytics. Teaching in Higher Education, 6(August), 683–695. doi:10.1080/13562517.2013.827653

Dietrichson, A. (2013). Beyond Clickometry: Analytics for Constructivist Pedagogies. International Journal on E-Learning, 12(4), 333–351.

Elias, T. (2011). Learning Analytics: Definitions, Processes and Potential. Learning.

Essa, A., & Ayad, H. (2012). Student success system: risk analytics and data visualization using ensembles of predictive models. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge - LAK ’12 (pp. 2–5). Vancouver: ACM Press.

García-Solórzano, D., Cobo, G., Santamaría, E., Morán, J. A., Monzo, C., & Melenchón, J. (2012). Educational monitoring tool based on faceted browsing and data portraits. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 170–178). New York, NY, USA: ACM. doi:10.1145/2330601.2330645

Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing Pedagogical Action: Aligning Learning Analytics With Learning Design. American Behavioral Scientist, 57(10), 1439–1459. doi:10.1177/0002764213479367

Lodge, J., & Lewis, M. (2012). Pigeon pecks and mouse clicks : Putting the learning back into learning analytics. In M. Brown, M. Hartnett, & T. Stewart (Eds.), Future challenges, sustainable futures. Proceedings ascilite Wellington 2012 (pp. 560–564). Wellington, NZ.

Norman, D. A. (1993). Things that make us smart: defending human attributes in the age of the machine. Cambridge, Mass: Perseus. Reading, MA: Addison Wesley.

Ramamurthy, K., Sen, A., & Sinha, A. P. (2008). Data warehousing infusion and organizational effectiveness. Systems, Man and …, 38(4), 976–994. doi:10.1109/TSMCA.2008.923032

Reiser, R. (2001). A History of Instructional Design and Technology: Part 1: A History of Instructional Media. Educational Technology Research and Development, 49(1), 1042–1629.

Schiller, M. J. (2012). Big data fail: Five principles to save your BI butt. CIO Insight.

Siemens, G. (2013). Learning Analytics: The Emergence of a Discipline. American Behavioral Scientist, 57(10), 1371–1379. doi:10.1177/0002764213498851

Swanson, E. B., & Ramiller, N. C. (2004). Innovating mindfully with information technology. MIS Quarterly, 28(4), 553–583.

Ullmann, T. D., Wild, F., & Scott, P. (2012). Comparing Automatically Detected Reflective Texts with Human Judgements. In 2nd Workshop on Awareness and Reflection in Technology-Enhanced Learning. 7th European Conference on Technology-Enhanced Learning (pp. 101–116). Saarbruecken, Germany.

Slide 5, 49: "Question Everything / Nullius in verba / Take nobody's word for it" by Duncan Hull available at http://flickr.com/photos/dullhunk/202872717 under Attribution License http://creativecommons.org/licenses/by/2.0/

Slide 53: "University of Michigan Library Card Catalog" by David Fulmer available at http://flickr.com/photos/dfulmer/4350629792 under Attribution License http://creativecommons.org/licenses/by/2.0/

Slide 3: "Warehouse" by Michele Ursino available at http://flickr.com/photos/micurs/6118627854 under Attribution-ShareAlike License http://creativecommons.org/licenses/by-sa/2.0/

Slide 15: "Stream" by coniferconifer available at http://flickr.com/photos/coniferconifer/9535872266 under Attribution License http://creativecommons.org/licenses/by/2.0/

Slide 50, 51, 52: "The British Library" by Steve Cadman available at http://flickr.com/photos/stevecadman/486263551 under Attribution-ShareAlike License http://creativecommons.org/licenses/by-sa/2.0/

Slide 17: "Lawyer Crystal Ball" by CALI - Center for Computer-Assisted Legal Instruction available at http://flickr.com/photos/cali.org/6150105185 under Attribution-NonCommercial-ShareAlike License http://creativecommons.org/licenses/by-nc-sa/2.0/

http://creativecommons.org/licenses/by-sa/2.0/

http://creativecommons.org/licenses/by-sa/2.0/

Slide 23: "Dashboard" by Marko Vallius available at http://flickr.com/photos/markvall/3892112410 under Attribution-NonCommercial-ShareAlike License http://creativecommons.org/licenses/by-nc-sa/2.0/

Slide 4: "framework" by kaz k available at http://flickr.com/photos/kazk/198640938 under Attribution License http://creativecommons.org/licenses/by/2.0/

Slide 47: "25.365" by romana klee available at http://flickr.com/photos/romanaklee/5391995939 under Attribution-ShareAlike License http://creativecommons.org/licenses/by-sa/2.0/

Slide 18: "The Internet" by Martin Deutsch available at http://flickr.com/photos/MartinDeutsch/3190769121 under Attribution-NonCommercial-NoDerivs License http://creativecommons.org/licenses/by-nc-nd/2.0/

Slide 8, 9: "day 140" by mjtmail (tiggy) available at http://flickr.com/photos/mjtmail(tiggy)/2518317362 under Attribution License http://creativecommons.org/licenses/by/2.0/

Slide 6: "Purpose" by Seth Sawyers available at http://flickr.com/photos/sidewalkflying/3534131757 under Attribution License http://creativecommons.org/licenses/by/2.0/

Slide 12: "Making Omelettes" by PhotoGraham available at http://flickr.com/photos/PhotoGraham/260939952 under Attribution-NonCommercial-ShareAlike License http://creativecommons.org/licenses/by-nc-sa/2.0/

Slide 2, 48: "Smoking pile of sh*t" by David Jones available at http://flickr.com/photos/DavidTJones/3626888438 under Attribution-NonCommercial-ShareAlike License http://creativecommons.org/licenses/by-nc-sa/2.0/

Slide 40: "Change Allley sign" by Matt Brown available at http://flickr.com/photos/MattFromLondon/3163571645 under Attribution License http://creativecommons.org/licenses/by/2.0/