Learning Analytics - Competen-SEAcompeten-sea.eu/wp-content/uploads/2018/02/... · Recommendations...

LearningAnalytics–

what,how,why

MarenScheffelIoanaJivet

OpenUniversiteitNederland

Outline

•  Whatislearninganalytics?•  Whereareweat?•  Howaredashboardsused?•  Howcanweevaluatelearninganalyticstools?•  Howaboutaninstitutionallearninganalyticspolicy?•  Wheredowegofromhere?

Insight is the buzzword in the data revolution. Peoplewho have insight into their behaviour will be able toworkmoreefficientlyfromnowon.Thisequallyappliesto organisations, higher education institutions andnations. The digital revolution has brought about atremendous increase in the volume of data and it iscomparativelyeasytoaccess.Itwouldseemtobealostopportunitytoignoresuchdata.

MarjoleinvanTrigt(2016)

https://cdn.static-econo

mist.com

fault/files/im

ages/prin

TheUnitedNationsarecallingforactions:

“dataarethelifebloodofdecision-making”

“the new world of data,a world in which dataare bigger, faster andmore detailed than everbefore”

But what about theeducationaldomain?

Dataanalysisiscommon inbusinessandcommerce.

http://arcbridge.com

/wp-conten

loads/20

15/07/bu

siness-analytics.pn

Collecting,storingandanalysingstudentrecords

hasalwaysbeendone

inonewayoranother.

http://balfour.rb

e.sk.ca/sites/balfour/files/Da

vid-Oscar-Brown-Stud

ent-Re

cord-side-2.jpg

http://abo

vethelaw

/wp-conten

loads/20

13/03/file-cabine

learningacquisitionofknowledgeorskills

throughstudy,experience,orbeingtaught

analyticssystematiccomputationalanalysisofdataor

statisticsor

informationresultingfromthesystematicanalysisofdataorstatistics

Learning analytics is the measurement, collection,analysis and reporting of data about learners andtheir contexts, for purposes of understanding andoptimising learningand theenvironments inwhichitoccurs.

Siemens(2011)

Outline

LevelsofLearningAnalytics

LearningAnalyticsFramework

Greller,W.andDrachsler,H.(2012).TranslatingLearningintoNumbers:AGenericFrameworkforLearningAnalytics.EducationalTechnology&

Society,15(3):42–57.

LAshouldbeforteachersandlearners

LASophisticationModel

Siemens,G.,Dawson,S.,&Lynch,G.(2014).ImprovingtheQualityandProductivityoftheHigherEducationSector–PolicyandStrategyforSystems-LevelDeploymentofLearningAnalytics.Canberra,Australia:OfficeofLearningandTeaching,AustralianGovernment.

Drachsler,H.andGreller,W.(2016).PrivacyandAnalytics–it’saDELICATEissue.AChecklisttoestablishtrustedLearningAnalytics.InProceedingsoftheSixthInternationalConferenceonLearningAnalyticsandKnowledge,LAK’16,pages89–98,NewYork,NY,

USA.ACM.DOI:10.1145/2883851.2883893.

http://www.laceproject.eu/ethics-privacy-learning-analytics/

http://www.eugdpr.org/

Outline

LearningAnalyticsDashboards

http://w

ww.learno

nsolutions.com

/wp-conten

loads/20

16/10/learning-analytics-learno

Structure

Source:https://www.flickr.com/photos/bruneluniversity/12926707974/

Learninganalyticsdashboardsare…

…“singledisplaysthataggregatedifferentindicatorsaboutlearner(s),learningprocess(es)and/orlearningcontext(s)intooneormultiplevisualisations”Targetusers:teachers(75%),learners(51%),administratorsandresearchersLearningscenarios:formal(91%),non-formal,informalPurpose:Supportinformeddecisionmaking

Schwendimann,B.A.,Rodriguez-Triana,M.J.,Vozniuk,A.,Prieto,L.P.,Boroujeni,M.S.,Holzer,A.,...&Dillenbourg,P.(2017).Perceivinglearningataglance:Asystematicliteraturereviewof

learningdashboardresearch.IEEETransactionsonLearningTechnologies,10(1),30-41.

Thelearninganalyticsprocess

Verbert,K.,Duval,E.,Klerkx,J.,Govaerts,S.,&Santos,J.L.(2013).Learninganalyticsdashboardapplications.AmericanBehavioralScientist,57(10),1500-1509.

Visualisation

Indicators

Educationalfoundation

• Barchart• Linegraph• Piechart• Scatterplot• Socialgraphs• Text

• Learnercharacteristics• Contextoflearning• Content• Learneractions• Learningoutcomes• Socialinteractions

• Feedbacktheory• Self-regulatedlearning• Collaborativelearning• 21stcenturyskills• Bloom’staxonomy• Socialcomparison

Learninganalyticsdashboards

Jivet,I.,Scheffel,M.,Drachsler,H.,andSpecht,M.(2017).Awarenessisnotenough.Pitfallsoflearninganalyticsdashboardsintheeducationalpractice.InProceedingsofthe12thEuropean

ConferenceonTechnologyEnhancedLearning(EC-TEL2017),LNCS,pages82-96.Berlin.Springer.

Learninganalyticsdashboardsforteachers

Monroy,C.,Rangel,V.S.,&Whitaker,R.(2013).STEMScopes:contextualizinglearninganalyticsinaK-12sciencecurriculum.InLAK’13(pp.210-219).ACM.

Learninganalyticsdashboardsforlearners

Corrin,L.,&deBarba,P.(2014).Exploringstudents’interpretationoffeedbackdeliveredthroughlearninganalyticsdashboards.ASCILITE2014.pp.629-633.

Beheshitha,S.S.,Hatala,M.,Gašević,D.,&Joksimović,S.(2016).Theroleofachievementgoalorientationswhenstudyingeffectoflearninganalyticsvisualizations.InProceedingsLAK’16(pp.54-63).

Aljohani,N.R.,&Davis,H.C.(2013).Learninganalyticsandformativeassessmenttoprovideimmediatedetailedfeedbackusingastudentcenteredmobiledashboard.NGMAST2013,pp.262-267.IEEE.

Learninganalyticsdashboardsforteachers

TheLearningTracker

Designgoals1.  Traceable2.  Scrutable3.  ActionableIterativedesign1.  Interactive2.  Planningahead3.  Framing

TheActivityWidget

TheLearningPathWidget

Recommendationsfordesigningdashboards

1.  Define the goal of the dashboard and desired impact on the learners.

2.  Research educational concepts as theoretical grounding. 3.  Describe data that supports the goal. 4.  Experiment with different visualisations. 5.   Customise the dashboard to provide the same support to all

users. 6.  Integrate the dashboard into the learning activities. 7.  Focus dashboard evaluation on goal fulfillment, impact on

learners' affect and motivation, and on the usability of the tool. 8.  Tool's usability should also assess how learners interpret

information. 9.  Use multiple data sources to validate dashboard effects.

Jivet,I.,Scheffel,M.,Drachsler,H.&Specht,M.(2017).Awarenessisnotenough.Pitfallsoflearninganalyticsdashboardsintheeducationalpractice.EC-TEL2017.

Outline

Whereistheevidence?

http://www.laceproject.eu/evidence-hub/

Learninganalyticstools

•  Demandsofthestakeholders?•  Empiricalevidence?•  Desiredeffect?•  Indicatorsfortoolevaluation?•  Comparability?•  Standardisedevaluation?•  Impact?•  Benefit?

TheEvaluationFrameworkforLA

Finally, if there isn’t a relevant questionnaire foryoursituation,youcandeviseone.

JohnBrooke(2013)

ObjectivesforEFLAcreation

•  Identifyqualityindicatorsforlearninganalytics

•  Createanapplicableevaluationinstrumentbasedontheseindicators

•  Validatetheevaluationinstrument

ObjectivesAwareness Reflection Motivation Behavioural

Change

Learning SupportPerceived Usefulness Recommendation Activity

ClassificationDetection of

Students at Risk

Learning Measures and OutputComparability Effectiveness Efficiency Helpfulness

Data AspectsTransparency Data Standards Data Ownership Privacy

Organisational AspectsAvailability Implementation Training of Educa-

tional StakeholdersOrganisational

Change

DATA-I know what data is being collected-I have access to my data-I understand the presented results

L E A R N E R S

-I know what data is being collected-I have access to my students’ data-I understand the presented results

AWARENESS-I am aware of my current learning status-I comprehend my current learning status-I can project my future learning status

-I am aware of my students’ current learning status-I comprehend my students’ current learning status-I can project my students’ future learning status

REFLECTION-I reflect on my learning activities-I reflect on alternative learning activities-I know when to change my behaviour

-I reflect on my teaching activities-I reflect on alternative teaching activities-I know when to change my behaviour

T E A C H E R S

IMPACT-I can detect whether I am falling behind-I study more efficiently-I study more effectively

-I can detect whether my students are falling behind-My students learn more efficiently-My students learn more effectively

DATA-For this LA tool it is clear what data is being collected-For this LA tool it is clear why the data is being collected-For this LA tool it is clear who has access to the data

L E A R N E R S

-For this LA tool it is clear what data is being collected-For this LA tool it is clear why the data is being collected-For this LA tool it is clear who has access to the data

AWARENESS-This LA tool makes me aware of my current learning situation-This LA tool makes me forecast a possible future learning situation given my (un)changed behaviour

-This LA tool makes me aware of my students’ current learning situation-This LA tool makes me forecast my students’ possible future learning situation given their (un)changed behaviour

REFLECTION-This LA tool stimulates me to reflect on my past learning behaviour-This LA tool stimulates me to adapt my learning behaviour if necessary

-This LA tool stimulates me to reflect on my past teaching behaviour-This LA tool stimulates me to adapt my teaching behaviour if necessary

T E A C H E R S

IMPACT-This LA tool increases my motivation to study-This LA tool stimulates me to study more efficiently-This LA tool stimulates me to study more effectively

-This LA tool increases my motivation to teach-This LA tool stimulates me to teach more efficiently-This LA tool stimulates me to teach more effectively

DATA-For this LA tool it is clear what data is being collected-For this LA tool it is clear why the data is being collected

L E A R N E R S

-For this LA tool it is clear what data is being collected-For this LA tool it is clear why the data is being collected

AWARENESS & REFLECTION-This LA tool makes me aware of my current learning situation-This LA tool makes me forecast my possible future learning situation given my (un)changed behaviour-This LA tool stimulates me to reflect on my past learning behaviour-This LA tool stimulates me to adapt my learning behaviour if necessary

-This LA tool makes me aware of my students’ current learning situation-This LA tool makes me forecast my students’ possible future learning situation given their (un)changed behaviour-This LA tool stimulates me to reflect on my past teaching behaviour-This LA tool stimulates me to adapt my teaching behaviour if necessary

T E A C H E R S

IMPACT-This LA tool stimulates me to study more efficiently-This LA tool stimulates me to study more effectively

-This LA tool stimulates me to teach more efficiently-This LA tool stimulates me to teach more effectively

•  isvalid&reliable,

•  addressesthelackofevaluationinstruments,

•  complementsevaluationonaninstitutionallevel,

•  addressestheneedforevidencethathasbeencalledfor,

•  iseasytousewithtemplates&interactivespreadsheet

DATAFor this LA tool it is clear what data is being collected

The Evaluation Framework for Learning AnalyticsE F L A

AWARENESS & REFLECTION

IMPACT

forL E A R N E R S

For this LA tool it is clear why the data is being collectedstrongly

disagreestrongly agree

This LA tool makes me awareof my current learning situation

This LA tool stimulates me to reflecton my past learning behaviour

This LA tool makes me forecast my possible future learning situation given my (un)changed behaviour

strongly disagree

strongly agree

stronglydisagree

strongly agree

strongly disagree

strongly agree

stronglydisagree

strongly agree

This LA tool stimulates me to adaptmy learning behaviour if necessary

stronglydisagree

strongly agree

This LA tool stimulates me to study more efficientlystrongly

This LA tool stimulates me to study more effectivelystrongly

DATAFor this LA tool it is clear what data is being collected

AWARENESS & REFLECTION

IMPACT

forT E A C H E R S

For this LA tool it is clear why the data is being collectedstrongly

This LA tool makes me awareof my students’ current learning situation

This LA tool stimulates me to reflecton my past teaching behaviour

This LA tool makes me forecast my students’ possible future learning situation given their (un)changed behaviour

stronglydisagree

strongly agree

stronglydisagree

strongly agree

stronglydisagree

strongly agree

stronglydisagree

strongly agree

This LA tool stimulates me to adaptmy teaching behaviour if necessary

stronglydisagree

strongly agree

This LA tool stimulates me to teach more efficientlystrongly

This LA tool stimulates me to teach more effectivelystrongly

EFLA scoring per stakeholder groupStep 1: Calculate the average value for each item based on the answers given for that item.

Step 2: Calculate the average value for each dimension based on the average of its items.

Step 3: Calculate the dimensional scores by rounding the results of ((x-1)/9)*100 where x is the average value of a dimension.

Step 4: Calculate the overall EFLA score by taking the average of the three dimensional scores.

http://www.laceproject.eu/evaluation-framework-for-la/

Outline

•  Whatislearninganalytics?•  Whereareweat?•  Howaredashboardsused?•  Howcanweevaluatelearninganalyticstools?•  HowaboutaninstitutionalLApolicy?•  Wheredowegofromhere?

PolicyFramework

http://sheilaproject.eu/

PolicyFramework

ACTION CHALLENGES POLICY

culture capabilities datamanagement

ethics&privacy culture evaluation

financial&humanresources

ethics&privacy methodology

infrastructure infrastructure policymanagement

internal&externalsupport management purpose

methodology methodology stakeholderengagement

purpose

stakeholderengagement

Outline

LASophisticationModel

Siemens,G.,Dawson,S.,&Lynch,G.(2014).ImprovingtheQualityandProductivityoftheHigherEducationSector–PolicyandStrategyforSystems-LevelDeploymentofLearningAnalytics.Canberra,Australia:OfficeofLearningandTeaching,AustralianGovernment.

ProblemofLAUptake

EducationalScience

ComputerScience

LearningAnalytics

MIDDLESPACE

Solution:UseLAasarealmiddlespace

EducationalScience

ComputerScience

LearningAnalytics

MIDDLESPACE

Learninganalytics…

… is not about the creation of intelligent or stupidsystems…… but about the creation of intelligent and successfulstudents…

…andthepromotionofeducation.

Ifeducationisviewedasavehicleforadvancingtheglobaleconomy,thenitmustbetheNorthStarthatguides societies to the next big thing, illuminatingnew ideas that solve pressing challenges andcreatingopportunitiestoshapeabetterfuture.

SamanthaAdamsBeckeretal.(2017)

Nowit’syourturn…

…tocreateyourowndashboard!

Thankyouforyourattention!

maren.scheffel@ou.nl ioana.jivet@ou.nl@m_a_s_c @IoanaJJEFLA:www.laceproject.eu/evaluation-framework-for-la/LAattheOU:www.ou.nl/en/welten-open-learning-analytics

Learning Analytics - Competen-SEAcompeten-sea.eu/wp-content/uploads/2018/02/... · Recommendations...

Documents

Developing Dashboards

08. Dashboards

MiPCT Dashboards & Reports Introduction to MiPCT Dashboards and Reports

Multiple Dashboards

Visualisation dashboards

digitalenergy dashboards

Dashboards - Pedigree Technologies€¦ · Dashboards Dashboards Dashboards are the centralized view for day-to-day operations. They can be customized to display the information you

Dashboards Scorecards

Network Scan Data - Avis Lombardia · 2013-09-04 · cultura e sport), casa, CAPACITÀ E COMPETEN TECNICFE Cen cornpLrter, attrezzature specific s, macchinarj, e CAPACITÅ E COMPETEN

Tableau Dashboards

DIGITAL DASHBOARDS

Winning the Hearts and Minds of Followers_ the Interactive Effects of Followers' Emotional Competen

Introducing Dashboards

Business Model for a MOOC - competen-sea.eucompeten-sea.eu/wp-content/uploads/2018/02/MOOCs_Economics.pdf · – Remedial courses ... How are we integrating them with customer routines?

FEASIBILITY AND READINESS STUDY OF COASTAL ...competen-sea.eu/.../2018/02/UNSRAT-feasibility-report.pdfFEASIBILITY AND READINESS STUDY OF COASTAL COMMUNITY IN THE ISLANDS OF NORTH

Data dashboards

Marketing Dashboards

Crystal Dashboards

CAST Dashboards

Brochura Dashboards