Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
15 July 2020: 07:00AM UTC Assessment in Higher Education conference panel: looking to 2021 Session chair: Fabio R Aricò (University of East Anglia, UK)
Just to let you know: By participating in the webinar you acknowledge and agree that: The session may be recorded, including voice and text chat communications (a recording indicator is shown inside the webinar room when this is the case). We may release recordings freely to the public which become part of the public record. We may use session recordings for quality improvement, or as part of further research and publications.
Webinar Series
e-Assessment SIG
Webinar Hosts Professor Geoff Crisp, Deputy Vice-Chancellor & Vice-President Academic University of Canberra g.crisp[at]canberra.edu.au Dr Mathew Hillier, Macquarie University mathew.hillier[at]gmail.com
FabioR.Aricò
AHE:LeadingAssessmentforLearninginHigherEducaAon
FionaMeddings
AHEConferenceCommi.eeHosts
AHEisanindependentnetworkfocusedondevelopingresearch-
informedpracAceinassessmentandfeedbackinHE.
InternaAonalExecuAveCommiKeehostedbytheLEDResearch
CentreattheUniversityofCumbriainEngland.
AHEnetworkandeventsacrossfullrangeofsubjectdisciplines
andprofessionalfields-evaluaAng,researchinganddeveloping
theory,research,policyandpracAceinassessmentand
feedback.
AHE:LeadingAssessmentforLearninginHigherEducaAon
AHENetwork–ahenetwork.org
ForumforcriAcaldebateofresearchandinnovaAvefocused
assessmentandfeedbackpracAceandpolicy.
Keynotes,peer-reviewedpresentaAons,masterclassesfrom
leadingexperts.
30June–2July2021ManchesterMacDonaldHotel
Manchester,England
AHE:LeadingAssessmentforLearninginHigherEducaAon
AHEConference2021
AHE:LeadingAssessmentforLearninginHigherEducaAon
AHEConference2021-Preview
JamesMichaelWood(SeoulNaAonalUniversity,SouthKorea)
‘Feedbackisuselessifno-onecangetwhatitmeans’:Theeffect
oftechnology-mediatedpeerfeedbackonuptakeandliteracy
JoannaTai,MaryDracup,MerrinMccracken,YasminMobayad
(DeakinUniversity,Australia)
Inclusiveassessmentdesign:whatcanwelearnfromassessmentadjustments?
RobLowneandSuzanneStone(DublinCityUniversity,Ireland)
Assessmentanddigitalcompetences:buildingcapacitywithacademic
staff
‘Feedback is useless if no-one can get what it means’: The relationship between technology-mediated peer feedback and
feedback uptake and literacy
DrJamesWood
AHEConferencePanel:TransformingAssessment
Understanding relationship between technology-mediated peer and teacher feedback and feedback uptake and literacy
4 Results and discussion
1 Feedback literacy & paradigm shifts
2 The USM Model
3 Online Feedback practices & methodology
5 Implications & conclusions
Feedback is powerful! But…
(Carless&Boud,2018)
For feedback to be effective…
Itmustbeusedto‘alterthegap’(Sadler,1989)
For feedback to be effective…
Itmustbeusedto‘alterthegap’(Sadler,1989)
Halfofstudentsfailtocollectfeedback,orengagewithitpoorly(Evans,2013;Handley,Price&Millar,2011)
Problemsexacerbatedbyviewoffeedbackas
transmission(Eretal.,2020;Molloyetal.,2019)
However
Feedback literacy
Howcanitbedeveloped?
Through,analysingexemplars,
workingwithassessmentcriteriaPeerfeedbacketc.(CarlessandBoud,2018)
Dialoguesmayenhancebenefitsoftheseprocesses(Carless,2015;Zhu&Carless,2018) (Carless&Winstone,2020)
The changing context of feedback practice COVID19–goingonlinechangesmessageandrelaAonalandpracAcalaspectsoffeedback–mediatetheeffectsofthepracAce(Winstoneetal.,2017)
Gapindialogicstudiesindigitalsenngsthattargetfeedbackuptake&literacy(Ajjawi&Boud,2017;Steen-Utheim&Hopfenbeck,2018)
NeedfornewperspecAvesonfeedbackuptake&literacyfordigitalenvironments
What is the impact of dialogic feedback on feedback uptake in the literature? Helped‘negoAatemeaning’(Zhu&Carless,2018)
anddevelopfeedbackliteracyoverAme(Reddyetal.2020;Han&Xu,2019)
What is the impact of dialogic feedback on feedback uptake in the literature? Helped‘negoAatemeaning’(Zhu&Carless,2018)
anddevelopfeedbackliteracyoverAme(Reddyetal.2020;Han&Xu,2019)
LimitaTonsofdialogicfeedbackin
pracTce
Dialogicpeerfeedbackmet‘contextual
difficulAes’
(Zhu&Carless,2018)
AffecAveandrelaAonalissuesinvolved(Han&Xu,2019)
ThedialogicUSMmodelof
feedbackuptake&literacy
(blendedandonline)
(Wood,2020)
(simplified)
Formative assessment and technology-mediated dialogic peer feedback (Nicol 2010; Carless, 2015)
Feedback viewed as social and constructed dialogic process (Nicol, 2010; Price et al., 2011)
Understanding the assessment
& feedback landscape
Self-assessment, goal setting &
regulation
ThedialogicUSMmodelof
feedbackuptake&literacy
(blendedandonline)
(Wood,2020)
(simplified)
Formative assessment and technology-mediated dialogic peer feedback (Nicol 2010; Carless, 2015)
Feedback viewed as social and constructed dialogic process (Nicol, 2010; Price et al., 2011)
• Dialogic and technology-mediated peer (and teacher) feedback (Nicol 2010; Carless, 2015)
• Feedback as social and constructed dialogic process (Nicol,2010;Priceetal.,2011)• Emphasis on student role in making feedback processes effective (NashandWinstone,2017)
Understanding the assessment
& feedback landscape
Self-assessment, goal setting &
regulation
Output
ThedialogicUSMmodelof
feedbackuptake&literacy
(blendedandonline)
(Wood,2020)
(simplified)
Ongoing technology-mediated dialogues support navigation of the USM processes
through co-regulation (Panadero et al. 2018; Er et al. 2020)
• Dialogic and technology-mediated peer (and teacher) feedback (Nicol 2010; Carless, 2015)
• Feedback as social and constructed dialogic process (Nicol,2010;Priceetal.,2011)• Emphasis on student role in making feedback processes effective (NashandWinstone,2017)
Understanding the assessment
& feedback landscape
Self-assessment, goal setting &
regulation
Motivation affect &
receptivity
Motivation affect &
receptivity
Motivation affect &
receptivity
Output
Improved Feedback Uptake & Literacy
ThedialogicUSMmodelof
feedbackuptake&literacy
(blendedandonline)
(Wood,2020)(simplified)
Ongoing technology-mediated dialogues support navigation of the USM processes
through co-regulation (Panadero et al., 2018; Er et al., 2020)
Online feedback practices (based on USM)
GoogleDriveandDocsforpeerfeedback,(andscreencast/Doc)teacher-
studentfeedback(toenablediscussion)ona1,200-wordessayover16
weekcourse)
Stages:
1.PreparaAon:socialisaAon,exemplars,criteria,modellingandpracAce
ofgoodsindrive.Introduceconceptsofdialogicfeedback,ZPD,
growthmindsetas‘primingforrecepAvity’.
2.Submitfirstdrat:fortechnology-mediatedpeerfeedback&ongoing
dialogue(1week)
Online feedback practices (based on USM)
GoogleDriveandDocsforpeerfeedback,(andscreencast/Doc)teacher-
studentfeedback(toenablediscussion)ona1,200-wordessayover16
weekcourse)
Stages:
1.PreparaAon:socialisaAon,exemplars,criteria,modellingandpracAce
ofgoodsindrive.Introduceconceptsofdialogicfeedback,ZPD,
growthmindsetas‘primingforrecepAvity’.
2.Submitfirstdrat:fortechnology-mediatedpeerfeedback&ongoing
dialogue(1week)
3.Submittoteacher:Teacherscreencastfeedback,anddialogue,(1
week)forsocialpresenceandenhancedfeedbackmessage
4.Finalsubmissiontoteacher:(screencastfeedbackandGoogleDocs
dialogue)teacherfeedbackfollowedbyareflecAontask
Research question
Whatweretheperceivedeffectsonfeedbackuptakeoftechnology-mediateddialogicfeedbackpracAce,basedontheUSMmodel?
Methodology
Insiderresearch,UCLethicsandBERA2018guidelines
ParAcipants:14SouthKoreanstudentsonadvancedacademicwriAngcourse(16weeks)
Socio-construcAvist,process-orientedinterpreAvistperspecAve
Methods
ReflecAvewriAng(N=14)(wriAng-to-learntasks)and
QuesAonnaires(N=14)
Providedprogressivefocusfor:
Methods
ReflecAvewriAng(N=14)(wriAng-to-learntasks)and
QuesAonnaires(N=14)
Providedprogressivefocusfor:
Semi-structuredinterviews(N=13),
InducAvethemaAccoding(Braun&
Clarke,2006)
DeducAveanalysisagainstmodel
Results and discussion
Feedback is useless if no one can get what it means (Jennyinterview)
1. Facilitates ‘negotiation of meaning’ development of peer feedback points
Thisclasswasspecialbecausethefeedbackwasmoretwo-way…Icouldques?onthefeedback,andwhythepeerthoughtIshouldrevisethat
partoftheessay.Thisaspectoftheac?vi?eswasthemosthelpfulforme.
(QuesAonnaire9)
AlsoIcanaskandrefutethefeedback…Throughthisprocess,IcanreflectanddevelopthefeedbackandeventuallyimprovemywriAng.
(QuesAonnaire7)
1. Facilitates ‘negotiation of meaning’ development of peer feedback points
Thisclasswasspecialbecausethefeedbackwasmore
two-way…Icouldques?onthefeedback,andwhythepeer
thoughtIshouldrevisethatpartoftheessay.Thisaspect
oftheac?vi?eswasthemosthelpfulforme.
(QuesAonnaire9)
AlsoIcanaskandrefutethefeedback…Throughthis
process,Icanreflectanddevelopthefeedbackand
eventuallyimprovemywriAng.
(QuesAonnaire7)
IcouldunderstandbeCerwhatotherscommentedabout
mywri?ng,andalsoIcouldmakemypointsclearerand
deliveredbeCerwhentheydidn'tunderstandthem.
(Ques?onnaire3)
Withoutdialoguetherewillbenoresultofthefeedback.
Feedbackisuselessifnoonecangetwhatitmeans,
havingadiscussionandhaving?metoclarifyitcanmake
thatfeedbackuseful.
(JennyInterview)
2. Multiple Learning cycles in online learning communities
Usuallyifit[peerreview]wasinpaperformitwouldbeaone-waythinganditwouldfinish,buthereIcoulddoafollow-upques?onoraskthemforfeedbackonmyanswer“ohisitokaynow?”andtheywouldsay“ohIthinkit’smuchbeCer”.
(Judyinterview)
…Icanexpanduponthatfeedback,andlikenotjustclarifythemeaning…ifsomeonesaidthispartofmysentenceiswrongsoIcould,forexample,say“whataboutthispart,isthispartokay”andtheywouldsay“that’sokay,butinconjunc?onwithmyabovefeedback
ifyoucouldcombinethistomakethis,that’llbeevenbeCer”.
(JunoInterview)
Thecycleshappenedthreeorfour?mes,Ithinkitwasfour
(Hayleyinterview)
Discussion of 1 & 2
1. NegoAaAngmeaning
AlsonotedbyZhu&Carless,(2018)but‘contextualdifficulAes’werealsonoted.
Inthisstudy:MulApleonlineexchangesoccurredoverAme
NegoAaAonofmeaningofpeerandfeedbackledtopercepAonofenhanceduptakeanduseandimprovedqualityofacAononfeedback(theorisedbyEretal.,
2020b)
2.FacilitatedcollecAvelearningcyclesinlearningcommuniAes
LearnershadopportunitytoelicitandcollaborateoncreaAonofaddiAonalscaffoldingtoimproveacAonabilityoffeedback
StudentsgothroughcompletefeedbackcycleswithpeersonlinetoconfirmachievementsindigitallearningcommuniAes
(seeReddyetal.2020).
3. Dialogue Reduced impact of cultural and emotional barriers to peer feedback engagement
InKorea,having“redmarks”alloveryourpapermeansabadthing.Itusuallyimpliesthatthereissomethingvery‘wrong’withthepaperandmanymayconsiderthisascri/cismratherthanhelpfulsugges?ons.
(JudyreflecAon)
Enablingfeedbacktobelikeatwo-wayconversa?onhelpsinseveralways.First,thiscreatesasenseof
rela?onshipwithpeersandthusstudentsfeelmorecomfortableingivingmanycommentstoeachotherbecausetheyknowthattheyarenotcri?cizingeachotherbutratherhelping.
(QuesAonnaire3)
Ithoughtatfirstfeedbackwasjustlikemarking…nowIknowthatit'shelpingeachother,it'sforothersandforme,alsogivingthefeedbackwouldalsohelpmineimprovelater…thebarrierswentdown
(Kevininterview)
4. Mutually supportive relationships – a pedagogic alliance of peers?
Itmadememoremo?vatedtoactuallyfixit,soIknowthateveryone'sengagedandpeoplearepuXngtheir?meandeffortindoingmyfeedback…itwouldbeasintonotuseit…Ithinkit'sjustlikeamo/vator,it'slikeasynergyeffect…
(JudyInterview)
IfeelI'mbeinganimportantperson,soeveryoneishelpingmetobeabeCerwriter,thatemo/onitselfhelpedmealotinwri?ng.
(Kevininterview)
I'mnotdoingthisworkaloneandthereissomeonewhoaresolookingforwardtomyimproving…Ithinkitwassupportandfeedbackthatmakemereallymo?vatedto
workhardandimprove.
(Jennyinterview)
Discussion of 3 & 4
CulturecannegaAvelyinfluenceunderstandingoftheconceptandfuncAonoffeedback(Evans,2013;Ryan&Henderson,2018)andemoAonalreacAonstoit(Tian&Lowe,2013)
Theneedtomanageaffectiswidelyacknowledgedinfeedbackliteracyresearch(Forsythe&Johnson,2017;PiK&Norton,2017;Priceetal.2011;Carless&Boud,
2018)
Resultsshowtechnology-mediaTon:
• EnhancedabilitytoposiAvelyhandleaffect(CarlessandBoud,2018)
And
• therelaAonalaspectsofpeerfeedbackpracAce–a‘pedagogicalliance’(Leighton&BustosGómez,2018)ofpeers
5. Online peer feedback processes nurtured feedback literacy (Carless and Boud 2018)
AppreciaTngfeedbackIlearnedthatwecangivefeedbackwhenwehaveopinion,
adviceandfeelings.Allthiscanhelpwritertoreflectreaders.Methodssuchas…googledrivemadeusgiveorgetfeedbackanywhereany?me,theprac?cesmadeusfreelytalkingwithpeopleregardlessofposi?on(whethersuperiororinferior).
(Ques?onnaire7)
Mybiggesttake-awayisthatfeedbackisopentoeveryone,
everywhereandevery?me.Itmademefeelfeedbackmorecomfortableandeasier.Thisledmetoac?velypar?cipateinfeedbackprocess.
(Ques?onnaire10)
Hmm,Ijustwantmorefeedbacknow,onallmyworks
(Hollyinterview)
Icouldn'tthinkofanyvalueofthefeedbackbefore,butthis
coursemademethinkthefeedbackisreallyvaluableandreallyappreciatesomeonegivingmefeedbackandhelpingmeimprove.
(Nahyuninterview)
5. Online peer feedback processes nurtured feedback literacy
MakingandrefiningjudgmentsQ:Howdidyouunderstandwhatgoodstandardsare?
Q:Howdidyoujudgethequalityofyourwork?
Peerreview!ItwasaliClebithardtotrytobecri?calofthree
otherwri?ngs,butIlearnedhowtobemorecri?calofmyownwri?ng.
(JudyreflecAon)
Givingfeedbackalsohelpsme,becauseIlearnalotaboutgoodwri?ngitselfwhenI'mgivingfeedback…ifIseeotherpeople'swri?ng,thenit'smuchmorecleartomewhatcanbedonebeCer,comparedtolookingatmyownwri?ng
(KylieInterview)
Igotmoreempiricalgroundsforwhyfeedbackisuseful…
moreexamplesoftheevidencebase…seeingthatoverandoverandexperiencingitmyself,hmmyouknowittotallyaffectedme,ittotallymademebelievethatfeedbackissoimportantandthenthischangeofbeliefIthinkitwillcon?nueevenaeertakingthisclassbecauseit'skindofchangeofyouraXtude.
(Kylieinterview)
6. Overcoming barriers/opening new learning spaces
Youknowlikeonnormalprojects…that'sreallyhardfor
peopletosynchronizetheir?me,sotheyusuallyhave
veryfewmee?ngsclosetothedeadlineandthenthat
that'sit.Butdrivedidn’thavea?melimit,soitwasable
tofacilitateconversa?onatall?mes,…thathelpedalot.
(Junointerview)
UsingDriveforpeerfeedbackgivespeerandmeenough/metothinkabouteachother'sworks.Ithinkithelpsustogivehigherqualityfeedback,makingusmore
mo?vated.
(Ques?onnaire4)
Theessaywaslike,eightpages,butthecommentson
thesideswereliketen,twentypages,soIdon’tthinkwe
couldhavedonethatmuchandthatextensivelyifyou
weredoingjustonpaperorduringclassorjusttalking
aboutitface-to-face…
(Junointerview)
6. Overcoming barriers/opening new learning spaces
Itwasalsoefficientintermsoffeedbackbecausewehad
thisunifiedpla{orm/placewherewecanfindallofeach
other'sworks,readthem,commentonthem,get
immediatenoAficaAonswhengenngcommentsonour
work,etc…
UsingDrivegreatlymoAvatedmetoacAvelyparAcipate
infeedback.
(QuesAonnaire10)
Ithinkthetechnologythatweuseduringtheclasswas
reallyimportantforfacilita?ngengagementofthe
feedbackprocess…becauseit'swayeasier,andwayfaster,way[more]approachablethannothavingit
(KylieInterview)
Ithinkthebestthingisthatyoucanaccesstheclass
materialsonmobile(Readingthingsonsubwayis
extremelyconvenient).
(Ques?onnaire6)
Discussion of 6: Overcoming barriers/opening new learning spaces
Undergraduatesreport:morelikelytoengage
withsimpleunified&convenientfeedback
technology (Winstoneetal.
2020)
Theseresultssuggest:
• convenience(unifiedpla{orm)
• noAficaAons• mobilepla{orms
• Ametoengage
Relateto:
• Strongengagementindigitalsenngs
• Openingnewspacesforlearningwithpeers
Lowered
‘behaviouralfricAon’(seeBehaviouralInsights
Team,2014)
Implications
Helpsinformgapregardingimpactofdialogic
peerfeedbackpracAcesinonlinesenngson
assessmentliteracy
DemonstrateshowlogisAcal,spaAaland
resourcelimitaAonsofdialogicfeedbackcanbe
overcome(synchronousdiscussionthrough
zoomalsobeneficial)
Illustratesthattheprocessesinvolvedin
feedbackuptakeandliteracy(Winstoneetal.2017;
Carless&Winstone2020)canbesupportedthrough
onlinepeerdialogue
PosiAveimplicaAonsforcontextswithlarge
cohorts
WorkloadposiAve
USM Offers insight into designing for uptake, & handling relational & practical issues for online & blended environments
(CarlessandWinstone,2020)
Limitations
LimittoclaimsaboutpercepAons,
generalisability.
Deployabilitywillvary.
Purelyonlineenvironmentsneedmore
socialpresence(CarlessandWinstone,2020)and
socialisaAonprocessestoinculcatetrust,for
example,5-stepsocialisaAonmodel(seeSalmon,2013).
Feedbackuptakeandliteracyisalsorelated
touniversityengagement,digitalliteracy(see
GourlayandOliver2018)andecological(Chong,
2020)orsociomaterialfactors(GraveK,2020).
Future research
Generalisability-StudycouldbereplicatedindifferentcontextsanduAlisingcohortstudiestodetermineimpactonaKainmentandsaAsfacAon.
Scalability,workloadneutrality,trainingandtechnologyacceptanceissues
Onlineonlyvsblendedenvironments
Final Conclusions
DemonstratedpotenAalofonlinefeedbackpracAcesbasedontheUSMmodelforfeedbackliteracy
Informsteacherfeedbackliteracyfordigitalsenngs(CarlessandWinstone,2020)
Exemplifieshowbenefitsofonlineenvironmentscanbeleveragedin‘TransformingAssessment’pracAces
ReferencesAjjawi, R., & Boud, D. (2017). Researching feedback dialogue: an interactional analysis approach. Assessment and Evaluation in Higher Education, 42(2), 252–265.
https://doi.org/10.1080/02602938.2015.1102863 Behavioural Insights Team. 2014.East—Four Simple Ways to Apply Behavioural Insights, April 11,
http://www.behaviouralinsights.co.uk/publications/east-four-simple-ways- to-apply-behavioural-insights/(accessed January 2020). Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Carless, D. (2015). Excellence in university assessment: Learning from award-winning practice. Routledge. Carless, D., & Boud, D. (2018). The development of student feedback literacy: enabling uptake of feedback. Assessment and Evaluation in Higher Education, 43(8), 1315–
1325. https://doi.org/10.1080/02602938.2018.1463354 Carless, D., & Winstone, N. (2020). Teacher feedback literacy and its interplay with student feedback literacy. Teaching in Higher Education, 1–14. https://doi.org/
10.1080/13562517.2020.1782372
Chong, S. W. (2020). Reconsidering student feedback literacy from an ecological perspective. Assessment and Evaluation in Higher Education, 0(0), 1–13. https://doi.org/10.1080/02602938.2020.1730765
Er, E., Dimitriadis, Y., & Gašević, D. (2020a). A collaborative learning approach to dialogic peer feedback: a theoretical framework. Assessment & Evaluation in Higher Education, 1–15. https://doi.org/10.1080/02602938.2020.1786497
Er, E., Dimitriadis, Y., & Gašević, D. (2020b). Collaborative peer feedback and learning analytics: theory-oriented design for supporting class-wide interventions. Assessment & Evaluation in Higher Education, 1–22. https://doi.org/10.1080/02602938.2020.1764490
Evans, C. (2013, March). Making Sense of Assessment Feedback in Higher Education. Review of Educational Research. https://doi.org/10.3102/0034654312474350 Gravett, K. (2020). Feedback literacies as sociomaterial practice. Critical Studies in Education, 00(00), 1–14. https://doi.org/10.1080/17508487.2020.1747099 Gravett, K., & Winstone, N. E. (2018, July 11). ‘Feedback interpreters’: the role of learning development professionals in facilitating university students’ engagement with
feedback. Teaching in Higher Education, pp. 1–16. https://doi.org/10.1080/13562517.2018.1498076 Han, Y., & Xu, Y. (2019). The development of student feedback literacy: the influences of teacher feedback on peer feedback. Assessment & Evaluation in Higher
Education, 1–17. https://doi.org/10.1080/02602938.2019.1689545 Handley, K., Price, M., & Millar, J. (2011). Beyond ‘doing time’: investigating the concept of student engagement with feedback. Oxford Review of Education, 37(4), 543–
560. https://doi.org/10.1080/03054985.2011.604951 Hill, J., & West, H. (2020). Improving the student learning experience through dialogic feed-forward assessment. Assessment and Evaluation in Higher Education, 45(1),
82–97. https://doi.org/10.1080/02602938.2019.1608908 Malecka, B., Boud, D., Carless, D., Malecka, B., & Boud, D. (2020). Eliciting , processing and enacting feedback : mechanisms for embedding student feedback literacy
within the curriculum embedding student feedback literacy within the curriculum. Teaching in Higher Education, 0(0),
ReferencesMolloy, E., Boud, D., & Henderson, M. (2019). Developing a learning-centred framework for feedback literacy. Assessment and Evaluation in Higher Education. https://
doi.org/10.1080/02602938.2019.1667955 Nash, R. A., & Winstone, N. E. (2017). Responsibility-Sharing in the Giving and Receiving of Assessment Feedback. Frontiers in Psychology, 8, 1519.
https://doi.org/10.3389/fpsyg.2017.01519 Leighton, J. P., & Bustos Gómez, M. C. (2018). A pedagogical alliance for trust, wellbeing and the identification of errors for learning and formative assessment. Educational
Psychology, 38(3), 381–406. Reddy, K., Harland, T., Wass, R., & Wald, N. (2020). Student peer review as a process of knowledge creation through dialogue. Higher Education Research &
Development, 1–13. https://doi.org/10.1080/07294360.2020.1781797 Ryan, T., & Henderson, M. (2018). Feeling feedback: students’ emotional responses to educator feedback. Assessment & Evaluation in Higher Education, 43(6), 880–892.
https://doi.org/10.1080/02602938.2017.1416456 Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional science, 18(2), 119-144. Steen-Utheim, A., & Hopfenbeck, T. N. (2018). To do or not to do with feedback. A study of undergraduate students’ engagement and use of feedback within a portfolio
assessment design. Assessment & Evaluation in Higher Education, 1–17. https://doi.org/10.1080/02602938.2018.1476669 Steen-Utheim, A., & Wittek, A. L. (2017). Dialogic feedback and potentialities for student learning. Learning, Culture and Social Interaction, 15, 18–30. https://doi.org/
10.1016/j.lcsi.2017.06.002 Tian, M., & Lowe, J. (2013). The role of feedback in cross-cultural learning: a case study of Chinese taught postgraduate students in a UK university. Assessment &
Evaluation in Higher Education, 38(5), 580-598. Winstone,N.E.,Nash,R.A.,Parker,M.,&Rowntree,J.(2017).SupporAngLearners’AgenAcEngagementWithFeedback:ASystemaAcReviewandaTaxonomyofRecipienceProcesses.
Educa?onalPsychologist,52(1),17–37.hKps://doi.org/10.1080/00461520.2016.1207538
Winstone,N.E.,Nash,R.A.,Rowntree,J.,&Parker,M.(2017).‘It’dbeuseful,butIwouldn’tuseit’:barrierstouniversitystudents’feedbackseekingandrecipience.StudiesinHigher
Educa?on,42(11),2026–2041.hKps://doi.org/10.1080/03075079.2015.1130032
Winstone,N.,Bourne,J.,Medland,E.,Niculescu,I.,&Rees,R.(2020b).“Checkthegrade,logout”:students’engagementwithfeedbackinlearningmanagementsystems.Assessment&
EvaluaAoninHigherEducaAon,1–13.hKps://doi.org/10.1080/02602938.2020.1787331
Wood,J.M.(2020).ADialogic,Technology-MediatedApproachtoSupporAngFeedbackEngagementinaHigherEducaAonContext:PerceivedEffectsonLearners’Feedback
Recipience(DoctoraldissertaAon,UCL(UniversityCollegeLondon)).
Zhu,Q.,&Carless,D.(2018).HigherEducaAonResearch&DevelopmentDialoguewithinpeerfeedbackprocesses:clarificaAonandnegoAaAonofmeaningDialoguewithinpeer
feedbackprocesses:clarificaAonandnegoAaAonofmeaning.hKps://doi.org/10.1080/07294360.2018.1446417
Questions?
@DrJoannaT
deakin.edu.au/learning/enhancing-your-practice/cradle
Inclusiveassessmentdesign:whatcanwelearnfromassessmentadjustments?
JoannaTai,MaryDracup,MerrinMcCracken,
YasminMobayad
TransformingAssessmentwebinarseries
AHEconferencepanel:lookingaheadto2021
15July2020
DeakinUniversityCRICOSProviderCode:00113B
@DrJoannaT
InclusiveAssessment:Whyisitimportant?
DeakinUniversityCRICOSProviderCode:00113B
IncreasingnumbersofstudentswithadisabilityinhighereducaAon(Grimesetal2017)
WearemorallyandlegallyobligedtoensurestudentshavethesameopportuniAestodemonstrateachievement(DisabilityDiscriminaAonAct(Cth)
1992;DisabilityStandardsforEducaAon(Cth)2005)
Adjustmentsdonotaddressstudentneeds(Waterfield&West2006)ANDthereisalackofevidence(Weis&Beauchemin2019)
SomestudentsdonotdisclosetheircondiAon(Grimesetal2019)
@DrJoannaT
InclusiveAssessmentDesign
DeakinUniversityCRICOSProviderCode:00113B
‘thedesignanduseoffairandeffec?veassessmentmethods
andprac?cesthatenableallstudentstodemonstratetotheir
fullpoten?alwhattheyknow,understandandcando’(Hockings2010,34)
• LinkedtoUniversalDesignforLearning(CAST2018)• Manyguidesforinclusiveassessmentdesignexist
(e.g.UniversityofPlymouth–Waterfield&West2006)
@DrJoannaT
CurrentProblemsintheRealWorld
DeakinUniversityCRICOSProviderCode:00113B
Staffarelessconfidentaboutinclusiveassessmentdesign(Morris,Milton&Goldstone2019)
StudentsaresAllexperiencingconsiderabledisadvantage
inassessment(Grimesetal2019)
Inclusioncouldbealsoconsideredmorebroadly(StenAford&
Koutsouris2020)
Howshouldwedirectourefforts?
@DrJoannaT
ResearchQuesTon&Aims
DeakinUniversityCRICOSProviderCode:00113B
WhataretheprioriTesforinclusiveassessmentdesign?
Whorequestsassessmentadjustments?
Whatisthenatureoftheadjustmentsrequested?
@DrJoannaT
Methods
DeakinUniversityCRICOSProviderCode:00113B
2018DisabilityResourceCentreAccessPlanrecords
DescripAvestaAsAcs
QualitaAveanalysisofopen-textentries
Ethicswaiverforaccessingde-idenAfiedUniversity
heldrecords
@DrJoannaT
Results
DeakinUniversityCRICOSProviderCode:00113B
Total2860AccessPlanscreatedin2018
72%studentshadanongoingcondiAon
17%hadmorethanonecondiAon
54%reportedamentalhealthcondiAon
31%reportedamedicalcondiAon
54%ofstudentsreportamentalhealthcondiTon
DeakinUniversityCRICOSProviderCode:00113B
n Ofanydisability Ofmentalhealth
Anxiety 826 29% 53%
Depression 531 19% 34%
PTSD 95 3% 6%
Bipolar 94 3% 6%
SchizoaffecAve
disorders&psychosis 41 1% 3%
PanicaKacks 30 1% 2%
@DrJoannaT
Adjustmenttypes
DeakinUniversityCRICOSProviderCode:00113B
Exams-2016(71%)
Onlinetests–1038(36%)
Assignments
Groupwork–319(11%)
OralpresentaAons–239(8%)
Labreport&pracAcaltasks–177(6%)
InclassparAcipaAon–256(9%)
@DrJoannaT
Examadjustments
DeakinUniversityCRICOSProviderCode:00113B
Typeofadjustment N %ofallAPs
Restbreaks 1612 56%
AddiAonalexamAme 1414 49%
AddiAonalreadingAme 176 6%
@DrJoannaT
QualitaTvecommentsaboutadjustments
DeakinUniversityCRICOSProviderCode:00113B
OngoingcommunicaAonandnegoAaAonshouldtake
place
Groupworkisimportantbutnotalwayspossible
Studentsfeltbadaboutlenngothersdown
In-person,in-classtaskscanbedifficult
Temporalflexibilityisvalued
@DrJoannaT
ImplicaTonsforassessmentdesign
DeakinUniversityCRICOSProviderCode:00113B
HowcanweberesponsivetocondiAonswithfluctuaAngor
variableimpact?
Implementstudentchoiceandflexibilityinassessment
ConsiderassessmentprogrammaAcally
Howcanweensureequivalenceofstandardsacrosstasks?
UseaninvitaAonalmetaphorforcriteria(BearmanandAjjawi2019)
@DrJoannaT
ImplicaTonsforassessmentdesignpt.2
DeakinUniversityCRICOSProviderCode:00113B
Forexams:
Improveeaseofaccesstoadjustments
Onlineexamsparadoxicallymayprovidebenefits(locaAon;inputopAons)
Forin-classassessment:
CertaintyaboutparAcipaAon&presentaAonschedules
AlternaAveopAons(pre-recorded;videolink)
Forgroupassignments:
SetupcommunicaAon/parAcipaAonagreements
IncludeconAngencyplanstomanageexpectaAons
@DrJoannaT
References
DeakinUniversityCRICOSProviderCode:00113B
Bearman,M.,&Ajjawi,R.(2019).Canarubricdomorethanbetransparent?InvitaAonasanewmetaphorforassessmentcriteria.Studiesin
HigherEduca?on,1–10.hKps://doi.org/10.1080/03075079.2019.1637842
CAST.2018.“UniversalDesignforLearningGuidelinesVersion2.2.”2018
DisabilityDiscrimina?onAct(1992)CommonwealthofAustralia
DisabilityStandardsforEduca?on(2005)CommonwealthofAustralia
Grimes,S.,Scevak,J.,Southgate,E.,&Buchanan,R.(2017).Non-disclosingstudentswithdisabiliAesorlearningchallenges:characterisAcs
andsizeofahiddenpopulaAon.AustralianEduca?onalResearcher,44(4–5),425–441.hKps://doi.org/10.1007/s13384-017-0242-y
Grimes,S.,Southgate,E.,Scevak,J.,&Buchanan,R.(2019).Learningimpactsreportedbystudentslivingwithlearningchallenges/disability.
StudiesinHigherEduca?on,0(0),1–13.hKps://doi.org/10.1080/03075079.2019.1661986
Hockings,C.(2010).InclusivelearningandteachinginhighereducaAon:asynthesisofresearch.RetreivedAugust,(April),1–67.
Morris,C.,Milton,E.,&Goldstone,R.(2019).Casestudy:suggesAngchoice:inclusiveassessmentprocesses.HigherEduca?onPedagogies,
4(1),435–447.hKps://doi.org/10.1080/23752696.2019.1669479
StenAford,L.,&Koutsouris,G.(2020).WhatareinclusivepedagogiesinhighereducaAon?AsystemaAcscopingreview.StudiesinHigher
Educa?on,0(0),1–17.hKps://doi.org/10.1080/03075079.2020.1716322
Waterfield,J.,&West,B.(2006).InclusiveAssessmentinHigherEduca?on:AResourceforChange.hKps://doi.org/10.4324/9781315045009
Weis,R.,&Beauchemin,E.L.(2019).AreseparateroomtestaccommodaAonseffecAveforcollegestudentswithdisabiliAes?Assessmentand
Evalua?oninHigherEduca?on,0(0),1–16.hKps://doi.org/10.1080/02602938.2019.1702922
Assessment & digital competences: building capacity with academic staff
The DCU-IUA Enhancing Digital Teaching and Learning Project
Suzanne Stone & Rob Lowney Teaching Enhancement Unit, Dublin City University
[email protected] ; [email protected] @suzielearning ; @lowneyrob
Slides at: bit.ly/EDTLAHE20
Overview Background to the EDTL project
Structure of project
#IUADigEd Community
DCU pilot development
Technology-enhanced assessment professional learning programme
Post-pilot implementation
Adapting to the Covid-19 crisis
Background to project: Europe
New Skills Agenda for Europe 2016 - Digital technologies are driving change in the global economy and we need to ensure our students are suitably prepared
EU Digital Education Action Plan 2018 - 11 actions to support technology use and the development of digital competences in education: 1. Making better use of digital technology for teaching and learning 2. Developing digital competences and skills
Courtesy of Dr Sharon Flynn, IUA
Background to project: National
National Skills Strategy 2025 - technology is one of the key drivers of change and improved digital skills will be vital for Ireland’s future
National Strategy for Higher Education to 2030 - teaching methods should increasingly be accompanied by e-learning and blended learning opportunities
Courtesy of Dr Sharon Flynn, IUA
Background to project: National
Digital Strategy for Schools 2015-2020 - provide rationale and action plan for embedding digital technologies in teaching, learning and assessment
Courtesy of Dr Sharon Flynn, IUA
Background to project: Sectoral / Local
IUA Universities Charter 2018 - need to build on the quality of the student experience in a digital age, commit to developing a coherent national programme in digital learning
DCU Strategic Plan 2017-2022 - University-wide curriculum review and an increase in digital learning initiatives and blended learning modules
Courtesy of Dr Sharon Flynn, IUA
Opportunity
IUA devised the project
Applied for funding under HEA Innovation and Transformation Call 2018
Digital learning programme
Courtesy of Dr Sharon Flynn, IUA
European context
Irish national context
University sector context
Institutional
context
Enhancing Digital Teaching and Learning (EDTL) Three-year nationally-funded project. Project aim:
enhance the digital attributes and educational experiences of Irish university students . develop, pilot, review and roll out an ambitious staff development programme to enhance the digital confidence, skills and competences of those who teach in Irish universities.
Courtesy of Dr Sharon Flynn, IUA
Approach
This project aims to mainstream digital in teaching and learning activities in Irish Universities, by addressing the professional development of all who teach or support teaching and learning.
Courtesy of Dr Sharon Flynn, IUA
Approach Courtesy of Dr Sharon Flynn, IUA
Pedagogy first - DigCompEdu Courtesy of Dr Sharon Flynn, IUA
European Framework for the Digital Competence of Educators (DigCompEdu)
Project structure Courtesy of Dr Sharon Flynn, IUA
Steering group ○ Senior management
IUA project manager ○ Dr Sharon Flynn
Project team ○ Project leads in 7
universities ○ Student intern working
at national project level ○ Local level interns at
each university
See edtl.blog for more information
One pilot, seven flavours Courtesy of Dr Sharon Flynn, IUA
Dublin City University
Focus on assessment through structured workshops and development activities, involving 3 pilot groups.
Trinity College Dublin
Redesigning an accredited module in Technology Enhanced Learning to focus on the Flipped Classroom.
University College Dublin
Leveraging digital for student feedback in the College of Science with the move to the new VLE ; initial needs analysis in College of Engineering, focus on creation of interactive animations for engagement.
Maynooth University
Conducted a needs assessment with 3 separate pilot groups, focus on information literacy and student skills, and aligning the learning outcomes of a new accredited module.
University College Cork
Leveraging the move to the new VLE to focus on staff digital skills.
University of Limerick
Conducting a needs assessment with 2 pilot groups, enhancing staff and student digital skills to support peer and self-assessment/
NUI Galway Focus on curriculum (re)design and development of digital resources through a structured approach. Alignment with existing accredited CPD.
#IUADigEd Community Courtesy of Dr Sharon Flynn, IUA
See edtl.blog for more information
#IUADigEd webinars
EDTL Approach: Consider Content & Activities, Project Team, Monday 20 July.
International Perspectives on Teaching Online, panel session with Maha Bali, Jesse Stommel and
Sukaina Waljil, hosted by Kate Molloy, Monday 27 July at 4pm.
EDTL Approach: Consider Communication & Engagement, Project Team, Monday 10 August.
EDTL Approach: Consider Assessment & Feedback, Project Team, Monday 17 August.
DCU Pilot phase (Sep-Dec 2019) DCU’s focus: technology-enhanced assessment (TEA)
Why assessment?
4 work packages: Staff Development Programme
Development and Maintenance of Online Resources
Ongoing Technical Enhancements for the VLE (Moodle)
Communications and Dissemination
Staff development programme based around suite of 10 workshops drawing on established professional learning opportunities and staff expertise
DCU Pilot phase (Sep-Dec 2019)
Focus on ‘Why’ of TEA, ‘How to’
later
Starting point: workshops already
offered by TEU
Refine workshops for discipline
specific needs Input to come later
Staff development Programme: Underlying principles
Iterateandsustain Reviewand
refine ImplementTEAassessment Ongoingsupport
PlanningaTEAassessment Individual/groupconsultancies
Workshopx3
engagement Pre-workshoplearning
Customworkshop
development Developmentofsuiteof
Digitalassessmentworkshops
Iconscourtesyof
Fontawesome.com
Disciplinespecificneedsanalysis:
EducaAon&Psychology
Project structure
Staff Development Programme: Lessons/Adaptations Timing is crucial! Early engagement with schools required Flexibility needed for delivery Even before Covid-19! Discussion more important than practical work - removed Pedagogy focused workshop needs to be followed by technology ‘how to’ workshop - now built into programme Pre-workshop tasks too time consuming - removed DigCompEdu: Participants need scaffolding to engage. Gamification Digital Pursuit adapted from Jisc Digital Pursuit* Development of additional online resources (non-tech focused) required - work ongoing, collaborations with participants explored
*Hibberson, S. (2020). Digital Pursuits - playing the digital capability game! - Inspiring learning. [online] Inspiring learning. Available at: https://inspiringlearning.jiscinvolve.org/wp/2019/03/digital-pursuits-playing-the-digital-capability-game/ [Accessed 13 Feb. 2020].
DCU Pilot phase (Sep-Dec 2019)
Evaluation underway Student voice = key - focus groups Bamber’s framework - beyond quantitative data towards evidence of impact on practice National Forum badge on offer to participants Reference Bamber, V. (Ed.). (2013). Evidencing the value of educational development. SEDA Special No 34. London: SEDA. ISBN 978-1-902435-56-5.
DCU Pilot phase Evaluation
At national level - project refocused to support the blended/online delivery for 2020/2021
EDTL approach - curation of resources/framework
Crisis = Opportunity: Alternative assessment
Impact of pilot: EDTL participants felt prepared for crisis particularly in respect of designing alternative assessments (anecdotal feedback)
Adapting to Covid-19
Driven by identified needs at various levels
Discipline-specific
Tailored to participants’ needs
Scaffolded and ongoing support
Student voice central to align skills development for staff with student needs
Mixture of type of engagement
Adaptation to needs resulting from Covid-19 crisis - a real opportunity
In summary
Participants: Testimonials “I was delighted to have the opportunity to partake in the EDTL Project. The team were fantastic and provided a great platform for learning. It was very practical. There was a great system of support provided when we were trying the new techniques which gave me confidence in using these. I enjoyed embedding these new strategies in my teaching as I could feel the level of engagement [from students] rising. The students were definitely engaged and enjoyed asking questions (e.g. mentimeter) and receiving feedback on their learning (E.g. in Kahoot)! Feedback from the students on these techniques was very positive. these were definitely informing teaching and learning”
DCU Pilot phase (Sep-Dec 2019)
Participants: Testimonials “Participating in the EDTL workshops benefitted me because while there was an overall framework for the CPD, the detail was built around the identified needs of the group in question and this was really useful and one of the key reasons (I think) that staff engaged well with the initiative.”
DCU Pilot phase (Sep-Dec 2019)
Assessment and digital competences: building capacity with academic staff The DCU-IUA Enhancing Digital Teaching and Learning Project Suzanne Stone & Rob Lowney Teaching Enhancement Unit [email protected] ; [email protected] @suzielearning ; @lowneyrob Slides at: https://bit.ly/EDTLAHE20
Webinar Series
Webinar Session feedback With thanks from your hosts Professor Geoff Crisp, Deputy Vice-Chancellor & Vice-President Academic University of Canberra g.crisp[at]canberra.edu.au Dr Mathew Hillier, Macquarie University mathew.hillier[at]gmail.com Recording available http://transformingassessment.com
e-Assessment SIG
Next session 5 Aug 2020 Adapting to COVID19 by
ignoring proctoring: catalysing alignment of
online teaching, learning and assessment
Register http://taw.fi /5aug2020