Upload
abelardo-pardo
View
653
Download
1
Embed Size (px)
Citation preview
Abelardo Pardo (@abelardopardo)Faculty of Engineering and IT slideshare.net/abelardo_pardo
Queen’s U
niversity flickr.com
Feedback at scale with a little help from my algorithms
Susana Fernandez flickr.com
Research Seminar Universitat Pompeu Fabra
Barcelona Spain 5 May 2016
Feedback at scale with a little help from my algorithmsAbelardo Pardo 2
Rishi S flickr.com
About me
Teaching
First year engineering course
Computer systems
350 students
Plenary 2 hour lecture + 2 hour tutorial + 3 hour laboratory session
Active learning
Heavy use of technology
Research
Educational technology
Learning analytics
Data-guided feedback
Naturalistic experiments
Feedback at scale with a little help from my algorithmsAbelardo Pardo 3
gingiber flickr.com
Feedback and Data
Decision Trees
Data driven rubric
Design and production
Feedback at scale with a little help from my algorithmsAbelardo Pardo 4
gingiber flickr.com
Feedback and Data
Feedback at scale with a little help from my algorithmsAbelardo Pardo 5
Eric flickr.com Systematically low ratings of
the feedback provided to the students
Feedback at scale with a little help from my algorithmsAbelardo Pardo 6
Krause, K.-L., Hartley, R., James, R., & McInnis, C. (2005). The First Year Experience in Australian Universities: Findings from a decade of National Studies. University of Melbourne: Centre for the Study of Higher Education.
Eleaf flickr.com The feedback question gets systematically lower values in student surveys
Feedback at scale with a little help from my algorithmsAbelardo Pardo 7
Huxham, M. (2007). Fast and effective feedback: are model answers the answer? Assessment & Evaluation in Higher Education, 32(6), 601-611. doi:10.1080/02602930601116946
• Lateness
• Uncertainty about criteria and contexts
Loonyhiker flickr.com
• Ambiguity or opacity (What do you mean?)
• Negativity
Loonyhiker flickr.com
Feedback at scale with a little help from my algorithmsAbelardo Pardo 8
Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher Education: Learning for the Longer Term. London and New York: Routledge.
Perceived as an administrative chore instead of a pedagogical necessity
Marcin W
ichary flickr.com
Feedback at scale with a little help from my algorithmsAbelardo Pardo 9
Joel Sage flickr.com“sustainability of feedback is under threat… pervasive student concerns about the provision of feedback in
an era of mass education”
Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher Education: Learning for the Longer Term. London and New York: Routledge.
Feedback at scale with a little help from my algorithmsAbelardo Pardo 10
Hounsell, D. (2008). The Trouble with Feedback. New Challenges, Emerging Strategies. Interchange(2).
Feedback has a Cinderella statusBitslam
mer flickr.com
Feedback at scale with a little help from my algorithmsAbelardo Pardo 12
Allen Lai flickr.com
Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in Higher Education: Learning for the Longer Term. London and New York: Routledge.
Downward spiral: Student disenchantment mounts when feedback is uninformative and still,
less feedback is provided
Solution: provide high-value feedback, rethink the role of student, enhance relation between guidance and feedback
Feedback at scale with a little help from my algorithmsAbelardo Pardo 13
Brandon Martin-Anderson flickr.com
• Can data guide the provision of effective feedback
• Can this provision be done at scale?
Feedback at scale with a little help from my algorithmsAbelardo Pardo 14
Dm
itry Grigoriev flickr.com
Learning Analytics: measure, collect, analyse data about learners to understand and improve their learning and the environment in which it occurs
Feedback at scale with a little help from my algorithmsAbelardo Pardo 15
Eustaquio Santimano flickr.com
Collect
Report
Analyze
Act
Refine
Feedback at scale with a little help from my algorithmsAbelardo Pardo 17
Eustaquio Santimano flickr.com
Collect
Report
Analyze
Act
Refine
Feedback at scale with a little help from my algorithmsAbelardo Pardo 18
gingiber flickr.com
Feedback and Data
Decision Trees
Feedback at scale with a little help from my algorithmsAbelardo Pardo 19
William
Murphy flickr.com
• 13 Week first year Engineering • Weekly activities (formative/summative) • Videos, MCQ, Exercises, dashboard • n = 272, Weeks 2-5 and 7-13
Pardo, A., Mirriahi, N., Martínez-Maldonado, R., Jovanović, J., Dawson, S., & Gasevic, D. (2016). Generating Actionable Predictive Models of Academic Performance. Paper presented at the International Conference on Learning Analytics and Knowledge, Edinburgh. doi:10.1145/2883851.2883870
Feedback at scale with a little help from my algorithmsAbelardo Pardo 20
Oliver Braubach flickr.comObjective
1. Data indicators close to learning design
2. Predictive model
3. Bridge between model and application
4. Straightforward delivery method
Feedback at scale with a little help from my algorithmsAbelardo Pardo 21
Louish Pixel flickr.com
• Event counts from interactive course material
• Midterm/final exam scores
• Recursive partitioning
• Divide cohort into performance categories
Feedback at scale with a little help from my algorithmsAbelardo Pardo 22
Data collected
• Indicators are directly connected with learning design
• Data structure shaped by the schedule (weeks)
• Data available in a per-week basis
• What is the expected midterm/final score in week n?
Feedback at scale with a little help from my algorithmsAbelardo Pardo 23
Result Example• Week 10
• Predicted score at leaves (out of 40)
• Conditions at nodes
• If (EXC.in >=22) and (VID.PL < 8.5) then score = 6
Feedback at scale with a little help from my algorithmsAbelardo Pardo 24
• Each leaf node represents a group of students with their estimated score.
• Example: 6, 8.3, 8.4, 9.4, 9.9, 10, 15 (out of 40)
• Intervention: suggested work before exam
Result interpretation
Feedback at scale with a little help from my algorithmsAbelardo Pardo 25
shabnam m
ayet Flickr.com
RMSE: Root mean square error, MAE: Mean absolute error
Performance
Feedback at scale with a little help from my algorithmsAbelardo Pardo 26
gingiber flickr.com
Feedback and Data
Decision Trees
Data driven rubric
Feedback at scale with a little help from my algorithmsAbelardo Pardo 27
2 Hours
2 Hours
3 Hours
Pardo, A. & Mirriahi N. (in press). Design, Deployment and Evaluation of a Flipped Learning First Year Engineering Course. In C. Reidsema, L. Kavanagh, R. Hadgraft & N. Smith (Eds.), Flipping the Classroom: Practice and Practices. Singapore: Springer.
2014 Edition
Feedback at scale with a little help from my algorithmsAbelardo Pardo 29
1. realtime feedback
2. convey engagement
3. compare with rest of cohort
4. reset weekly
5. one click away from notes
Approach 1 (2014)Sean D
reilinger flickr.com
Feedback at scale with a little help from my algorithmsAbelardo Pardo 31
Technology
Technology
Week N
Feedback at scale with a little help from my algorithmsAbelardo Pardo 33
No statistically significant difference in the rating of feedback (2013 edition, M=3.25, SD=0.97; 2014 edition, M=3.35, SD=1.03); t(389.78) = -0.97, p <0.17
Feedback at scale with a little help from my algorithmsAbelardo Pardo 34
1. encourage contact between student and faculty (forum)
2. uses active learning techniques
3. gives prompt feedback
4. emphasizes time on task
5. kind, specific, helpful
Approach 2 (2015)Ben Manson flickr.com
Ben Manson flickr.com
Feedback at scale with a little help from my algorithmsAbelardo Pardo 36
You should take a more careful look at how symbols are encoded in the video. Would you be able to encode/decode UAL symbols without looking at the video?
Good initial work. However, did you understand the trick to handle encoding with a variable number of bits? Would you be able to provide an example?
Good work. Would you be able to come up with your own machine language and your encoding scheme? Remember that it has to be unambiguous.
Thorough work with the task about machine language encoding. Give it a quick review before the midterm.
Q1 Q2 Q3 Q4
Instructor
Feedback at scale with a little help from my algorithmsAbelardo Pardo 39
1. Data collected weekly
2. Email sent at end of week
3. 4 weeks before the midterm
nateOne flickr.com
Feedback at scale with a little help from my algorithmsAbelardo Pardo 40
Significant difference in the rating of feedback (2014 edition, M=3.35, SD=1.03; 2015 edition, M=3.82, SD=0.90); t(389.78) = -4.88, p <10-6 Effect size (Cohen’s d) = 0.49. Medium positive effect
Feedback at scale with a little help from my algorithmsAbelardo Pardo 41
Significant difference in the midterm score (2014 edition, M=12.80, SD=4.79; 2015 edition, M=13.83, SD=4.89); t(684.5) = -2.86, p < 0.002 Effect size (Cohen’s d) = 0.21. Small positive effect
Feedback at scale with a little help from my algorithmsAbelardo Pardo 42
gingiber flickr.com
Feedback and Data
Decision Trees
Data driven rubric
Design and production
Feedback at scale with a little help from my algorithmsAbelardo Pardo 43
Departm
ent for Business Innovation & Skill flickr.com • Content creation
• Interactive
• Multi-view (tutor, bilingual)
• Focus on content (not style)
• Platform agnostic
• Design-embedded analytics
Feedback at scale with a little help from my algorithmsAbelardo Pardo 44
David M
ichalczuk flickr.com
• Markup language
• Batch processing
• Sphinx-doc
• Webdav gateway for publishing
• Self-contained web site
• Extensible language: macros for design elements
Feedback at scale with a little help from my algorithmsAbelardo Pardo 47
Single source approach
• Plain text
• Version control
• Distributed production
• Suitable for ed designers
Feedback at scale with a little help from my algorithmsAbelardo Pardo 49
“The interactive quizzes during class are excellent” “I love the quizzes”
‘Learning was supported by useful electronic learning resources’ Agreement increased from 78% to 98%
Professional Practice of Radiography (PG). A/Prof. Mark McEntee
Feedback at scale with a little help from my algorithmsAbelardo Pardo 50
https://bitbucket.org/abelardopardo/reauthoring
•Activity duration
•MCQ
•Video embedding
•Difficulty/Usefulness 2D grid
•Tutor view
•Tutor feedback
•Coupled with tracking
• Automatic back-annotation
Feedback at scale with a little help from my algorithmsAbelardo Pardo 51
Fabien CAM
BI flickr.com
• Feedback is delicate and messy
• Data can help scaling
• Good focal point for analytics
• Provide intuitive tools to instructors
• Authoring is a challenge
Abelardo Pardo (@abelardopardo)Faculty of Engineering and IT slideshare.net/abelardo_pardo
Queen’s U
niversity flickr.com
Feedback at scale with a little help from my algorithms
Susana Fernandez flickr.com
Research Seminar Universitat Pompeu Fabra
Barcelona Spain 5 May 2016