Upload
vitomir-kovanovic
View
59
Download
1
Embed Size (px)
Citation preview
A Novel Model of Cognitive Presence
Assessment Using Automated
Learning Analytics Methods
Vitomir Kovanovic
School of Informatics
The University of Edinburgh
http://vitomir.kovanovic.info
#vkovanovic
2
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Global Grand ChallengesShifting from an era of scarcity to abundance http://singularityu.org/global-grand-challenges
3
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Four challenges for next decade
1. Affordable, sustainable energy.
2. Cures for HIV and neurodegenerative diseases like Alzheimer's.
3. Protection from future health epidemics.
4. Tools to provide a world-class education to all students.
4
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
1 GLOBAL NEED
FOR EDUCATION
Significant Challenges in Higher Education
2 INCREASE
STUDENT
ENGAGEMENT
3 DECREASING
UNIVERSITY
FUNDING
MORE AND MORE
UNIVERSITIES TURNING
TO TECHNOLOGY
MORE AND MORE
UNIVERSITIES TURNING
TO TECHNOLOGY
5
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
ONLINE &
BLENDED
Blended is the new norm.
Increasing interest in online learning.
MOOCs and new forms of delivery.
LEARNING
DATA
Large amounts of data collected.
Improve student experience.
Understand learning processes.
NEW MARKETS
AND MODELS
Workspace learning.
Lifelong learning.
Developing world.
Technology & Data TrendsThree important ways in which technology is shaping the future of education
6
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Learning analytics is the measurement, collection, analysis and reporting of data about learners and their
contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.
CFP for the First Learning Analytics and Knowledge Conference
https://tekri.athabascau.ca/analytics/
What is Learning Analytics?
Learning AnalyticsMaking sense of the available learning data
Measurement Collection Analysis Reporting
7
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
What can Learning Analytics bringImprovement in teaching quality, student retention, learning outcomes, and understanding of learning processes
8
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Traditional Online Learning
How can we use the educational data to understand student
learning?
Massive Open Online Courses
How to adapt existing models of online learning in MOOC
settings?
What are the challenges of MOOC pedagogies?
Learning Analytics for Online Learning
9
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
1 PedagogyThe use of Learning analytics must be aligned with the existing
pedagogical approaches used in particular learning context.
Driving PrinciplesFoundations for my learning analytics research
2 Data analysis
Use automated data mining techniques to process large amounts of
student-generated data to understand how student learn in online
setting
3 Tools
Develop automated learning analytics tools that can be used to assess
student learning and improve research in online and distance
education.
10
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Community of Inquiry FrameworkDimensions of student online learning experience
Social-constructivist model of learning
- Students construct knowledge rather than receive it
- Discussion is an essential part of learning
- Inquiry basic learning activity
Widely used in “traditional” online
and distance education
- Strong teacher presence
- Up to ~ 30 students
Cognitive Presence
Teaching Presence
Social Presence
Student
Experience
Development of critical
and deep thinking skills
Social climate in the course
1. Group cohesion
2. Interactivity
3. Group affectivity
Instructor’s role in the course:
1. Design & organization,
2. Facilitation,
3. Direct instruction
(Garrison, Anderson, and Archer, 1999)
11
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Cognitive PresenceOperationalization of critical and deep thinking skills
• Exploring and testing solutions
• Dilemma or problem identified
• Synthesis of relevant information
• Brainstorming
• Exploring ideas
2. Exploration
3. Integration
4. Resolution
1. Triggering
Event
Shared world of discourse
Private world of reflection
“an extent to which the participants
in any particular configuration of a
community of inquiry are able to
construct meaning through
sustained Communication”
(Garrison, Anderson, and Archer,
1999, p. 89)
Definition
12
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
1 Content analysisCoding scheme for each
presence
Assessment of three presencesHow do we measure levels of cognitive, social and teaching presence?
2 Self-reported survey34 items, 1-5 Likert scale
questions
• Experience with the coding scheme
• Inter-rater reliability issues
• Time-consuming
• Non-real time
• Primary use: research
• Fait accompli: results known after the
course is over
• Self-selection bias
• Invasive
(Arbaugh et al., 2008).
(Garrison, Anderson,
and Archer, 1999)
13
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
1 From the online
discussions
Assessment based on Learning AnalyticsAutomate as much as possible
2 Based on the use
of educational
technology
• Identifying different study
approaches
• How those approaches relate
to development of cognitive
presence?
• Clustering based on trace
data
Shared world learning
Private world learning
• Automating CoI coding
scheme for cognitive
presence
• Real-time analysis of student
cognitive presence
development
• Easier adoption and research
• Text mining of student online
discussion messages
1 Traditional online
courses
2 MOOCs
Athabasca University
fully online course
Delft University of
Technology MOOC
data
Types of data and analytics Contexts
14
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Cognitive presence assessment frameworkDriven by Evidence-Centered Design (ECD) framework
ASSESSMENT IMPLEMENTATIONAutomated cognitive presence classification system
Technology use profiling system
ASSESSMENT FRAMEWORKEvidence-centered design (ECD) assessment
framework: Student model, Evidence model and Task
model
EDUCATIONAL TECHNOLOGYTechnology use trace data and online discussion
data from Learning Management Systems (LMS)
and MOOC platforms:
THEORY & PEDAGOGYSocial constructivist learning & Community of Inquiry
model
Kovanovic et al. (in-press)
15
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Cognitive presence classification system
Five-class text classification problem:
1 – Triggering event,
2 – Exploration,
3 – Integration,
4 – Resolution,
0 – Other (non cognitive).
Data from six offers of a fully online course
1,747 messages coded for cognitive presence
Developed random forest classifier
Extracted features:
Linguistic Inquiry Word Count features (LIWC):
93 different counts indicative of different psychological processes (e.g.,
affective, cognitive, social, perceptual)
Coh-Metrix features:
108 metrics of text cohesion
LSA coherence:
Average LSA similarity of message’s paragraphs to each other.
LSA space is built from Wikipedia articles related to concepts extracted
from the topic start message (using TAGME).
Named entity count:
Number of concepts related to DBPedia computer science category (using
DBPedia spotlight)
Context features:
Number of replies
Message depth
Cosine similarity to previous/next message
Thread start/end Boolean indicators
Kovanovic et al. (2016)
16
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Performance evaluation
• We obtained 70.3% classification accuracy (95% CI[0.66, 0.75]) and 0.63 Cohen’s κ.
• Significant improvements over Cohen’s κ of 0.41 and 0.48 reported in Kovanovic et al. (2014)
and Waters et al. (2015) studies.
• The feature space ~ 100x smaller
• The feature space is also more generalizable
• We provided more detailed operationalization of the cognitive presence coding scheme.
Current work:
– Analysis of discussion messages from “Functional Programming” MOOC by TU Delft
– Investigation of the suitability of the same classification approach in various contexts
(Essay analysis, Twitter messages)
Kovanovic et al. (2016)
17
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Feature importance
Phase
# Variable Description MDG* Other TE Exp. Int. Res.
1 cm.DESWC Number of words 32.91 55.41 80.91 117.71 183.30 280.68
2 ner.entity.cnt Number of named entities 26.41 13.44 21.67 28.84 44.75 64.18
3 cm.LDTTRa Lexical diversity, all words 21.98 0.85 0.77 0.71 0.65 0.58
4 message.depth Position within a discussion 19.09 2.39 1.00 1.84 1.87 2.00
5 cm.LDTTRc Lexical diversity, content words 17.12 0.95 0.90 0.86 0.82 0.78
6 cm.LSAGN Avg. givenness of each sentence 16.63 0.10 0.14 0.18 0.21 0.24
7 liwc.Qmark Number of question marks 16.59 0.27 1.84 0.92 0.58 0.38
8 message.sim.prev Similarity with previous message 16.41 0.20 0.06 0.22 0.30 0.39
9 cm.LDVOCD Lexical diversity, VOCD 15.43 12.92 28.99 53.57 83.47 97.16
10 liwc.money Number of money-related words 14.38 0.21 0.32 0.32 0.65 0.99
11 cm.DESPL Avg. number of paragraphs 12.47 4.26 6.37 7.49 10.17 14.05
12 Message.sim.next Similarity with next message 11.74 0.08 0.34 0.20 0.22 0.22
13 Message.reply.cnt Number of replies 11.67 0.42 1.44 0.82 1.10 0.84
14 cm.DESSC Sentence count 11.67 4.28 6.36 7.49 10.17 14.29
15 lsa.similarity Avg. LSA sim. between sentences 9.69 0.29 0.47 0.54 0.62 0.67
16 cm.DESSL Avg. sentence length 9.60 11.88 13.62 16.69 19.36 21.73
17 cm.DESWLsyd SD of word syllables count 8.92 0.98 1.33 0.98 0.97 0.97
18 liwc.i Number of FPS* pronouns 8.84 4.33 2.82 2.37 2.51 2.19
19 cm.RDFKGL Flesch-Kincaid Grade level 8.29 7.68 10.30 10.19 11.13 11.99
20 cm.SMCAUSwn WordNet overlap between verbs 8.14 0.38 0.48 0.51 0.50 0.47
* MDG - Mean decrease Gini impurity index, FPS - first person singular
Phase
# Variable Description MDG* Other TE Exp. Int. Res.
1 cm.DESWC Number of words 32.91 55.41 80.91 117.71 183.30 280.68
2 ner.entity.cnt Number of named entities 26.41 13.44 21.67 28.84 44.75 64.18
3 cm.LDTTRa Lexical diversity, all words 21.98 0.85 0.77 0.71 0.65 0.58
4 message.depth Position within a discussion 19.09 2.39 1.00 1.84 1.87 2.00
5 cm.LDTTRc Lexical diversity, content words 17.12 0.95 0.90 0.86 0.82 0.78
6 cm.LSAGN Avg. givenness of each sentence 16.63 0.10 0.14 0.18 0.21 0.24
7 liwc.Qmark Number of question marks 16.59 0.27 1.84 0.92 0.58 0.38
8 message.sim.prev Similarity with previous message 16.41 0.20 0.06 0.22 0.30 0.39
9 cm.LDVOCD Lexical diversity, VOCD 15.43 12.92 28.99 53.57 83.47 97.16
10 liwc.money Number of money-related words 14.38 0.21 0.32 0.32 0.65 0.99
11 cm.DESPL Avg. number of paragraphs 12.47 4.26 6.37 7.49 10.17 14.05
12 Message.sim.next Similarity with next message 11.74 0.08 0.34 0.20 0.22 0.22
13 Message.reply.cnt Number of replies 11.67 0.42 1.44 0.82 1.10 0.84
14 cm.DESSC Sentence count 11.67 4.28 6.36 7.49 10.17 14.29
15 lsa.similarity Avg. LSA sim. between sentences 9.69 0.29 0.47 0.54 0.62 0.67
16 cm.DESSL Avg. sentence length 9.60 11.88 13.62 16.69 19.36 21.73
17 cm.DESWLsyd SD of word syllables count 8.92 0.98 1.33 0.98 0.97 0.97
18 liwc.i Number of FPS* pronouns 8.84 4.33 2.82 2.37 2.51 2.19
19 cm.RDFKGL Flesch-Kincaid Grade level 8.29 7.68 10.30 10.19 11.13 11.99
20 cm.SMCAUSwn WordNet overlap between verbs 8.14 0.38 0.48 0.51 0.50 0.47
Kovanovic et al. (2016)
18
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
The higher the cognitive presence…
• The longer the message, the more paragraphs and
sentences.
• More concepts mentioned (more named entities).
• The lower the lexical diversity (both at content level
and in general).
• The later its position in the thread. Except non-
cognitive messages, they tend to occur closer to the
end as well.
• The fewer the question marks (except non-cognitive,
they have fewest question marks)
• The higher the average length of sentence and their
similarity to each other.
• The more money-related terms.
Operationalization of cognitive presence
• Exploring and testing solutions
• Dilemma or problem identified
• Synthesis of relevant information
• Brainstorming
• Exploring ideas
2. Exploration
3. Integration
4. Resolution
1. Triggering
Event
Shared world of discourse
Private world of reflection
• Lowest message readability
• Syllabi count inconsistent
• Most replies
• Low similarity with the next message
• More replies than exploration and resolution
• Least replies
• Question marks more frequent than integration
2. Exploration
3. Integration
4. Resolution
1. Triggering
Event
Shared world of discourse
Private world of reflection
Kovanovic et al. (2016)
19
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Technology use profiling system
• Used 200,000 LMS trace data records of student learning activities
• Hierarchical clustering (Ward method, Euclidean distance)
• MANOVA analysis of differences in cognitive presence, followed by discriminant factor analysis (DFA) and ANOVA analysis
• We identified six different technology use profiles.
• Large number of students poorly regulated technology use.
• Students with different technology-use profiles had significant differences in the levels of their cognitive presence. We
observed large effect size (partial eta2 = .54).
• Quality of interactions is more important than the quantity for the development of cognitive presence.
Kovanovic et al. (2014)
20
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
In progress: Communities of Inquiry for
Massive Open Online Courses
Used the data from five TU Delft MOOCs
2,446 survey responses
5-item Likert scale responses
Originally, CoI studies found 3-factor structure to best describe CoI survey instrument
Goal: can we replicate those findings in MOOCs?
21
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Factor analysis
Original model holds in the MOOC context
CP6: Online discussions were valuable in helping me
appreciate different perspectives
Affectivity
Resolution
Course organization
& design
Better fit: 6 factor model:
22
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
How are they interactingStructural equation model between three presences
• Direct effect of Teaching Presence on Cognitive presence, and
• Mediating effect of social presence on teaching presence and cognitive presence relationship
Cognitive presence
Social presence
Teaching presence
Cognitive presence
Social presence
Teaching presence
Original SEM model MOOC model
0.52
0.51 0.40 0.60 0.22
0.35
23
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Technology-use in MOOCs
• Adopted same methodology as in “traditional” online setting
• Significant MANOVA, (p=0.038).
• Univariate differences in the cognitive presence levels for the resolution phase (C1 – C2 and C1 – C3 )
24
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Summary
• Important to realize the tremendous potential educational data has for improving teaching and learning.
• How this potential of the data fits the existing models of online and blended learning?
• How can we design analytics that are flexible enough and provide meaningful insights about student learning?
• Can we better support research in online education by automating existing approaches used by researchers?
• Can we use the existing pedagogical models in MOOC setting?
• How can we make student learning in MOOCs les solitary?
25
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Summary: cognitive presence assessment
framework
• We developed a framework for assessing the levels of cognitive presence
• We focused on two types of insights: from trace data & discussion data and
• We focused on two contexts: traditional online courses and MOOCs
• We developed an assessment framework for cognitive presence using Evidence-centered design (ECD) guidelines
• Automated existing tool for measuring cognitive presence
• Enables easier adoption of Community of inquiry model
• Better operationalization of the phases of cognitive presence
• We developed a student profiling method that can be used to identify different profiles of student technology use
• Better understand online learning, especially related to student self-regulation of learning
• Insights into their private solitary learning not displayed in online discussion transcripts
26
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Summary: Cognitive presence in MOOCs
• Explored the use of CoI model in MOOC context and use existing pedagogies at scale
• We conducted a factor analysis of Community of Inquiry model in the MOOC context:
• Original CoI model holds but with certain factors being more emphasized than others
• The role of social group is much weaker in MOOCs
• We identified profiles of students based on their technology use
• Identified profiles differ in the levels of their cognitive and social presence
• The differences are much smaller, primarily due to the use of survey-based instrument
27
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Current and future work
• Enable provision of formative feedback based on their use of educational systems
• Possible to support instructional interventions by embedding the developed analytics into the learning platform
• We are currently working on a MOOC experimentation platform that will include cognitive presence analytics
• At present, working on coding of MOOC discussion messages to evaluate the use of cognitive presence classifier in the MOOC context
• Also, exploring the use of the same classification methodology for similar problems
Student reflection in discussions (OU)
Essay grading (USF)
Twitter and YouTube comments (QUT)
28
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
The End
THANK YOU FOR YOUR TIME
Q/A
29
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
REFERENCES
Arbaugh, J.B. et al. (2008). “Developing a community of inquiry instrument: Testing a measure of the Community of Inquiry
framework using a multi-institutional sample”. In: The Internet and Higher Education 11.3–4, pp. 133–136.
Garrison, D. Randy, Terry Anderson, and Walter Archer (1999). “Critical Inquiry in a Text-Based Environment: Computer
Conferencing in Higher Education”. In: The Internet and Higher Education 2.2–3, pp. 87–105.
Kovanović, V., Gašević, D., Hatala, M., & Siemens, G. (in-press). A Novel Model of Cognitive Presence Assessment Using
Automated Learning Analytics Methods (Analytics4Learning). SRI Education. Available at:
http://vitomir.kovanovic.info/publications/
Kovanović, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., & Siemens, G. (2016). Towards automated content
analysis of discussion transcripts: A cognitive presence case. In Proceedings of the Sixth International Conference on Learning
Analytics & Knowledge (pp. 15–24). New York, NY, USA: ACM. https://doi.org/10.1145/2883851.2883950
Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of
learning technology use on cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74–
89. https://doi.org/10.1016/j.iheduc.2015.06.002
Kovanović, V., Joksimović, S., Gašević, D., & Hatala, M. (2014). Automated content analysis of online discussion transcripts. In
Proceedings of the Workshops at the LAK 2014 Conference co-located with 4th International Conference on Learning Analytics
and Knowledge (LAK 2014). Indianapolis, IN. Retrieved from http://ceur-ws.org/Vol-1137/