Upload
peter-brusilovsky
View
228
Download
4
Embed Size (px)
DESCRIPTION
An overview of PAWS Lab project related to competency modeling.
Citation preview
PAWS Lab Work onCompetencies and Student ModelingPeter Brusilovsky
School of Information Sciences
University of Pittsburgh, [email protected]
http://www.sis.pitt.edu/~peterb
Agenda
• Overview• ADAPT2 architecture• Original student modeling in CUMULATE
– Example, DB Exploratorium• Problems and solutions
– Multi-ontology issue – introduce ontology server
– Efficiency – pull to push switch• Cross-systems, cross-ontology, and cross-
domain modeling
University of Pittsburgh - PAWS Lab 3
Main Stages of Our Work
• Centralized user modeling (1990-1998)• Multi-system personalization based on ADAPT2 (2003-
2007)– CUMULATE 1: Single domain model (one system, one model)
(2003-2006) – CUMULATE 2: Parallel independent modeling using 2 models
(2004-2014)• Cross-domain mapping for cold start (2007)
– C to Java• Single domain guided evidence mapping (2008-2010)
– Topic to concept mapping for Java– Constraints to concepts mapping for SQL
• Single domain automatic mapping (2010-2012)
User Model
Collects informationabout individual user
Provides adaptation effect
AdaptiveSystem
User Modeling side
Adaptation side
Centralized Single System Modeling
Classic loop user modeling - adaptation in adaptive systems
University of Pittsburgh - PAWS Lab
KT Architecture
• Learning experiences are delivered by various [adaptive, smart] re-usable activities residing on distributed activity servers
• A portal provides single log-in and singe access point to all content
• A student modeling server maintains a centralized student model
• A value-added service could work as intermediary between “dumb” learning content and portal
• Brusilovsky, P. (2004) KnowledgeTree: A distributed architecture for adaptive e-learning. In: Proceedings of 13th International World Wide Web Conference, WWW 2004, New York, NY, 17-22 May, 2004, ACM Press, pp. 104-113
KT Architecture
Portal
ActivityServer
Student Modeling Server
Value-addedService
Making it Open
• There are no other requirements to the components than an ability to support standard protocols
• Any new activity server can be used as long as it complies to the protocols
• The architecture allows for different portals and value added services to co-exist as long as they support protocols
• Multiple student model servers allowed
Protocols
• Portal/service activity server/service– Request activities, respond with a list of
relevant activities, start activity• Portal/service/activity server student
model server– Report information about student, request
information about student• Student model server portal service activity server– Transparent chain of authentication
A student model server CUMULATE
Competencies-Based Modeling
• Lower level of student model has a flow of content-level events– Which content was used, who used, results (0-1)
• Each content item is connected to knowledge units– Topic-based modeling: coarse grain units, each content
“belongs” to topic (1->N), based on topic network– Concept-based modeling: fine grain units, each content is
indexed with related concepts, based on ontology• An inference agent processes events in the context of
KU connections and maintains up-to-date KU-Level model
• Cumulate allows multiple independent inference agents– Agents for different modeling approaches (i.e, BMA, BKT)– Agents that model content on different levels
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept N103
0
27
4
Concept-Level Knowledge Model
University of Pittsburgh - PAWS Lab
Example: Database Exploratorium• Knowledge Tree
portal for content access
• Three kinds of activities– Examples– Problems– SQL Lab
• Central user model serverCUMULATE
• Two levels of modeling– Topics (teacher)– Concepts (ontology)
• Both levels are used independently for adaptation
Brusilovsky, P., Sosnovsky, S., Lee, D., Yudelson, M., Zadorozhny, V., and Zhou, X. (2010) Learning SQL programming with interactive tools: from integration to personalization. ACM Transactions on Computing Education 9 (4), Article No. 19, pp. 1-15.
SQ
L O
nto
log
yWe created C, SQLand Java Ontologies
Two-level adaptation in DBE
Moving to many systems and ontologies
University of Pittsburgh - PAWS Lab
Problems with KT
• We started the integration of adaptive systems produced by other groups…
• Multiple ontologies (domain models)– Two systems complement each other, but use
different domain models for content indexing• Complex user modeling mechanisms
– User modeling server can’t replicate same level of inference student models from events
Cross-System Knowledge Modeling
http://adapt2.sis.pitt.edu/kt/
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyesno
no
noyes
yes
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyesno
no
noyes
yes
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyesno
no
noyes
yes
University of Pittsburgh - PAWS Lab
Missing links
The Approach: Ontology-Based Cross-System Personalization
University of Pittsburgh - PAWS Lab
Connect DM(ontologies)
UM of C knowledge
JavaC
UM ofJava
knowledge
How we started – from C to Java• Manual vs. Automatic
ontology mapping• Knowledge mapping using
ontology mapping• Compare predicted and
demonstrated knowledge• Automatic mapping is
comparable with manual• Overall gain for translated
knowledge is not high• We got concerned about
model to model mapping• Started exploring evidence
mapping
Sosnovsky, S., Dolog, P., Henze, N., Brusilovsky, P., and Nejdl, W. (2007) Translation of overlay models of student knowledge for relative domains based on domain ontology mapping. 13th International Conference on Artificial Intelligent in Education, AI-ED 2007, Marina Del Rey, CA, July 9-13, 2007, IOS, pp. 289-296
How we can deal with multiple competency organizations?• Content should be separated from its
content-metadata, i.e., ontology indexing or topic categorization
• The same smart content item could be classified under different topic networks or indexed using different ontologies
• We need to maintain and use multiple descriptions for the same item and multiple user models!
Solution: Ontology Server
• Ontology Server as a new component in the new ADAPT2
architecture• Ontology server maintains one specific domain ontology• Ontology Server collects metadata about everything
related to this ontology– Content-level metadata for all resources indexed with this
ontology– Overlay student models for all students that are modeled with
this ontology• A Student modeling server can use several ontology
servers in parallel to perform modeling in different ontologies
• Brusilovsky, P., Sosnovsky, S., and Yudelson, M. (2005) Ontology-based framework for user model interoperability in distributed learning environments. In: World Conference on E-Learning, E-Learn 2005, pp. 2851-2855.
Multiple Ontologies in ADAPT2
• The new architecture ADAPT2 allows the use of multiple ontologies for content and student modeling
• Each ontology is maintained by a dedicated ontology server
• Ontology server is handling all requests related with the ontology - about the ontology itself, learning activities, and users
Summary
• Learning activities are separated from its content metadata
• An activity server’s duty is to maintain and serve an activity (URI invocation)
• Each activity can be indexed in terms of several ontologies
• An ontology server (not activity server!) stores content metadata for all activities indexed in terms of this ontology
Ontology server
An ontology server support inference level of UM server
SEDONA: UM exchange with ontology servers
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyesno
no
noyes
yes
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyesno
no
noyes
yes
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyesno
no
noyes
yes
Ontology A
Ontology B
University of Pittsburgh - PAWS Lab
Practical Experience
• Implemented first version of an Ontology server Sedona
• Addressed more urgent student model efficiency issue
• Fully redesigned CUMULATE server, moved from pull to push, very efficient
• Ontology server as a unit has never been adapted to new CUMULATE, instead CUMULATE started to perform some of its functions
• Decided to collect more cross-ontology experience to redesign all Sedona functions properly
• Continued with a series of cross-ontology modeling experiments
SEDONA: UM Exchange
• Ontology server is an exchange point for concept-level overlay student models that are based on the stored ontology
• Each UM server or adaptive system that can deduce student knowledge in terms of this ontology reports it to the server
• Each adaptive system that need to know the level of student knowledge for concepts of this ontology can query the ontology server
University of Pittsburgh - PAWS Lab
Lightweight event-based centralized user modeling
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyesno
no
noyes
yes
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyesno
no
noyes
yes
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyesno
no
noyes
yes
Central UM
Concept 1
Concept 2
Concept 3
Concept 4
Concept 5
Concept Nyesno
no
noyes
yes
University of Pittsburgh - PAWS Lab
Sosnovsky, S., Brusilovsky, P., Yudelson, M., Mitrovic, A., Mathews, M., and Kumar, A. (2009) Semantic Integration of Adaptive Educational Systems. In: T. Kuflik, S. Berkovsky, F. Carmagnola, D. Heckmann and A. Krüger (eds.): Advances in Ubiquitous User Modelling. Lecture Notes in Computer Science, Vol. 5830, pp. 134-158.
• Student side:– Use systems in parallel (any order, any
combination)– No extra overhead (single sign-on,
single place to access)• System side:
– Integrated environment > (system1 + system2)
– Each system should try to increase the quality of user modeling and adaptation
What we Consider as True Integration
University of Pittsburgh - PAWS Lab
Explored Cases
• QuizJet integration with Problets in Java domain– One source KI to many target KI mapping– Brusilovsky, P., Sosnovsky, S., Yudelson, M., Kumar, A., and Hsiao,
I.-H. (2008) User Model Integration in a Distributed Adaptive E-Learning System. Workshop on User Model Integration at the 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems.
• SQL Exploratorium integration with SQL tutor in SQL domain– Many to many KI mapping from source to target domain– Sosnovsky, S., Mitrovic, A., Lee, D. H., Brusilovsky, P., and
Yudelson, M. (2008) Ontology-based integration of adaptive educational systems. 16th International Conference on Computers in Education (ICCE’2008), Taipei, Taiwan, October, 27-31, 2008, pp. 11-18
Java Problets: The Interface
Sample progra
m
Student’sanswer
Help
Questiontext
System’sfeedback
Java Problets: Domain Model• Problets implement traditional overlay user
modeling to adapt to student’s performance The domain
model of a problet is a concept map enhanced with learning objectives, that combine pedagogical and domain knowledge
QuizJET (1):System Description• QuizJet (Java Evaluation Toolkit) is a system for
authoring and delivery of online self-assessment quizzes for Java programming language
• A typical QuizJET problem is a sample program (consisting of one or several classes), that a student needs to evaluate and provide an answer a follow-up question
• QuizJET generates problems by substituting a numerical value in the program template with a randomized parameter
• Upon receiving a student’s answer QuizJET provides a feedback indicating the correctness of the answer and the right answer (if the student’s attempt was not successful)
QuizJET (2):Student Interface• Students can access QuizJET problems through
the KnowledgeTree portal
Topics in the course
Activities available for the current
topic
Problem text
Problem's classes
QuizJET’s feedback
QuizJET (3): Domain Model• Java Ontology
specifies about 500classes connectedwith 3 types of relations: subClassOf,partOf/hasPart, and related
• About 300 classes areavailable for indexing
• A class can play one of two roles in the problemindex: prerequisite or outcome
University of Pittsburgh - PAWS Lab
Domain Model Integration• Main problem: different modeling paradigms
– A learning objective models application of a concepts in the certain context
– Extra classes from the Java ontology have been used for context modeling
– Weights are assigned to prevent too aggressive propagation of classes responsible for context modeling
• Example:– This learning objective models a situation when the
conditional part of the if-else statement is a relational expression evaluated into true value
Evidence-based UM integration in CUMULATE
University of Pittsburgh - PAWS Lab
• An example of semantic integration of two working adaptive systems relaying on very different domain models
• Many to many KI mapping from source to target domain– Topology constructed by domain experts– Data could be used to improve the mapping
Integrating SQL Tutor and SQL Exploratorium
University of Pittsburgh - PAWS Lab
SQL-Exploratorium
University of Pittsburgh - PAWS Lab
SQL-Tutor
Goal: Integrated Environment
http://www.sis.pitt.edu/~paws/ont/SQL.owl
SQL Explorer: SQL Ontology
University of Pittsburgh - PAWS Lab
SQL-Tutor: Constraints
University of Pittsburgh - PAWS Lab
• Constraints and Concepts are too difficult to map them
• A typical constraint models syntactic or semantic relation between several concepts
• Manual connect constraint to concepts with somedegree (small-1, medium-2, or large-3)
Domain Model Mapping
University of Pittsburgh - PAWS Lab
• Solution to SQL-Tutor problem, triggers a number of constraints satisfied and or violated
• Mapping model calculates knowledge update for every concepts related to every triggered constrained:
• The updates are reported to SQL-Exploratorium’s user modeling server
Evidence-Based Modeling
University of Pittsburgh - PAWS Lab
Architecture
• University of Pittsburgh, 2 courses: undergraduate and graduate
• ½ of semester• 42 students tried SQL-KnoT, 18 –
SQL-Tutor• Out of 103 sessions of using SQL-
KnoT 66 co-located with SQL-Tutor usage
Evaluation
University of Pittsburgh - PAWS Lab
• Questionnaire (21 students)– I1 / I2: Overall, I like the interface of
SQL-KnoT/SQL-Tutor. – U1 / U2: SQL-KnoT/SQL-Tutor is a useful
learning tool.– C1 / C2: SQL-KnoT/SQL-Tutor problems
challenged me intellectually.
Results
49
Evaluating and improving mapping:SQL Exploratorium and SQL Tutor• Authoring constraint mapping is time
consuming• How we can evaluate weights?• How we can improve mapping?
University of Pittsburgh - PAWS Lab
SQL KnoT and SQL-Tutor (2)• 6 experts (2 teachers, 2 GSA, 2
practitioners)• 1012 constraint-concept relations: strong
(1/1), medium (2/3), weak (1/3)• Usage log of 3544 SQL-Tutor problem-
solving attempts of 38 users• Dataset specific subset
– 282 constraints, 576 relations, 61 concepts
University of Pittsburgh - PAWS Lab
51
Fitting The Source(Constraint) Model
• Experts only need to produce relations b/w KIs – the rest is automatic
University of Pittsburgh - PAWS Lab
University of Pittsburgh - PAWS Lab 52
References on cross-system modelingSosnovsky, S., Mitrovic, A., Lee, D. H., Brusilovsky, P., Yudelson, M., Brusilovsky, V., and
Sharma, D. (2008) Towards integration of adaptive educational systems: mapping domain models to ontologies. Proceedings of 6th International Workshop on Ontologies and Semantic Web for E-Learning (SWEL'2008) in conjunction with ITS'2008, Montreal, Canada, June 23, 2008.
Sosnovsky, S., Mitrovic, A., Lee, D. H., Brusilovsky, P., and Yudelson, M. (2008) Ontology-based integration of adaptive educational systems. Proceedings of 16th International Conference on Computers in Education (ICCE’2008), Taipei, Taiwan, October, 27-31, 2008, pp. 11-18.
Brusilovsky, P., Sosnovsky, S., Yudelson, M., Kumar, A., and Hsiao, I.-H. (2008) User Model Integration in a Distributed Adaptive E-Learning Systems. Proceedings of Workshop on User Model Integration at the 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems (AH'2008), Hannover, Germany, July 29, 2008.
Brusilovsky, P., Mitrovic, A., Sosnovsky, S., Mathews, M., Yudelson, M., Lee, D., and Zadorozhny, V. (2009) Database exploratorium: a semantically integrated adaptive educational system. In: Proceedings of Ubiquitous User Modeling Workshop at the 17th International Conference on User Modeling, Adaptation, and Personalization (UMAP 2009), Trento, Italy, June 22, 2009
Sosnovsky, S., Brusilovsky, P., Yudelson, M., Mitrovic, A., Mathews, M., and Kumar, A. (2009) Semantic Integration of Adaptive Educational Systems. In: T. Kuflik, S. Berkovsky, F. Carmagnola, D. Heckmann and A. Krüger (eds.): Advances in Ubiquitous User Modelling. Lecture Notes in Computer Science, Vol. 5830, pp. 134-158
53
Automatic Ontology Mapping
• SQL Integration demonstrated using expert-authored and automatically-tuned domain ontology mapping we can do efficient cross-system personalization with two conceptualizations (ontologies) in the same domain
• Expert labor is expensive. Could we do automatic mapping between two ontologies in the same domain?
• The case is explored in– Sosnovsky, S., Brusilovsky, P., and Hsiao, I.-H. (2012) Adaptation
"in the Wild": Ontology-based Personalization of Open-Corpus Learning Material. In: Proceedings of 7th European Conference on Technology Enhanced Learning (EC-TEL 2012), Saarbrücken, Germany, pp. 425-431.
– Sosnovsky, S. (2011). Ontology-based Open-Corpus Personalization for E-Learning PhD Thesis, University of Pittsburgh.
9/26/2010
54
What Happened with auto-mapping?
University of Pittsburgh - PAWS Lab Sergey Sosnovsky PhD Thesis
55
OOPS Interface: Reading Phase
content of the chosen
topic
Navigation links to the next and
the previous topics
Feedback/exit
buttons
University of Pittsburgh - PAWS Lab Sergey Sosnovsky PhD Thesis