A hybrid user-centred recommendation strategy applied to repositories of learning objects

302 Int. J. Web Based Communities, Vol. 8, No. 3, 2012

Copyright © 2012 Inderscience Enterprises Ltd.

A hybrid user-centred recommendation strategy applied to repositories of learning objects

Almudena Ruiz-Iniesta*, Guillermo Jiménez-Díaz and Mercedes Gómez-Albarrán Facultad de Informática, Universidad Complutense de Madrid, c/ Prof. José García Santesmases s/n, 28040 Madrid, Spain E-mail: [email protected] E-mail: [email protected] E-mail: [email protected] *Corresponding author

Abstract: This article describes the guidelines followed in the design of a framework for managing learning object repositories that can be applied to different domains. The main features of the framework are the engagement of the virtual learning community in authoring and maintenance tasks, along with the use of recommender system technology in order to provide personalised searching and retrieval features. This article mainly focuses on the recommendation tasks, which help to identify suitable resources for the students in the virtual learning community. The recommendation approach follows a cascade hybrid strategy that refines the decisions of a case-based recommender by using a collaborative one. The former provides resources that fit the current student profile and promote her learning process. The later includes in the retrieval process the opinion about the usefulness of the resources provided by other members of the virtual learning community.

Keywords: virtual learning communities; VLC; learning object; hybrid recommender; personalisation.

Reference to this paper should be made as follows: Ruiz-Iniesta, A., Jiménez-Díaz, G. and Gómez-Albarrán, M. (2012) ‘A hybrid user-centred recommendation strategy applied to repositories of learning objects’, Int. J. Web Based Communities, Vol. 8, No. 3, pp.302–321.

Biographical notes: Almudena Ruiz-Iniesta is a PhD student in the Department of Software Engineering and Artificial Intelligence at the Complutense University of Madrid (UCM), Spain. She is a member of GAIA, the Group of Artificial Intelligence Applications at the UCM. Her current research includes the analysis and use of the recommendation techniques in e-learning.

Guillermo Jiménez-Díaz is a Lecturer at the Department of Software Engineering and Artificial Intelligence at the Complutense University of Madrid (UCM), Spain. He is a member of GAIA, the Group of Artificial Intelligence Applications at the UCM. He received his PhD in Computer Science from the UCM for his work about the development of role-play virtual environments for learning object-oriented programming. His current research

A hybrid user-centred recommendation strategy applied to repositories 303

interests include the analysis and use of recommendation techniques in different domains, such as e-learning, recommendation to groups and matchmaking in videogames.

Mercedes Gómez-Albarrán is an Associate Professor in Software Engineering and Artificial Intelligence Department at the Complutense University of Madrid, Spain. She has developed her whole career at this university, where she took a degree in Physics in 1993 and her PhD (First Class Hons.) in Computer Science in 2000. Her main research interests are personalised recommendation in e-learning, virtual learning environments, and innovative techniques and tools for programming teaching. She is an author of over 40 technical publications in journals and proceedings of international conferences, as well as posters and oral talks.

1 Introduction

In the last few years, and in most educational disciplines, there is a trend to develop educational resources that are available to students in electronic repositories (Tzikopoulos et al., 2007). Traditionally, educational resources were designed by the instructors. However, the growing interest and the development of the virtual learning communities (VLC) have introduced students in the design and evaluation of the educational resources.

The availability of educational resources in these repositories eases and motivates student self-learning as a complementary activity to lectures in the classroom. However, the high number of learning objects (LOs) that exist in these repositories makes the access difficult to those adapted to the individual knowledge, goals and/or preferences of the students. It is necessary to provide support for personalised searching functionalities, which retrieve resources that fit the needs, goals and preferences of the students.

Recommender systems emerged as an independent research area in the mid-‘90s and have been traditionally applied in the field of e-commerce (Wei et al., 2007). There are two main classes of recommender systems based on the kind of knowledge source they employ (Brusilovsky et al., 2007): case-based (or content-based) recommenders and collaborative (or social) recommenders. Case-based recommender systems employ item descriptions in order to make a recommendation. Collaborative recommender systems exploit user preferences, usually in the form of ratings-based profiles, and apply collaborative filtering (CF) techniques. In order to improve the performance of the different techniques in recommender systems the hybrid recommenders appear. Hybrid recommender systems combine, in different ways, diverse recommendation techniques together to achieve some synergy between them (for example, case-based and collaborative techniques).

The use of the recommender systems has been recently transferred to the academic field. The works that we can found in the literature follow different approaches and they apply to different e-learning contexts. Some research has been conducted into developing recommender tools for courses and curriculum learning activities (Farzan and Brusilovsky, 2006; O’Mahony and Smyth, 2007; Castellano and Martínez, 2009). The work in Santos and Boticario (2008a, 2008b) focuses on how to provide personalised and inclusive support in a standard-based learning management system (LMS). In this work,

304 A. Ruiz-Iniesta et al.

authors first identify when and what recommendations are to be provided to the learners to improve their performance by addressing critical e-learning factors. Later, these recommendations will be included during the course execution. The work in Kardan et al. (2009) uses a collaborative tagging system in an e-learning environment and the tag collections of every user are employed by the recommender system as the user’s interest in a specific topic. The knowledge of each learner is illustrated with a concept map. Other works (Zaíane, 2002) take into account profiles of online learners, their access history and the collective navigation patterns, and use simple data mining techniques in order to generate the recommendations of learning activities. They suggest possible actions or web resources based on the understanding of the user’s access. Also, they use association rules to build a model representing the web page access behaviour or associations between on-line learning activities.

The majority of current web-based learning systems are binded to a LMS that contains all the material that can be provided to the student and the only dynamic aspect is the organisation of the material. An evolving web-based learning system which can adapt itself not only to its users, but it also adapt its contents in response to the usage of its learning materials, is described in Tang and McCalla (2003). The novelty of this system lies in its ability to find relevant content on the web and to personalise and adapt this content based on the system’s observation of its learners and their accumulated ratings. The major techniques adopted in this system are collaborative filtering and data clustering. It has been employed in a tutorial system to support an advanced course on data mining and web mining.

In this article we describe a framework for managing LO repositories (LOR) that can be applied to different domains. A first conception of the approach was described in Gómez-Albarrán and Jiménez-Díaz (2009). The originality of our proposal lies in the services offered to the VLC:

• searching and retrieving functionality based on personalised LO recommendation

• engagement of the VLC in authoring and maintenance tasks.

This article mainly focuses in the recommendation tasks. The recommendation approach follows a cascade hybrid strategy that refines the decisions of a case-based recommender using a collaborative one.

We extend the case-based recommender in Gómez-Albarrán and Jiménez-Díaz (2009) by exploring a model of high level of personalisation: this case-based recommendation strategy fits the student’s long-term learning goals without significantly compromising the in-session interests of the target student. The case-based recommendation model gives priority to those LOs in the repository that are most similar to the student’s short-term learning goals (the concepts that the student wants to learn in the session) and, at the same time, have a high pedagogical utility in sight of the student’s cognitive state (long-term learning goals). This model is supported by the definition of a flexible metric that combines the similarity with the query and the pedagogical utility of the LO.

The collaborative recommender predicts the utility that a LO will have for a concrete student, based on the ratings (i.e., relevance, preferences, and opinions) that similar students in the VLC (students with similar goals and knowledge level) provided about this LO. CF techniques help us to refine the LOs suggested by the case-based recommender and select the final subset of LOs that will be recommended. This way, the


opinion of other students in the VLC about the utility of the LOs is integrated in the recommendation process. The high utility of the student ratings in the recommendation process will help to encourage social dynamics and student participation within the community.

A sketched description of the global management setting appears in Section 2. Section 3 provides a high-level description of the different knowledge sources used by the recommendation strategy, independently of the educational domain. This section also extends the description of the authoring and maintenance tasks. Section 4 describes in detail the cascade recommendation strategy. In Section 5, we summarise the goal-oriented evaluation that we have designed in order to analyze different aspects of the recommendation and the authoring and maintenance processes. Last section concludes the article and outlines our future work.

2 Managing a repository of learning objects

The management of a LOR is not an easy task. Even in repositories deployed for small learning communities and maintained by a small number of instructors, some problems appear. First, the usefulness of the repository is based on the amount and quality of the resources that it stores. On the one hand, delegating the responsibility of creating all the content to a small group of people can be an overwhelming task. On the other hand, a resource created by one person could have low interest for the rest of the VLC even when the creator considered it useful. A possible way to alleviate this problem is delegating part of the authoring and maintenance tasks to the VLC.

Additionally, the number of educational resources contained in a repository usually swamps its users when browsing it. This overload becomes harder on the students, who usually do not know where to begin their learning task. LORs have to provide help to identify suitable resources for their users. First, they have to provide searching features for finding resources. Additionally, the search should be adapted to the user. In particular, it should provide resources that fit the current student profile and promote her learning process. Finally, it is necessary to help the user to choose the most interesting resources among the retrieved ones. The interest of a resource not only depends on the target student, but can also be influenced by the VLC. For instance, the opinion about the usefulness of the resources provided by other members of the VLC could be taken into account during the retrieval process.

The incorporation of the VLC in the management process of an educational repository has two main advantages. From the pedagogical point of view, the engagement of the students in tasks related to the creation of new resources and to the evaluation of the existing ones promotes the feeling in the students that they are taking active part of their learning process. Regarding to the accessibility to the resources, the information provided by the VLC members is a valuable resource for learning about the preferences from the community in order to generate recommendations of resources adapted to the students.

According to these ideas, we propose a framework for managing the LOs contained in web-based repositories. The framework deals with the authoring, recommendation and maintenance of LOs, promoting the collaboration of the VLC members (instructors and students) in these tasks. A general overview of the framework is sketched in Figure 1.


The framework provides support for several managing tasks to the learning community using two interfaces:

• The authoring and maintenance interface supports the creation and authorship recognition, the revision and the evaluation of new LOs. These activities are recognised as strategies that encourage social dynamics and participation around the repository (Monge et al., 2008). The community members add and catalogue the new resources according to the learning concepts and goals that they cover. Furthermore, this interface provides functionality to review resources recently created for other community members. Finally, the community opinion about existing resources can be reflected in terms of rating scores and comments.

• The recommender interface supports the retrieval of LOs. The retrieval process gathers the short-term needs of the user in terms of an explicit query, her long-term goals according to her profile, and the usefulness assessed by the VLC. The retrieval process is performed following a cascade hybrid recommendation strategy: a case-based recommender acts as the primary recommender and its decisions are refined by a collaborative one.

Next sections deal with the internals of the proposed framework. First, we detail the needs in terms of knowledge imposed by the recommendation interface and how they are managed using the authoring and maintenance interface (Section 3). Later, we describe the recommender interface and the process that integrates the short-term goals of the student, the adaptation to her profile and the aggregation of the community preferences in order to retrieve LOs within the repository (Section 4).

Figure 1 Sketching the approach


3 Knowledge sources: contents, authoring and maintenance

In this section we detail the knowledge sources in our recommendation approach: the LOs and their metadata (Subsection 3.1), the domain ontology (Subsection 3.2) and the student profile (Subsection 3.3). A brief description of the authoring and maintenance tasks appears in Subsection 3.4.

3.1 The learning objects

We understand a LO as a resource (usually digital) that can be used and reused as a support to teaching and learning and that is also a learning experience that contains objectives and a learning activity.

Recommendation needs information about the features that characterise a LO in the LOR. These features can be attached to a learning resource in different ways. We have decided to use metadata, so our LOs have been developed according to learning object metadata (LOM) (IEEE, 2002), a data model used to describe LOs and similar digital resources that support learning. LOM lets tag the LOs according to a set of predefined categories and assign values to each one. We propose to use the next upper-level LOM categories: life cycle, technical, educational, relation and general.

The life cycle category identifies the author and the status of the LO. The educational category helps us to identify the type of educational resource to which the LO corresponds – for instance, lecture notes, explained examples, quiz questions, tests (a set of quiz questions), assignments, etc. The technical category groups the technical requirements and characteristics of the LO. Additionally, the relation category let us identify if a LO is a version of another one.

The general category plays an important role in the retrieval functionality. This category contains keywords that describe the learning concepts covered by the LO. This category will be especially important in the recommendation process because it will be employed to compute the similarity between the LOs and the learning goals of the student. Additionally, these keywords are used to index the LO in the domain ontology used in the LOR, which is described next.

3.2 The domain ontology

In artificial intelligence, particularly in knowledge engineering, the word ontology is used to denote a reusable knowledge base, that is, an explicit representation, expressed in a formal language, of knowledge about a domain: “Ontology is a hierarchically structured set of terms for describing a domain that can be used as a skeletal foundation for a knowledge base” (Swartout et al., 1997).

In knowledge-based systems what exists is exactly what can be represented and formalised. An ontology, in this sense, refers to an attempt to formulate a conceptual framework within a given domain, so it can be reused by other system that works on the same domain. One widely accepted definition of an ontology refers to the concept of conceptualisation: “a formal, explicit specification of a shared conceptualization” (Gruber, 1993).

Due to the general scheme provided by ontologies, which allows us to include knowledge about similarity between the concepts representing the domain topics, we suggest to use an ontology for indexing the LOs within the repository. The knowledge


about the similarity between concepts and the relations established among them are crucial in our recommendation strategy. Besides, an ontology gathers a common parlance that can be used by the various members in the VLC when including new LOs and querying the recommender. Other authors make a successful use of ontologies for annotating LOs (Gasevic et al., 2007).

Our ontology is populated with concepts in the field of study (for instance, mathematics, law or computer programming). Concepts are organised in a taxonomy using the typical relation is_a. The ontology also establishes a precedence property among the concepts. We use this precedence to reflect a traditional sequence or order of concepts used when teaching in the corresponding field. As we will see, this precedence helps to establish the learning paths that are used in the retrieval stage of the case-based recommender. These learning paths help to filter out LOs that exemplify non-reachable concepts given a concrete cognitive state of the student. In our ontology, this precedence relationship is represented by the has_previous property and its symmetric property, has_next. Figure 2 is an example of a generic ontology. In this figure, we can see the hierarchy of concepts and the precedence property.

Figure 2 An example of an ontology with the precedence property between concepts, and some LOs together with the concepts covered by each one (see online version for colours)

Note: Some relationships are omitted for simplification

It is necessary to have a representation of each LOs in the ontology and link this representation with the concepts covered by the LO. This representation provides us knowledge that will be used later by the recommendation strategies in order to determine the suitability of each LO. The property covers represents the concepts covered by the LO. Figure 2 shows this property.


3.3 Student profile

The task of user modelling is a large research field which seeks to identify behaviour patterns and knowledge of users in different domains in order to provide personalised and adapted services. In recommender systems, as in other systems, the personalisation capability is related to the ability to store profiles for each user and the information contained in them.

In our work, the student profile stores information about the goals achieved in the learning process and the explored LOs representing her navigation history.

The learning goals achieved by a student are represented through the concepts that the student should know and the mastery level achieved in each of them. When a concept is explored by the student, it is assigned the competence level attained in it. This level is considered as a degree of satisfaction, a metric that allows the recommender to know about the student knowledge level on a particular concept. As we will see, the competence level will be an important element in the retrieval stage of the case-based recommender, because it allows us to know what concepts are most appropriate to the student.

The competence level attained by a student is extracted from the interaction of the student with some LOs contained in the LOR. Some types of LOs, like quiz questions, tests and assignments, allow to assess the current student knowledge. The results obtained by the students when working with these LOs will be assigned to the concepts covered by them. These concepts commonly belong to the lower level of the ontology. The hierarchical structure of the ontology – using the is_a relationship – allows to spread out the competence level to the upper levels by using a propagation function, such as inferring the competence level of a concept c as the arithmetic mean of its subconcepts.

3.4 Authoring and maintenance

The proposed framework provides an authoring and maintenance interface that lets the dynamic growing of the LOR and the collaboration of the VLC in authoring and maintenance tasks. This way, the content stored in the repository is enhanced with the student perspective and it is not limited to the point of view of the instructors. Moreover, the maintenance tasks are not exclusively assigned to the instructors, but they are distributed along the whole community. The participation of the students in these tasks has an additional benefit: student motivation also increases because they collaborate in their own learning process and in that of their colleagues. Next, we describe diverse elements that appear in Figure 1.

The authoring and maintenance interface provides a LO acquisition component for the inclusion of new LOs. This component is used to create a new LO, its metadata and to provide a tentative set of the domain concepts that the LO covers, extracted from the ontology. The new LOs are stored in a temporary LO repository, waiting for the instructor review and acceptance (maintenance component). Once the instructors permanently move the LOs to the repository, they can be accessed.

A peer review technique allows students to indirectly work in the maintenance process, examining and judging the quality of the LOs provided by other VLC members. Using the peer review component, students and instructors can browse the temporary LO repository, review the LOs stored provisionally there, correct mistakes in the metadata and add comments to improve the usefulness of the new resources. The information


provided by the community will be taken into account by the instructors during the real maintenance process. The community can also rate the attempting LOs, alleviating the classical first-rater problem in collaborative recommender systems.

In Figure 3, we show a snapshot of a preliminary version of the authoring tool developed for a programming course, referred in Section 6. On the left side, we show the interface to include a new LO in the temporary LO repository, the acquisition component. For each LO, the author has to specify the resource – the pedagogical content included in the LO – its title and a short description. The author should also choose the concepts that the LO covers selecting them from ontology. These concepts will be represented by the keywords contained in the LO. The author will select the type of the LO from a list of resource types. Finally, the author can select another LO from the repository which is related to the current LO. The author field is fixed by the acquisition component.

Figure 3 Preliminary version of the interface of the authoring tool in a prototype for recommending programming resources (see online version for colours)

Note: On the left the interface for the acquisition component and on the right the interface for the peer review component.

On the right side of Figure 3, we can see the interface that lets review the LOs in the temporary repository, the peer review component. The user can edit any field of the LO, and can also include comments about the revision. In addition, a user can view the list of all the changes that have been made in the LO. Finally, the peer review component


provides the ability to add a rating and/or a comment to the current LO about its usefulness.

The preference acquisition component is crucial in the inclusion of the VLC opinion and knowledge about the LOs already contained in the repository. This component lets the community provides ratings to LOs ready to be recommended. The rating assigned to each LO represents the utility that the recommended LO has for the student.

Ratings may be gathered through explicit means, implicit means or both. Explicit ratings are those where the user is asked to provide an opinion on an item. Implicit ratings are those inferred from user actions. According to Brusilovsky et al. (2005), we defend an explicit collection of student feedback, because the information obtained is more accurate than the one obtained by implicit approaches. The preference repository stores the rating scores explicitly assigned by the students, together with their profile when they rated them. The collected preferences are used by the recommender interface and provide a reliable form of collaboratively refining the relevance computed for each LO stored in the repository, as we will describe in Section 4.2.

Finally, our framework suggests a manual maintenance policy where instructors decide off-line which LOs they definitely incorporate into the LO repository. This reactive maintenance policy is periodically performed using the maintenance component, aiming basically at LO retention. Additionally, competence-preserving approaches for LO removal can profit from the rating scores assigned to the LOs in the repository. Low rating scores let identify redundant or uninteresting LOs, which the instructors could freely remove from the LOR.

4 Recommending learning objects

The learning field imposes specific requirements on the recommendation process (Drachsler et al., 2008). For instance, recommenders would benefit from taking into account the cognitive state of the student, which changes over time. Successful learning paths and strategies could also provide guiding principles for recommendation. For instance, recommendation could benefit from simple pedagogical rules like ‘go from simple to complex tasks’ or ‘gradually decrease the amount of guidance’. Learning paths could represent routes and sequences designed by the instructors and successfully tried in the classroom, or they could correspond with successful study behaviour of advanced learners. Our recommendation strategy takes into account these requirements.

The recommender interface is the access point for the VLC to the LOR. This interface follows a recommendation process adapted to our special requirements in order to retrieve LOs in response to a query. This query is expressed in terms of the concepts that the student wants to practice during a learning session. Once the query is posed, the recommender interface generates a set of LOs, considering the following relevance factors:

• The relevance due to the short-term learning goals satisfied by the LO. The short-term learning goals refer to the concepts contained in the query. The higher the number of concepts in the query that the LO covers, the higher the relevance value is. As we will detail, this relevance is computed using similarity measures between concepts.


• The relevance due to how the LO fits the long-term learning goals of the student. The long-term learning goals refer to the learning path and the knowledge acquisition process. The LOs are evaluated according to the student current knowledge, in order to give more relevance to the ones that cover weaknesses in the student knowledge or the LOs that promote the progress in the learning path discovering new learning concepts.

• The relevance due to the preferences assigned to the LO by other VLC members. CF techniques predict the rating that the current user should provide to a LO, so the most promising LOs have a higher relevance value. These techniques generate predictions using a neighbourhood of users with similar rating profiles. In our domain, the prediction is computed based on the ratings provided by the students that, when they ranked the LO, had a profile similar to the profile of the current student.

The recommendation is performed following a cascade-hybrid process: the recommendations from a first case-based recommender are refined using a collaborative recommender.

First, the case-based recommendation adapts its results to the student by exploring a strong personalisation model. This recommendation strategy runs in two stages: retrieval and ranking. The retrieval stage looks for LOs that satisfy, in an approximate way, the student short-term learning goals represented in the query (in-session learning goals). These LOs should be ‘ready to be explored’ by the student according to her current knowledge and the defined learning paths. Once LOs are retrieved, the ranking stage sorts them according to the quality assigned to each LO. The quality is computed so that priority is given to those LOs that are most similar to the query and, at the same time, have a high pedagogical utility in the light of the student’s cognitive state (long-term learning goals). Both stages are described in Section 4.1.

Next, the collaborative recommendation adds the community relevance component to the LOs returned by the previous recommender. This recommender predicts the relevance of each LO for the target student taking into account the LO ratings and the similarity between the profile of the target student and the profiles that the students had when they rated the LO. The idea behind this process is that a LO will have a high interest for a student if it was highly rated by other students within the VLC with similar profiles to the former. The similarity is computed in terms of the competence levels extracted from the student profile, so two students are more similar as long as they have similar competence levels in the learning concepts. Section 4.2 details the collaborative filtering process.

4.1 Case-based recommendation strategy

The case-based recommendation strategy presented here follows a reactive approach, that is, the student provides an explicit query and the recommender system reacts with a recommendation response. Once the student poses her query, the recommendation response is obtained in a two-step process – retrieval and ranking – which are described in the following subsections.

4.1.1 The retrieval stage

The retrieval stage looks for an initial set of LOs that satisfy, in an approximate way, the student query. The retrieval process tries first to find the LOs indexed by the query


concepts. If there are no LOs that satisfy this condition, or if we are interested in a more flexible location, LOs indexed by a subset of the (same or similar) concepts specified by the student are retrieved.

This initial set of LOs is filtered and only those LOs that cover ontology concepts ‘ready to be explored’ by the target student are finally considered in the ranking stage. We say that an ontology concept is ‘ready to be explored’ by a given student if, according to her current profile and the learning paths defined in the ontology, it fulfils any of the following conditions:

• It is a concept already explored by the student, so that it appears in her profile with its corresponding competence level.

• It is a concept that the student has not explored yet but she can discover it: if a concept c1 precedes a concept c2 in the ontology, a student can discover c2 if the student competence level for c1 exceeds a given progress threshold. If several concepts c1, c2,…,ck directly precede a concept cx, cx could be discovered if the student competence level in all the directly preceding concepts exceeds the given ‘progress threshold’.

In short, the goal of the filtering process is to discard LOs indexed by concepts non-reachable by the target student.

This filtering step has been incorporated into the retrieval stage of the case-based recommender presented in Gómez-Albarrán and Jiménez-Díaz (2009) and gives way to a light long-term personalisation in this first phase of this recommendation strategy. This way, when two students pose the same query within a session but their subject masteries differ, the set of retrieved LOs could be different.

4.1.2 The ranking stage

Recently, there is a growing interest to look for alternative ways to judge the utility of an item in a given context (McSherry, 2002). In this work, we replace the pure-similarity based approach used in Gómez-Albarrán and Jiménez-Díaz (2009) with a quality-based approach that fosters a strong personalised recommendation. So, once the LOs are retrieved, the ranking phase sorts them according to the quality assigned to each LO. The quality is computed so that priority is given to those LOs that are most similar to the student query and, at the same time, have a high pedagogical utility in the light of the student profile.

In order to compute the quality of a given LO L for a student S that has posed a query Q we propose to use a quality metric defined as the weighted sum up of two relevancies: the similarity (Sim) between Q and the concepts that L covers, and the pedagogical utility (PU) of L with respect to the student S:

( , , ) ( , ) (1– ) ( , ) where [0, 1]Quality L S Q Sim L Q PU L Sα α α= + ∈ (1)

In order to compute the two partial relevancies, Sim and PU, different approaches and metrics can be tried. In particular, the similarity Sim(L, Q) between the concepts gathered in the query Q and the concepts that L covers can be calculated by using any existing similarity metric for comparing sets of concepts in an ontology. For instance, in González-Calero et al. (1999) and in Recio-García et al. (2006) we can found some of these metrics.


In order to measure the pedagogical utility PU(L, S) that the LO L shows for a given student S, we consider interesting adopting instructional strategies that promote filling the student knowledge gaps by including remedial knowledge (Siemer and Angelides, 1998). For instance, remediation can help to treat either a misconception or a missing conception. In the first case, we can use a metric that assign a high pedagogical utility to L if it covers concepts in which the student has shown a low competence level. In the second case, we can assign a high pedagogical utility to L if it covers concepts that the student has not explored yet.

We can found a detailed example of use of concrete Sim and PU metrics for (1) in Ruiz-Iniesta et al. (2009).

The resulting quality metric defined in (1) introduces a considerable degree of personalisation in the ranking stage. The quality of a LO finally proposed to the student partly depends on the pedagogical utility that the LO has for attaining the student long-term goals (for instance, reminding concepts or learning new concepts). The final influence of the pedagogical utility and, as a consequence, the level of long-term personalisation achieved in the definitive list of LOs suggested by the case-based recommender, could be controlled by means of the value assigned to the weight α used in (1). Low values of α give priority to the pedagogical utility against the similarity to the query. In particular, α = 0 represents the highest level of long-term personalisation, and, in this case, the query (in-session goals) is used only in the retrieval stage. This ensures that the recommender system proposes LOs that meet the in-session goals at a minimum level, although the order in which they are proposed to the student is totally influenced by the long-term goals they let achieve.

Once the value of α used in (1) is fixed, the resulting recommender system exhibits the same behaviour, with respect to the type of personalisation it provides, for all the users. We can obtain a more flexible behaviour if, in a given recommender, α could take different values. This way, the recommender exhibits a higher adaptability to the potential users. For instance, the value of α could depend on the kind of student that uses the recommender.

Let’s consider, for instance, a pedagogical utility metric that assigns high utility values to L if it covers concepts in which the student has shown a low competence level. In this case, high values of α in (1) could be appropriate for good students, those students whose profiles exhibit good performance. These students seldom need knowledge reinforcement training and the recommender could focus on their in-session learning goals giving priority to those LOs that are highly correlated with the query. Low values of α on the contrary, could be appropriate for students with lower performances, such that the recommender fosters filling misconceptions without significantly compromising the in-session interest that the recommended LOs can have for these students.

So, the ranking approach presented in (1) offers a framework which, once it is instantiated, results in diverse recommendation approaches.

4.2 Collaborative recommendation strategy

The main goal of the collaborative recommendation strategy is to select the LOs provided by the case-based recommender with higher interest to the target student according to the opinions of other VLC members. CF is the process of filtering or evaluating items using the opinions of other people. In a VLC, CF techniques help to identify those LOs which are more useful for learners. Instructors and students create new LOs but the whole


learning community is responsible for measuring the real utility of each LO, by means of their ratings.

We have explored the use of the well-known user-based nearest neighbour algorithm (Brusilovsky et al., 2007; Sarwar et al., 2001) in our collaborative recommender. This algorithm generates predictions for users based on ratings from similar users. First, the algorithm identifies the users that, in the past, exhibited a similar behaviour. Next, it analyzes their ratings to identify the items that the target user should like. These similar users are the neighbours.

We have adapted the general CF process to our domain (recommendation of LOs). The collaborative recommender uses as input the LOs provided by the case-based one. Overall, the adapted process runs as follows:

• Neighbourhood formation. We form the neighbourhood in order to find those students who are more similar to the target student by using the competence level in the profile. This step requires a proximity measure to generate like-minded peers and to select the top N neighbours.

• Rating prediction and top-k selection. For each LO proposed by the case-based recommender, we generate a prediction using neighbours’ ratings. Finally the top-k LOs are selected.

Next subsections describe each step of the process.

4.2.1 Neighbourhood formation

The first step in the CF process is the selection of the students who are more similar to the target student to form the neighbourhood. Only the students who rated the LOs proposed by the case-based recommendation strategy are candidates to form part of the neighbourhood. It is worth noting that each rating of a LO in the preference repository is accompanied by the profile that the student had when she rated it. We need this information due to the evolving nature of the student profile on time. This profile is the one employed to measure how similar the corresponding potential neighbour is with respect to the target student.

As we described in Section 3.3, the student profile includes the concepts along with the competence level attained in each one. It is necessary to represent this information in order to compare the student profiles. Hence, we propose to employ the approach described in Ziegler et al. (2004), which represents users by vectors of interest scores assigned to topics taken from a taxonomy. In our case, the topics are the concepts in the ontology and the interest score are the competence level attained in each concept.

This approach generates a flat profile vector for each student. This way we can use the same technique employed in CF to measure the proximity of each user in order to form the neighbourhood. The most common approaches to measure the user similarity in CF are those employed in information retrieval, such as Pearson correlation and cosine similarity (Sarwar et al., 2001). We have chosen the cosine similarity, which measures the similarity by computing the cosine of the angle formed by the two vectors.

This cosine similarity measure reports a value in [0, 1] that represents how similar are two students according to their profiles. Then, the top-N students with higher similarity values with the target student are selected as neighbours. Later, we use the ratings of these users to perform our recommendation.


4.2.2 Prediction computation and top-k selection

Finally, we have to generate predictions for the LOs proposed by the case-based recommender. We analyse the ratings for each LO from students in the target student neighbourhood and, next, we compute the weighted average of ratings. The value of each weight is directly related with the similarity of the target student and the corresponding neighbour, computed during the neighbourhood formation.

Once we have calculated the prediction for the LOs proposed by the case-based recommender, we sort them according to the prediction and we select those with a higher score. This way, the most interesting LOs for the target student (the top-k LOs) will be finally proposed.

5 Designing an evaluation of the LOR management framework from the perspective of the students

Nowadays, we are designing an evaluation to determine the impact that the use of the LOR management framework has on the students. This evaluation will let us know the weaknesses of the framework and improve it developing alternatives to the strategies and interfaces described here.

In Worthen et al. (2003), authors identify the following generic evaluation methods: Goal-oriented evaluation, management-oriented evaluation, consumer-oriented evaluation, expert evaluation, participatory evaluation and dialectic evaluation. The selection of a particular evaluation method is based on the analysis of various factors: the goals of the evaluation (that is, for which the evaluation is conducted), the object of evaluation (that is, which will be evaluated) and the participants in the evaluation.

The main goal of our evaluation is to determine the impact in the students’ learning, satisfaction and engagement that the use of the described LOR management framework has. The object of the evaluation is the LOR management framework and the participants will be the students in the VLC.

We have decided to employ a goal-oriented evaluation based on the goal-question-metric (GQM) method (Basili and Rombach, 1988). This method points out that in order to improve a process we have to define measurement goals, which will be refined into questions, and, consecutively, into metrics which will supply all the necessary information for answering those questions. The analysis and interpretation of the answers let us know if the goals were attained (Solingen and Berghout, 1999). GQM method has been selected due to our previous and successful experience for the evaluation of adaptive courses in a virtual campus.

Our main evaluation goal is broken down into four sub-goals:

1 analysing the usability of the repository from the students’ perspective

2 analysing students’ grades

3 analysing the recommendation interface from the students’ point of view

4 analysing the authoring and maintenance interface from the students’ point of view.

The impact of the LOR will be measured comparing the data obtained with the metrics defined for each sub-goal in different learning scenarios. In each learning scenario, the LOR will show different configurations. This way, we will be able to compare the use of


the repository with and without the recommendation interface or using different stages during the recommendation process.

From our point of view, sub-goal 1 can be evaluated by means of a set of attributes like ‘easy to use’, ‘efficiency’, or ‘subjective satisfaction’, among others. For each one of these attributes, questions and metrics are developed in order to gather the data that will be processed later. Table 1 shows some questions and metrics for the attribute ‘efficiency’. Table 1 Some questions and metrics for the efficiency attribute within the usability sub-goal

Questions Metrics

How often is the LOR used? Number of logins for each student Time of use for each student

How do the students use the resources within the repository?

Data history about the repository visits

Table 2 Some questions and metrics for the subjective satisfaction attribute within the recommendation interface analysis sub-goal

Questions Metrics

Do you consider the recommender interface easy to use?

Questionnaire: It is easy to formulate queries to the recommender Questionnaire: More information about the terms available to pose the query is required Answer (for both): Liker scale (Totally agree – Agree – Neither agree nor disagree – Disagree – Totally disagree)

What do you think about the recommended LOs?

Questionnaire: The recommender interface provides me with useful resources Answer: Likert scale (Totally agree – Agree – Neither agree nor disagree – Disagree – Totally disagree) Questionnaire: I trust in the recommendations provided by the recommender interface Answer: Likert scale (Always – Frequently – Sometimes – Rarely – Never)

After using the LOR, do students solve problems more efficiently and quickly?

Questionnaire: “Now I am able to solve problems in less time than before using the LOR”. Answer: Likert scale (Always – Frequently – Sometimes – Rarely – Never) Questionnaire: “I have less errors in my solutions”. Answer: Likert scale (Totally agree – Agree – Neither agree nor disagree – Disagree – Totally disagree)

Table 3 Some questions and metrics for the accuracy attribute within the recommendation interface analysis sub-goal

Questions Metrics

How accurate are the recommendations?

Number of times the student selects a LO in the top-3 Precision metrics Recall metrics

The analysis of students grades (sub-goal 2) will be made from the results of the tests passed by the students. In order to make a right analysis of these grades, we need to know


the student grades before they use the LOR. Thus, we may know if the students who use the LOR improve their grades.

Sub-goal 3 can be evaluated by means of a set of attributes like ‘usability’, ‘accuracy’ and ‘subjective satisfaction’ related to the recommender interface. An example of questions and metrics for the attribute “subjective satisfaction” appears in Table 2. Table 3 shows a sample question and three metrics for the attribute ‘accuracy’.

Finally, subgoal 4 evaluates the authoring and maintenance interface in order to identify how useful this tool is for the students. Like in the recommender interface, some of the attributes to evaluate this sub-goal are ‘usability’, ‘accuracy’ and ‘subjective satisfaction’. Table 4 shows some questions and metrics for the attribute ‘usability’. Table 4 Some questions and metrics for analysing the usability attribute within the authoring

and maintenance interface analysis sub-goal

Questions Metrics How much time do students use the authoring interface?

Total number of hits Total number of hits for each student Total time of use (all students) Average time for each access

Do you consider the authoring and maintenance interface easy to use?

Questionnaire: It is easy to tag a new LO with the concepts of the ontology Questionnaire: It is easy to provide reviews to the new LOs created by other students Questionnaire: It is easy to assign ratings to the LOs Answer (for all of them): Likert scale (Totally agree – Agree – Neither agree nor disagree – Disagree – Totally disagree)

We plan to perform the described evaluation using a repository of LOs for computer programming. Currently, we maintain a web-based repository available through the Virtual Campus at the Complutense University which contains more than 400 computer programming LOs organised in thematic packages. Additionally, we have more than 500 students every year who could become potential users of the resulting application. So we expect to obtain significant results that let assess our recommendation approach in the learning domain.

6 Conclusions and future work

In this article we have described a framework for managing LOR where all the members of the VLC – instructors and students – are involved. The framework describes the main guidelines for the design of a LOR that supports an authoring and maintenance interface for the incorporation of new LOs, their review and the assessment of their usefulness. Additionally, the framework considers the existence of a recommender interface for a personalised access to the stored LOs.

The authoring and maintenance interface lets the LOR evolve over time, due to the inclusion of new LOs as well as the deletion of those that are uninteresting for the VLC. It also lets rate LOs after they have been recommended and review new LOs that are proposed by VLC members. This interface provides two additional advantages. From the student learning perspective, we think that the engagement of the students in the authoring and maintenance tasks induces an improvement in the student motivation


because she takes part of its own learning process and in that of the whole VLC. From the perspective of the personalised content retrieval, this interface collects the preferences of the VLC about the LOs contained in the repository. These preferences are subsequently employed to assess the usefulness of the LOs in order to generate recommendations about the most interesting LOs for a concrete student.

This article mainly focuses on the recommendation interface. This interface alleviates the overload of information that students suffer due to the amount of resources available in the LOR. This way, students receive a personalised access to the LOs contained in the repository according to their short-term learning goals, their mastery level of the learning domain, and the VLC opinion about the LO. The recommendation approach follows a cascade hybrid strategy that refines the decisions of a case-based recommender using a collaborative one. The case-based recommendation strategy promotes personalised access to the LO and the collaborative recommendation strategy adds the community relevance component to the LOs returned by the previous recommender.

In order to evaluate the impact of the proposed managing framework in the VLC, we are designing a goal-oriented evaluation that follows the GQM methodology. The evaluation described here proposes to measure the impact that the use of the described LOR management framework has in the student learning process. This evaluation proposes to achieve the main goal by means of the analysis of diverse aspects of the framework, like the recommendation, authoring and maintenance processes from the student perspective. We work on the improvement of this evaluation by extending it also to the instructor point of view. In this sense, we expect to reuse some of the attributes, questions and metrics already developed.

This approach has been started to use with a web-based repository of LOs for Computer Programming (Ruiz-Iniesta et al., 2009) available through the Virtual Campus at the Complutense University with more than 400 computer programming LOs. The large number of resources in the repository makes difficult the access to them. So, although students really appreciated the support that the repository provides, almost 70% of the students missed facilities for accessing the LOs. Besides, students can only give comments about the LOs in a forum. The framework for managing LOR described here could be applied to such repository providing better support to the use, the maintenance and to the inclusion of new resources.

Acknowledgements

This work has been supported Supported by: Spanish Ministry of Science and Education under grant TIN2009-13692-C03-03; and Complutense University of Madrid and BSCH under grant 921330-1079 for consolidated Research Groups.

References Basili, V.R. and Rombach, H.D. (1988) ‘The TAME project: towards improvement-oriented

software environments’, IEEE Transactions on Software Engineering, Vol. 14, No. 6, pp.758–773.

Brusilovsky, P., Farzan, R. and Wook Ahn, J. (2005) ‘Comprehensive personalized information access in an educational digital library’, Proceedings of ACM/IEEE Joint Conference on Digital Libraries, (JCDL 2005), ACM, pp.9–18.


Brusilovsky, P., Kobsa, A. and Nejdl, W. (Eds.) (2007) The Adaptive Web: Methods and Strategies of Web Personalization, Lecture Notes in Computer Science, Vol. 4321, Springer, Berlin.

Castellano, E.J. and Martínez, L. (2009) ‘A web-decision support system based on collaborative filtering for academic orientation: case study of the Spanish secondary school’, Journal of Universal Computer Science, Vol. 15, No. 14, pp.2786–2807.

Drachsler, H., Hummel, H.G. and Koper, R. (2008) ‘Personal recommender systems for learners in lifelong learning networks; the requirements, techniques and model’, International Journal of Learning Technology, Vol. 3, No. 4, pp.404–423.

Farzan, R. and Brusilovsky, P. (2006) ‘Social navigation support in a course recommender system’, Proceedings of the International Conference on Adaptive Hypermedia and Adaptive Web-based Systems, Dublin, Ireland, pp.91–100.

Gasevic, D., Jovanovic, J. and Devedzic, V. (2007) ‘Ontology-based annotation of learning object content’, Interactive Learning Environments, Vol. 15, No. 1, pp.1–26.

Gómez-Albarrán, M. and Jiménez-Díaz, G. (2009) ‘Recommendation and students’ authoring in repositories of learning objects: a case-based reasoning approach’, International Journal of Emerging Technologies in Learning, Vol. 4, Special Issue: SIIE 2008, pp.35–40.

González-Calero, P., Díaz-Agudo, B. and Gómez-Albarrán, M. (1999) ‘Applying DLs for retrieval in case-based reasoning’, Proceedings of the International Workshop on Description Logics, Linköping, Sweden, pp.51–55.

Gruber, T.R. (1993) ‘A translation approach to portable ontology specifications’, Knowledge Acquisition, Vol. 5, No. 2, pp.199–220.

IEEE (15 July 2002) Draft Standard for Learning Object Metadata. Kardan, A., Abbaspour, S. and Hendijanifard, F. (2009) ‘A hybrid recommender system for

e-learning environments based on concept maps and collaborative tagging’, Proceedings of the 4th International Conference on Virtual Learning, Jassy, Romania, pp.300–307.

McSherry, D. (2002) ‘Diversity-conscious retrieval’, Proceedings of the 6th European Conference on Advances in Case-Based Reasoning, Fethiye, Turkey, pp.219–233.

Monge, S., Ovelar, R. and Azpeitia, I. (2008) ‘Repository 2.0: social dynamics to support community building in learning object repositories’, Interdisciplinary Journal of E-Learning and Learning Objects, Vol. 4, pp.191–204, available at http://www.informingscience.us/ icarus/journals/ijello/publications.

O’Mahony, M. and Smyth, B. (2007) ‘A recommender system for on-line course enrolment: an initial study’, Proceedings of the ACM Conference on Recommender Systems, New York, USA, pp.133–136.

Recio-García, J.A., Díaz-Agudo, B., González-Galero, P.A. and Sánchez-Ruiz, A.A. (2006) ‘Ontology based CBR with jCOLIBRI’, Proceedings of AI-2006, the 26th SGAI International Conference on Innovative Techniques and Applications of Artificial Intelligence, Springer, Cambridge, UK, pp.149–162.

Ruiz-Iniesta, A., Jiménez-Díaz, G. and Gómez-Albarrán, M. (2009) ‘Promoting strong personalization in content-based recommendation systems of learning objects’, 11th International Simposium on Computers in Education (SIIE 2009), Coimbra, Portugal.

Santos, O.C. and Boticario, J. (2008a) ‘Recommendation strategies for promoting elearning performance factors for all’, Proceedings of the 6th Workshop on Intelligent Techniques for Web Personalization & Recommender Systems in Conjunction with the 23rd AAAI Conference on Artificial Intelligence, Illinois, USA, pp.89–98.

Santos, O.C. and Boticario, J. (2008b) ‘Recommendations for providing dynamic inclusive learning’, Proceedings of the 8th IEEE International Conference on Advanced Learning Technologies, Santander, Spain, pp.491–495.

Sarwar, B., Karypis, G., Konstan, J. and Riedl, J. (2001) ‘Item-based collaborative filtering recommendation algorithms’, Proceedings of the 10th International World Wide Web Conference, Hong Kong, China.


Siemer, J. and Angelides, M. (1998) ‘Towards an intelligent tutoring system architecture that supports remedial tutoring’, Artificial Intelligent Review, Vol. 12, No. 6, pp.469–511.

Solingen, R. and Berghout, E. (1999) The Goal/Question/Metric Method, McGraw-Hill Publishing Company, USA.

Swartout, B., Patil, R., Knight, K. and Russ, T. (1997) ‘Toward distributed use of large-scale ontologies’, Proceedings of AAAI97 Spring Symposium Series, Workshop on Ontological Engineering, AAAI Press, pp.138–148.

Tang, T.Y. and McCalla, G. (2003) ‘Smart recommendation for an evolving e-learning system’, Proceedings of the 11th International Conference on Artificial Intelligence in Education, Workshop on Technologies for Electronic Documents for Supporting Learning, Sydney, Australia, pp.699–710.

Tzikopoulos, A., Manouselis, N. and Vuorikari, R. (2007) ‘An overview of learning object repositories’, in P.A. Hershey (Ed.): Learning Objects for Instruction: Design and Evaluation, pp.29–55, Idea Group Publishing, USA.

Wei, K., Huang, J. and Fu, S. (2007) ‘A survey of e-commerce recommender systems’, Proceedings of the International Conference on Service Systems and Service Management, Chengdu, China, pp.1–5.

Worthen, B.R., Sanders, J.R. and Fitzpatrick, J.L. (2003) Program Evaluation: Alternative Approaches and Practical Guidelines, 3rd ed., Allyn & Bacon, USA.

Zaíane, O. (2002) ‘Building a recommender agent for e-learning systems’, Proceedings of the International Conference on Computers in Education, Washington, DC, USA, pp.55–59.

Ziegler, C-N., Lausen, G. and Schmidt-Thieme, L. (2004) ‘Taxonomy-driven computation of product recommendations’, in CIKM ‘04: Proceedings of the 13th ACM International Conference on Information and Knowledge Management, ACM, New York, USA, pp.406–415.