PEER-REVIEW OF REUSABLE LEARNING OBJECTS FOR EBOOKS IN A

PEER-TO-PEER NETWORK ARCHITECTURE

Rajendra G. Singh and Margaret Bernard Department of Computing and Information Technology

Faculty of Science and Agriculture

The University of the West Indies

St. Augustine, Trinidad, W.I.

rajendra.singh@sta.uwi.edu and margaret.bernard@sta.uwi.edu

ABSTRACT It is a challenging task to take standards-based eLearning

content from a client-server environment and place it on a

Peer-To-Peer network to achieve similar goals. The

client-server eLearning system architecture is designed

for storing, sharing and delivering course content. In an

initial pursuit of this challenge, a design for a Peer-To-

Peer Reusable Learning Object Repository was proposed

and prototyped. Now in this continuing work, the

challenge of performing and managing peer-reviews for

the content being shared on the Peer-To-Peer network is investigated, along with how to aggregate and sequence

the content into a course for delivery.

The solution introduces a scalable server design, mainly

to maintain the integrity of peer-review data while leaving

the eLearning content on network peers. From the

prototype, the user can also now aggregate and sequence

peer-reviewed Reusable Learning Objects into a course

after which it is passed to a commercial compiler via

scripting, along with preprogramed templates to produce

eBooks.

The result, is that this extended Peer-To-Peer prototype

application demonstrates an end-to-end solution which can be used to manage and share peer-reviewed Reusable

Learning Objects over a Peer-To-Peer network, and

achieve the final goal of packaging and delivering shared

content as online and offline eBooks.

KEY WORDS EBOOKS, PEER-REVIEW, PEER-TO-PEER and

REUSABLE LEARNING OBJECTS.

1. INTRODUCTION A Teacher who may be an average Internet user and

wants to produce or acquire eLearning content with the

goal of creating and delivering an online or offline course,

may consider that it is still a significant and challenging

task, notwithstanding current pervasive client-server

technologies which suggest that this should be easy to

achieve. The Teacher usually has to select an eLearning

platform in advance, then produce, and/or amend, and/or

acquire from other users the eLearning content that is

compatible with the platform and suitable for the course.

While most client-server eLearning systems are

promoting standard-based content creation, including the use of standards for assessment content, there are further

challenges to produce, amend and acquire the assessment

content which would be relevant for the online or offline

course that is to be built. This task is not trivial, even

when the Teacher is one from a higher-level Institution

where Information Technology (IT) resources are more

readily available; including sometimes specialized

eLearning IT support staff and repositories of content. We

need to therefore consider making the task of producing

and sharing eLearning content and the related assessment

content easier throughout a global education system.

In previous research, the idea of using Peer-To-Peer

(P2P) network architectures instead of the traditional client-server architecture was considered, and the design

of a P2P Reusable Learning Object Repository (RLOR)

was presented by Singh and Bernard [1] to facilitate the

sharing of content over a P2P network.

In pursuing the P2P network architecture, the design

needed to consider further a solution for rating the quality

of the content, which exists in the form of Reusable

Learning Objects (RLOs), without possible loss in

integrity of the feedback data. It also needed to consider

how selected RLOs from the Repository can be

aggregated and sequenced to produce a course, and

facilitate delivery to students. In this paper, we discuss these two additional issues and solutions to these

problems. The solutions are implemented as an extension

of the initial prototype application. The final outputs of

the extended application are eBooks, which can be

delivered to students both as offline and online content.

Following Section 1 (INTRODUCTION), the paper

gives a brief background summary in Section 2

(BACKGROUND) of the work that was previously

completed with respect to the definition of a RLO suitable

for a P2P network, and the design of a P2P RLOR,

including some details of improvements made to the prototype. In Section 3 (PEER-REVIEWS IN A P2P

NETWORK), the challenge and approach adopted for

doing Peer-Reviews and maintaining Ratings for RLOs in

a P2P network is presented and then discussed. Next, in

Section 4 (PRODUCING EBOOKS), the extension to the

P2P prototype application is presented which facilitates

for the aggregation and sequencing of RLOs from the P2P

network into courses and then discuss how it can be

delivered to students as online and offline eBooks. A

discussion relating to the eBook production then follows.

The conclusions are then given in Section 5

(CONCLUSIONS).

2. BACKGROUND

In first considering P2P architectures for managing

eLearning content, Halm et al. [2] looked at the challenge

faced by content authors and instructors when using

client-server solutions. These ranged from managing different versions across different systems, offerings from

one time period and learning situation to another, and a

lack of comfort-level when placing content on Servers

outside the control of the content authors and instructors.

It was felt that if content authors and instructors had a

facility on their own machine for storing their content, it

will be easier to maintain one authoritative version of the

content, and by introducing application features similar to

traditional P2P file sharing network, it will allow for the

content to be Internet accessible as in client-server models

but without added overheads of hardware and complex

network infrastructure. Overall, there should be an improvement in opportunities for content reuse. Halm et

al. [2] on this proposal produced Lionshare, a P2P

application for sharing eLearning content. Their

requirement for the use of the authentication mechanism

of Shibboleth [3] restricted adoption to only a few

selected Tertiary level institutions that had or can access

the required infrastructure. Hatala et al. [4] later

developed an open communication layer for improving on

the P2P content repository security and scalability to have

a wider adoption. This however is a complex design by

their own admission. Richards and Hatala [5] also worked on a P2P learning object system POOL, POND

and SPLASH adopting use of the communication layer

mentioned above. These successes at creating a P2P

system for sharing eLearning content was the motivation

for Singh and Bernard [1] to look at designing a similar

P2P system but this time to consider any Internet-based

user, not just technologically mature Tertiary level

intuitions.

The P2P repository design of Singh and Bernard [1],

built on a definition of Reusable Learning Objects (RLOs)

that seems to be a better fit for P2P sharing than just the

traditional eLearning content, standard-based or otherwise. The RLO to be used on P2P networks

according to this definition must have a required single

stated Learning Objective, reference to the content that

relate to the Learning Objective, and, reference to the

formative and summative assessment content that relate to

the learning content. In a later prototype enhancement of

the repository, the assessment content must also be

formatted using the Question and Test Interoperability

(QTI) standard [6]. This simple RLO would make it easier

to store, manage and share content and handle one of the

major challenge of acquiring assessment content when the RLO is used in a course, something which none of the

popular client-server systems can do without maintaining

databases with questions and answers and providing

search and discovery features to match content with

possible assessment items.

Singh and Bernard [1] also introduced the use of email

certificates for signing the RLO so that changes can be

properly managed. The email certificates are also used to

identify users, which acts as a method to discourage

copyright violations common to the traditional P2P file

sharing networks and applications. This email certificate

identity also allows for users to optionally form groups

and restrict sharing to amongst their group members. The P2P RLOR prototype is based on the open source file

sharing Shareaza project [7] and restricted it to only using

the more mature P2P communication protocol Gnutella2

[8].

The following work on peer-reviews for RLOs and

eBook generation extends this P2P RLOR prototype to

produce a more powerful and useful application for

elevating a P2P eLearning system to a closer or better

level than client-server eLearning systems.

3. PEER-REVIEWS IN A P2P NETWORK Rating files in the many traditional P2P file sharing

applications [9] are done by users. Generally the users

can assign a value between 1 (Poor) and 5 (Excellent),

and they may also have the facility to enter a comment.

This data is stored on the client where it is easy to

manipulate and compromise the integrity of previously

stored review data entered by other users, in effect

making this approach an undesirable one for rating RLOs

in the P2P RLOR. This seems to suggest that any peer-

review data for RLOs need to be either digitally signed or

kept on managed Database (D/B) servers. The latter appears to be especially necessary, if the review data is to

be kept fully updated at all times, to all copies of a given

RLO residing on different peers on the P2P network.

Further, it will be more useful to have subject matter

experts perform reviews on the RLOs and not just any

random user on the P2P network, again suggesting some

kind of account management for reviewer registration

using managed servers.

In further reviewing the approach used by the well-

known server-based repository, Merlot [10], and

considering the above challenges along with some

reviews of research done with respect to rating Learning Objects for quality and effectiveness [11, 12, 13, 14, 15,

16], it was decided that an adapted LORI (Learning

Object Review Instrument) approach would be a good fit

for reviewing RLOs in the P2P RLOR. The design would

incorporate the use of a scalable D/B server architecture

for the peer-review data, with account management

features for managing reviewers, along with a registration

system for users to identify RLOs that are to receive a

review. The use of the servers will not include storing

eLearning content on the D/B or other servers. Instead,

the eLearning content will remain entirely on the P2P network where the P2P application will be extended so

that a registered RLO that is assigned to a reviewer will

be automatically downloaded to the reviewer’s P2P

RLOR. The extension will facilitate the reviewer to enter

all review related data using the P2P application, where it

is then automatically and transparently uploaded via the

reviewer’s account to the D/B on which the RLO is

registered. The process allows for any network peer with

a copy of a given registered RLO to access and retrieve

the peer-review data transparently from the server on

which it is registered, thereby addressing the previously

mentioned issue of having the most current review data

across the entire P2P network. In adapting LORI 1.5 it was determined that only four

of the nine dimensions needed to be captured explicitly,

as some of the items were either not applicable because it

was already inherent in the RLO design that the

dimension is covered sufficiently, or, not applicable at the

RLO level, again because of the design, but is to be given

consideration when aggregating and sequencing for

delivery to the learner. The following gives the adaption

to LORI 1.5 which is used in the design, where the nine

dimensions summary descriptions are taken from [17]:

1. Content Quality (veracity, accuracy, balanced

presentation of ideas, and appropriate level of detail) – the reviewer is presented with a form to

fill-in, built from the rubric given in Figure 1.

2. Learning Goal Alignment (alignment among

learning goals, activities, assessments, and

learner characteristics) – the RLO has a required

learning objective by design, so the reviewer

only needs to indicate if the stated objective is

aligned with the reviewed content using a given

scale.

3. Feedback and Adaptation (adaptive content or

feedback driven by differential learner input or learner modeling) – with the goal to have RLO

with only a single learning objective (as fine a

granularity as possible), we propose transferring

this dimension to a higher level where RLOs are

aggregated and sequenced into a course. It is at

the course level that we should be able to give

more meaningful feedback and adaptation for

learning styles, possibly leveraging information

from a learner’s profile which can be maintained

for structured course offerings over a given

period of time.

4. Motivation (ability to motivate, and stimulate the interest or curiosity of, an identified population

of learners) – as part of the RLO registration

process, the user is required to fill out a form

which captures the attributes Category, sub-

Category, Educational Level and pre-Requisites.

The reviewer is asked if the RLO content

matches against the user selections. If a “No” is

given, the reviewer should enter a comment.

These same attributes are also used to match

against the reviewer’s profile to select possible

subject matter experts to be assigned for the review.

5. Presentation Design (design of visual and

auditory information for enhanced learning and

efficient mental processing) – this is part of the

reviewer’s form to fill-in, as given in the rubric

of Figure 1.

6. Interaction Usability (ease of navigation,

predictability of the user interface (UI), and the

quality of UI help features) – this is part of the

reviewer’s form to fill-in, as given in the rubric

of Figure 1.

7. Accessibility (support for learners with disabilities) – the belief is that this may be better

determined at the course level and not at the

RLO level. It is therefore omitted from the RLO

review.

8. Reusability (ability to port between different

courses or learning contexts without

modification) – the RLO is designed to have the

single learning objective, content and assessment

contents so no additional information has to be

explicitly gathered from the reviewer.

9. Standards Compliance (adherence to

international standards and specification) – the main content may or may not be standards-

based. The reviewer indicates if it is in any

standard-based format such as LMS or SCORM.

It is only informational and does not impact on

the review score. In the updated design of the

RLO it is now necessary for the all assessments

to be in QTI format. The reviewer checks this

and indicates with a Yes/No. A No immediately

causes the assessment score to be set to zero.

The following summarized algorithm is used to arrive

at a rating score for the dimensions of the LORI 1.5 implemented:

Begin P2P peer-review algorithm {

• The user request registration of a RLO on a

given server, is assigned a GUID (Globally

Unique Identifier) and digitally signs the RLO

with the GUID inserted, the RLO state is set to

“Request Review”, the user is presented with a

data-form to capture the following:

1. Select Category from a predefined list; if

item does not exist, allowed to add;

2. Select sub-Category from a predefined list; if item does not exist, allowed to add;

3. Select Educational Level (pre-school /

primary / secondary / university); if item does

not exist, allowed to add;

4. State any pre-requisites (restricted to fixed

number of characters);

5. Comment (restricted to fixed number of

characters);

6. Select the number of reviewers required for

reviewing the RLO (minimum of 2);

• Match registered subject matter experts

(reviewers) with 1 to 3 of the data-form; (alert if no matches found);

• Randomly select reviewers up to the amount

assigned in 6 of the data-form; (weighted against

selected attributes of reviewer’s history and

current load when assigning); (alert if not

enough reviewers);

• If no alerts, set RLO state to “Reviewers

Assigned” and create a temporary group with all

selected reviewers and the user, to facilitate

reviewers to access the RLO; else, inform user of

alert(s);

• Set RLO state to “Review in Progress”;

• For each reviewer selected {

• The reviewer’s P2P RLOR is automatically

updated to include a download of the RLO

from the user RLOR requesting the review.

Each reviewer can confirm the download

and when the RLO is fully retrieved they can

proceed with the review;

• Review RLO and assign a rating score for A.

Learning Objective, B. Content, C.

Assessment (Formative) and D. Assessment

(Summative), using the P2P RLO rubric of Figure 1;

• Average Score = (A+B+C+D)/4;

• If A < 2 or B < 2 or data-form 1 to 4 does

not match the content, then set Average

Score to 0;

• Calculate Average Rating Score for all

reviews completed thus far and assign to

RLO;

• Indicate the number of reviews completed

against the number remaining;

• Flag RLO for removal from reviewer’s P2P RLOR;

}

• Set RLO state to “Review Completed”;

• Delete the previously created temporary group

with reviewers and user;

}

End P2P peer-review algorithm.

All rating and review data is stored on the D/B where

the RLO is registered and is retrieved transparently using

the GUID by the P2P RLOR where any copy of the

registered RLO may subsequently exist. The user who

initially registered and digitally signed the RLO can also

resubmit an improved version for a new review. Any copy of the previous version will receive notification that it has

been updated when the rating and review data is accessed

from a P2P RLOR. The P2P RLOR will be notified that a

new version exists on the P2P network giving the new

registered GUID of the RLO allowing for an automatic

process for downloading the new version of the RLO if

desired.

3.1 Peer-Review Discussion

The peer-review approach for the RLOs in P2P

networks cannot be the simple process similar to rating files in a traditional P2P file sharing application. It needs

to be similar to strategies adopted in client-server

systems, and in this case, the adaption of LORI seems to

be a reasonable fit. The strategy of the user initiating a

registration of RLOs for reviews, allows for the user to

consider only when the RLO is matured. It may be

possible to employ a peer-review process amongst users

on the P2P network similar to [18] for maturing the RLOs

prior to submission for peer-review by subject matter

experts.

The reviewer’s task is kept to a minimum as much as possible by having users supply some information similar

to what is required for filling-in metadata tags for

standards-based content such as [19], and then having the

reviewers validate with a simple option selection, to

facilitate some aspects of the rating. This reduces the

work considerably and more focus can be placed on

Scale: 0 (None), 1 (Poor, needs work), 2 (Acceptable, but can be improved), 3 (Average), 4 (Above average), 5 (Excellent).

Learning Objective:

A. How well stated is the learning objective? (1 – 5) [Comments]

B. Is it a single learning objective? (Yes/No)

If B = Yes, Learning Objective Score = A else if A > 0 Learning Objective Score = 1

Content Quality:

A. How accurate is the content material? (1 – 5) [Comments]

B. How detail is the content material? (1 – 5) [Comments]

C. How well related is the content material to the stated Learning Objective? (1 – 5) [Comments]

D. How well is the presentation of the content material? (1 – 5) [Comments]

E. How well can the user navigate the content material? (1 – 5) [Comments]

F. (Informational only) Is the content in a standards-based format? (Yes/No) [List format(s)]

Content Quality Average Score = (A + B + C + D + E) / 5

If the Assessments are not in QTI format, Assessment Content Quality Average Score = 0, otherwise proceed with rating the Assessments.

Assessment Content Quality (separate reviews are done for each of Formative and Summative Assessment Content):

A. How well do the questions match the content? (1 – 5) [Comments]

B. How accurate are the answers to the questions? (1 – 5) [Comments]

C. How is the variety and number of questions for the content? (1 – 5) [Comments]

Assessment Content Quality Average Score = (A + B + C) / 3

Figure 1 – P2P RLO Review Rubric

reviewing the content and assessment content itself

against the learning objective for the quality

determination.

While the algorithm and overall peer-review process

explained seems reasonably sound it can be considered

for further validation using quantitative studies. At this time however, the RLOs in the P2P network are too

minimal to consider such studies, but one which would be

possible later on, as more RLOs are developed and added

on the network for sharing and reviews.

4. PRODUCING EBOOKS

The goal of producing RLOs and sharing with other users

is to have them adopted for delivery to learners in the

form of a computer-based course. In initially designing

the P2P RLOR, it was being considered that the RLOs

from the Repository would be used in the traditional client-server eLearning systems, where appropriate

plugins can be built to allow the RLOs to be used in such

systems. A Moodle plugin similar to DOOR [20] was

being considered. It was later noted that since the P2P

RLOR was facilitating any Internet-based user to be a

producer then it should allow similarly for any group of

Internet-based users to be consumers. Further, having

Internet-based course-creators would mean that only such

users with access to a client-server eLearning system (and

plugin for the P2P RLOR) would be able to create and present a course. This limitation was considered too

restrictive for the average Internet-based user. As a result,

the idea of using the RLO to produce eBooks was

explored and determined to be a better fit into the P2P

RLOR philosophy.

The general approach used to generate eBooks employs

the use of an eBook compiler. It was discovered that the

feature lists for eBook compliers vary widely and the

review at [21] proved useful. The final choice however,

was the compiler at [22] simply because of the maturity

for programming features, licensing and eBook activation options. For the P2P RLOR prototype the commercial

version of the compiler was acquired in order to have the

full feature set. The Digital Rights Management (DRM)

and activation management were two important features

necessary to determine if the eBook would be used to

deliver secure summative assessments. The combination

of having JavaScript and imbedded programming facility

also influenced the choice. The support for many audio

and video formats apart from including HTML and PDF

formats of content were also key considerations. Further,

online webpages can be viewed within the eBook without

the use of an external browser. The first step in generating an eBook using the P2P

RLOR, is to select the RLOs for the course from peers on

the P2P network, and create new ones as needed for the

specific course which may not exist already on the

network. The RLOs are then sequenced into a course

structure via a drag-and-drop Table of Content (TOC)

Interface built into the extended P2P RLOR. The interface

allows for including documents which are needed for the

course but not RLOs, and web links to Internet sites

which can be viewed from within the eBook. An example of a document which may be used that is not a RLO is

one with the details on the course itself, and a web link

example could be an Internet site linking to the Teacher’s

information or supplementary course content and

references. Once the TOC is built, the P2P RLOR

generates an XML document that is passed to the

Figure 2 – Screenshot of an eBook created using RLOs from the P2P RLOR prototype

compiler which has all the information required to access

all RLOs, Scripts, documents and external web links. Pre-

built templates can also be selected and passed to the compiler which determines the look and behavior of the

eBook, for example, a template can be passed with an

expiry date set for the eBook. This allows users to register

and activate the eBook before the content is accessible,

present the modules in a given order (only when pre-

requisites modules are completed then the user is allowed

access to the next module), and release the summative

assessments on a specific date and time or when the user

applies for an activation key. An important design in the

TOC is the facility to aggregate and sequence the

assessment content from several RLOs to a location such

as at the end of a module, and to repeat some assessments at different review points. The XML TOC file, templates

and other files and parameters are passed to the eBook

compiler using a command line script generated by the

P2P RLOR. The user does not have to interact directly

with the compiler. Figure 2 shows a screenshot of an

eBook for a first year undergraduate Database

Management Course. The TOC on the left allows the user

to navigate to different topics, and on the right, the

documents, web site and RLO content is displayed. In the

screenshot an RLO is selected and the content tab shows

the main content of the RLO. In Figure 3, another screenshot is shown with the same RLO selected, but this

time the assessment Tab and Formative Assessment Tab

(Practice Test) is selected. The user can proceed to answer

the questions and review the answers for reinforcing the

learning content. An open source QTI player written in

JavaScript by [23] is embedded into the eBook to

facilitate the Assessment interactions.

Figure 4 shows the screenshot of the Rating tab. If the

eBook is used while online it displays the real-time rating

and review data details for the RLO retrieved from the server on which the RLO was registered for review and

where the data is stored. The Signature tab will show the

digital signature for the RLO and indicate if it is signed

using a valid email certificate.

4.1 eBook Discussion

At this stage producing standard-based eBooks was not

considered simply because there was not a good fit which

exists for managing the summative assessment. The

DRM, activation of the eBook and activation of specific

pages were important features for facilitating the summative assessment, with ongoing plans to now have

the scores from the assessment securely uploaded to a

server where the user profile from activation of the

eBooks is maintained. Also, similar to how the rating is

retrieved and displayed for a RLO the scores can remain

secure and be displayed for the assessments. The

production of an eBook as an EXE (executable) file limits

the platform to Microsoft Windows clients; they cannot

be used on MAC and Linux clients or mobile devices.

This limitation was due to the security features already

identified, however, an eBook player using Microsoft Silverlight [24] and HTML5 [25] is being prototyped by

an undergraduate computer science student which will

allow for reviewing two possible technologies that can be

used to widen the platform for eBook consumption.

Achieving the goal of delivering an eBook to any

Internet-based learner, and having being produced from

peer-reviewed RLOs shared on a P2P network comprising

Figure 3 – Screenshot of eBook with Formative Assessment Tab selected

users from any location on the Internet, is significant. It

may now be possible to exceed the reach of Teaching and

Learning beyond what is common to client-server

eLearning systems!

5. CONCLUSION

Peer-review of RLOs in a P2P network is more

challenging than a client-server environment because the

integrity of the feedback from the reviews can be easily

compromised if it resides on a peer. The decision to maintain this data integrity by using a D/B Server is a

necessary one. In the prototype implementation, it is

shown that the user can from the peer initiate a

registration of an RLO on the server and reviewers can

from their own peer complete the review without

accessing the server directly. Apart from the registration

process, all other server involvement to have the feedback

stored and accessed on the D/B is transparent to all peers

involved in the review process and subsequently the peers

that will later access the rating information when users are

viewing the feedback. The similar techniques of the eBooks accessing the rating for registered RLOs are also

transparent. The server component is built using

technologies that allows for a scalable implementation,

where any number of servers with different owners can

participate in the process. This modularity and scalability

is important, since it lends supports for different groups of

P2P users to have their own servers and management

process, as well as for more public groups and individuals

to use publicly available servers for their management of

the reviews.

Aggregating and sequencing peer-reviewed RLOs from

the P2P RLOR into eBooks introduces a sense of

completion when creating and sharing RLOs with peers

on a P2P network. The inherent design of the RLOs for

the P2P network to have the Learning Objective, content

and assessment content, all related and contained as the

structure of the RLO, along with the digital signature of

the author, inherently introduce a level of reusability

which reduces effort required to implement LORI 1.5 for

the peer-review, and the effort of retrieving assessment

content when generating eBooks. Overall, the introduction of the peer-review and eBook

generation has matured the initial prototype P2P RLOR

into a possible first generation useful product, for any

group of Internet-based users wanting to produce and

share eLearning content and deliver (as eBooks) to

students leveraging P2P network architecture. It is now

with eager anticipation as the P2P RLOR is adopted and

used by both users from the teaching and learning

domains, that further investigations involving quantitative

methods can be used to determine possible improvement

for eLearning implementations in a P2P network architecture.

ACKNOWLEDGEMENTS

We would like to express our gratitude to two very

enthusiastic students: Naresh Seegobin, a Post Graduate

student and avid Software Developer, and Samuel

Thompson, a final year Undergraduate Computer Science

student for their skillful programming contributions in

developing the group management and content rating

ASP.NET DNN (DotNetNuke) modules. Also, it is not a

Figure 4 – Screenshot of Rating tab showing rating and review data

common occurrence to discover a first year

Undergraduate who can program at the level of graduates.

Jarrerd Hosien was such a discovery and we thank him

immensely for his contributions in attempting to develop

the first alpha Silverlight and HTML5 eBook

reader/player for the P2P RLOR prototype. We wish them all the best in their future endeavors.

REFERENCES [1] R. G. Singh & M. Bernard, The design of a Peer-To-Peer Reusable Learning Object Repository, Proc. 6th International

Conference on Education, Samos Island, Greece, 2010, 412-418. [2] M. J. Halm, R. Shuey, M. Hatala, G. Richards & M. Dooris, LionShare: Connecting and Extending Peer-to-Peer Networks, A Penn State Proposal to the Andrew W. Mellon Foundation, accessed April 30, 2010, http://www.cob.sjsu.edu/jerrell_l/ lionshare_ mellon_pdf.pdf. [3] Shibboleth, accessed March 20, 2012, http://shibboleth.internet2.edu/.

[4] M. Hatala, G. Richards, T. Eap & A. Shah, Secure Communication Layer for Scalable Network of Learning Object Repositories, In S. Pierre, E-Learning Networked Environments

and Architectures: A Knowledge Processing Perspective, New York: Springer, 2006, 276-305. [5] G. Richards & M. Hatala, POOL, POND and SPLASH, In R.

McGreal, Online Education using Learning Objects, London: RoutledgeFalmer, 2004, 236-243.

[6] IMS Question and Test Interoperability Integration Guide, Version 2.0 Final Specification, accessed April 30, 2010, http://www.imsglobal.org/question/qti_v2p0/imsqti_intgv2p0.html. [7] Shareaza, accessed April 30, 2010, http://shareaza.sourceforge.net. [8] Gneutella2, accessed April 30, 2010, http://g2.trillinux.org. [9] Top 10 Most Popular P2P File Sharing Programs in 2009, accessed May 10, 2010, http://www.p2pon.com/top-10-most-

popular-p2p-file-sharing-programs-in-2009-at-your-request. [10] MERLOT: Multimedia Educational Resources for Learning and Online Teaching, accessed February 18, 2012, http://merlot.org/. [11] J. Nesbit, B. Belfer & J. Vargo, A Convergent Participation Model for Evaluation of Learning Objects, Canadian Journal of

Learning and Technology, 28 (3), 2002, accessed March 20, 2012, http://www.cjlt.ca/index.php/cjlt/article/view/110/103.

[12] I. Box, Submission and Peer Review of Learning Objects Using a Community-Based Repository, Proc. of the 2004

Informing Science and IT Education Joint Conference, Rockhampton, Australia, 2004, 113-121. [13] Y. Akpinar & H. Simsek, Should K-12 Teachers Develop Learning Objects? Evidence from the Field with K-12 Students, International Journal of Instructional Technology and Distance

Learning 4(3), 2007, 31-44, accessed March 1, 2012,

http://www.itdl.org/Journal/Mar_07/Mar_07.pdf. [14] J. McKenzie, L. Pelliccione & N. Parker, Developing peer review of teaching in blended learning environments: Frameworks and Challenges, Proc. Ascilite, Melbourne, 2008, 622-627. [15] Y. Akpinar, Validation of a Learning Object Review Instrument: Relationship between Ratings of Learning Objects and Actual Outcomes, Interdisciplinary Journal of E-Learning

and Learning Objects 4, Ed. Heinz Dreher, 2008, 291-302. [16] A. Auvinen, The challenge of quality in peer-produced eLearning content, eLearning Papers, No. 17, 2009, accessed

February 18, 2012, http://www.elearningeuropa.info/files/media/ media21212.pdf. [17] Learning Object Review Instrument accessed February 18, 2012, http://edutechwiki.unige.ch/en/Learning_Object_Review_ Instrument.

[18] E. Gehringer, et al., Reusable Learning Objects Through Peer Review: The Expertiza approach, Innovate 3 (5), accessed March 3, 2012, http://www.innovateonline.info/pdf/vol3_issue5/ Reusable_Learning_Objects_Through_Peer_Review__The_Expertiza_Approach.pdf. [19] IMS Global Learning Consortium Specifications, accessed April 30, 2010, http://www.imsglobal.org/specifications.html. [20] DOOR - Digital Open Object Repository, accessed April

30, 2010, http://door.sourceforge.net/. [21] eBook Complier Review, accessed February 18, 2012, http://www.ebookfreeway.com/p-compiler-review.html. [22] HTML Executable eBook Compiler, accessed February 18, 2012, http://www.htmlexe.com/index.php. [23] qti–player, accesses February 18, 2012, http://code.google.com/p/qti-player/. [24] Sliverlight, accessed February 18, 2012,

http://www.silverlight.net/. [25] Building Apps with HTML5: What You Need to Know, accessed February 18, 2012, http://msdn.microsoft.com/en-us/magazine/hh335062.aspx.