Evaluating the E-Learning packages on the …...3 (Bransford et al., 2000; Mödritscher, 2006). The e-learning system should have an intuitive design, where users can navigate, find,

Service Evaluation Project Evaluating E-Learning

0

Evaluating the E-Learning packages on the

University of Leeds Clinical Psychology

Doctoral Training Programme: Pedagogy

and Instructional Design

Jamie Barrow

Commissioned by Clare Dowzer, Leeds DClinPsychol programme


1

Introduction

Factors important for learning

When designing educational resources it is important to include factors that promote

successful learning, referred to as Pedagogy and Instructional Design components.

Pedagogical factors support the active nature of the learning process and are designed

to impart knowledge (Mehanna, 2004), with a focus on: learning content, user needs,

and learning outcomes (Lim & Lee, 2007). Instructional Design (ID) relates to

experiences of learning and the features of learning environments which promote

knowledge and skill development (Merrill, Drake, Lacy, & Pratt, 1966).

Pedagogical factors can be understood through a number of theoretical constructs such

as Behaviourism; where learning occurs via a transmission of knowledge, Cognitivism;

the creation of cognitive change and processing, and Constructivism; where learning

occurs through interaction (Lim & Lee, 2007).

Bransford, Brown, and Cocking (2000) identify four factors as being significantly

important for the learning process: attention, motivation, emotions, and experiences of

the learner. The authors describe how learners must be engaged with the information

and motivated to acquire knowledge, and that successful learning environments must

capture this by providing opportunity for interaction, “doing” and receiving feedback

on their performance to develop and refine their understanding.

Tavangarian, Leypold, Nölting, Röser, and Voigt (2004) discuss learning from a

constructivist standpoint, where learners assimilate knowledge by building on previous

knowledge and experiences. They describe “learning objects” metaphorically as Lego©

blocks which connect with one another to develop overall understanding. The size of

these learning objects must be considered to facilitate learning, as they cannot be too


2

big or unconnected to previous ideas as this will hinder overall understanding.

Mödritscher (2006) supports the idea that learner characteristics influence successful

learning, identifying prior knowledge, the individual’s cognitive and learning style, and

intellectual capabilities as important considerations.

E-Learning

E-Learning is use of digital technology to present information. The use of e-learning

resources has benefits to both the teaching institutions and learners, such as providing

a method through which to share information and resources quickly, on a large scale

and in a relatively cost-effective way (Gordon, 2014). They provide flexibility to

teachers and learners regarding when the resource is accessed and completed (Gordon,

2014; Yelland, Tsembas, & Hall, 2008).

Daskalakis and Tselios (2013) report that users are generally motivated to complete

online learning and that this trend has been increasing as technological support becomes

better able to facilitate interactive and multimedia features within the learning

materials. This allows a flexible method of study, where users can access information

remotely, any time, and at their own pace.

Ruiz, Mintzer, and Leipzig (2006) discuss using a ‘blended learning approach’ where

e-learning resources are used alongside classroom methods to support learning. They

identify benefits of e-learning use such as; cost-saving, increased access and availability

of material for learners, and student satisfaction.

Yelland et al. (2008) warn that simply transferring paper-based learning resources to an

online format is insufficient. Effective e-learning should motivate and engage users via

techniques such as story-telling, animation, and by building on prior knowledge


3

(Bransford et al., 2000; Mödritscher, 2006). The e-learning system should have an

intuitive design, where users can navigate, find, and access information without

distraction from poor design or function (Daskalakis & Tselios, 2013) allowing them

to maintain attention and focus to the content.

Engaging the user via objective setting, interaction and varied presentation of

information, provision of feedback and recognition can support learning outcomes

(Mehanna, 2004). Paechter, Maier, and Macher (2010) explored the opinion of 2196

university students using e-learning as part of their degree, which varied in terms of

subject and degree level (Batchelors, Masters, PhD). The authors report that

achievement goals were considered most important for learning. They therefore suggest

that the content of e-learning courses is designed to influence motivation through the

provision of clear learning objectives, opportunities to practice and apply knowledge,

including feedback to measure progress throughout.

Martín-Rodrígueza, Fernández-Molinab, Montero-Alonsoc, and González-Gómezd

(2015) report course design and content was the most relevant feature for student

satisfaction, including interaction with the course and tutor support to promote learning.

Interaction with course instructors can support learning outcomes and satisfaction in

the course, and so course instructors do not become less valuable in e-learning and

should be involved when developing resources (Mehanna, 2004; Paechter et al., 2010).

Whilst students value IT as a convenient resource which supports their learning (Smith

& Borreson Caruso, 2010), technology is not a substitute for face-to-face interaction

with tutors (Borreson Caruso & Salaway, 2007).

Interaction between students has been identified as an important feature in e-learning

environments however, the evidence is mixed regarding its use (Ituma, 2011). The


4

author reported limited support for the use of the ‘chat’ component and that this was

rarely used by students. It is therefore necessary to explore user feedback to discover

which resources students’ value or consider inessential.

A criticism of e-learning research and evaluation is that this often explores use and

satisfaction from developer/teacher perspective rather than learner experience (Ruiz et

al., 2006). Ruiz et al. (2006) promote the need to engage users in evaluation to ensure

that the materials are also meeting the needs and aims of students. Islam (2013)

identifies the value of teacher/learner collaboration in the development and use of such

resources as they should meet the needs of both of both user groups.

Use of E-Learning on the Doctorate of Clinical Psychology (DClinPsy) Training Course

The DClinPsy staff team made the decision to create e-learning resources to as a method

to provide trainees with information that can be accessed easily and flexibility to

support trainee learning.

Some aimed to provide a base level of information or to refresh student knowledge

prior to face-to-face teaching sessions; for example “Introduction to statistics”. This

was the first package to be developed based on trainee feedback that a refresher would

be useful prior to the teaching sessions, as individual experiences varied regarding use

of statistics in their pre-training careers.

Others were developed to deliver information that was not scheduled as a teaching

session but were potentially beneficial for trainees; such as Literature searching

package. Not all packages require completion as a course requirement but were

identified as useful for trainees to access as they conduct academic assignments,

dictated by individual learning goals and needs.


5

An anticipated benefit was that trainees would be able to spend as much or little time

completing e-learning as necessary for the individual, as well as being something which

can be accessed repeatedly as required. By providing refresher or base-level knowledge

prior to a face-to-face teaching session it was hoped that trainees are supported to make

the best use of the teaching time in a blended learning format (Ruiz et al., 2006).

The packages were developed by the learning technologist in collaboration with course

tutors or other subject experts.

How E-Learning has been evaluated by others

Due to the growing use of e-learning as a mode of information delivery, it is important

to assess that e-learning packages are capturing elements which facilitate learning in a

way which is as effective as face to face teaching methods.

The literature around evaluating e-learning is expansive and rather varied in regard to

what e-learning looks like, ranging from single session delivery to full virtual learning

environments. Due to this variation there are many different models of evaluation to

choose from, each with slightly different focus depending on the aims of the researcher.

A literature review by Dorobat (2014) identifies and discusses a number of evaluative

models before suggesting their own model that draws together the important features

of previous evaluative frameworks. They proposed a comprehensive framework for “E-

Learning System Success” (ELSS), a simplified version of which can be found in

Figure 1. Dorobat identified six areas which should be considered when exploring

successful e-learning and proposed that these can impact and influence one-another.

Whilst the ELSS provides a useful framework of pedagogy and ID features from which


6

to begin assessing e-learning, it does not provide specific components to assess when

conducting an evaluation.

Figure 1: Dorobat’s proposed simplified ELSS model (2014)

Oztekin, Kong, and Uysal (2010) proposed a checklist called “UseLearn”, made up of

36 criteria to evaluate the quality and usability of e-learning systems. They discuss the

importance of e-learning being easy for learners to use without system error. Attention

has been identified as an important component of successful learning, and so users

should be supported to access information and remain focused on learning content and

activities without distraction from functionality issues within the learning platform

(Daskalakis & Tselios, 2013; Islam, 2013). The UseLearn checklist provides useful

items from which to assess the functionality of ID features of a learning environment,

however there is an absence of factors assessing pedagogical features.

Holsapple and Lee-Post (2006) describe system features that support successful e-

learning conditions and propose assessment should include evaluation of the system

design, delivery, and user satisfaction. The authors acknowledge that this is an iterative

process and evaluation should include identifying barriers to learning, developing and

implementing improvements, and re-evaluation. This service evaluation project (SEP)

is the first step in evaluating the e-learning packages on the DClinPsy training

Perceived Control

Social Factors Benefits

Quality Usefulness &

Satisfaction

User Attitude


7

programme by assessing the ID and pedagogical components, including user feedback

where available.

There are a variety of different rubrics and checklists that can be used to evaluate e-

learning. The content of such tools varies depending on the aims of both the information

provider and learner however, a number of common themes were apparent in

supporting knowledge acquisition. The ideas from the literature will be used to inform

the present evaluation.

Commissioning and Project Aims

The purpose of this SEP is to investigate how well the current packages include

pedagogical and instructional design elements to promote successful learning. From

this we hope to make recommendations as to how existing packages can be refined and

improved, which may also influence how upcoming e-learning packages are developed.

The project was commissioned by Dr Clare Dowzer: Learning technologist and

Research Coordinator at the University of Leeds, and Dr Gary Latchford: Joint

Programme Director of the University of Leeds DClinPsy training programme. Over

the last three years seven e-learning resources have been developed:

1. History taking and Record Keeping (HTRK)

2. Single Case Design (SCD)

3. Neuropsychology

4. Presenting Data

5. Introduction to Statistics

6. Psychometrics

7. Literature Searching and Reviewing (LSR)


8

These were created by adding an element of interactivity to existing PowerPoint

presentations in order to transfer these teaching materials from more interactive, face

to face learning into an online resource. These packages range from those which replace

a single lecture (Presenting Data), to more comprehensive learning which may take

several hours to complete (Neuropsychology). The service evaluation project was

commissioned to assess these packages from a pedagogical and instructional design

perspective using a checklist tool based on the literature, and available feedback from

trainees who have accessed the e-learning packages. It is expected that this evaluation

will identify the strengths and limitations of the e-learning and make recommendations

as to how these might be improved for future users.

Research Aims:

1. To evaluate the Pedagogical and Instructional design properties of the e-

learning packages and make recommendations for improvement.

2. To assess user satisfaction through the analysis of evaluation questionnaires.

Method

Design

A quantitative approach was used to assess the presence of pedagogical and

instructional design features across the seven e-learning packages. A mixed measures

approach was considered to explore user experiences, however due to the size

restriction of this project it was thought this would be more thoroughly explored as a

separate project which could include teacher and learner perspectives.

Developing a Measure


9

From the initial literature review a measure was not identified which would meet the

evaluation requirements of this project, due to length or the scope of the tool. Therefore,

a brief content analysis identified recurring elements in measures which are used to

evaluate various e-learning packages. The ELSS (Dorobat, 2014), Uselearn (Oztekin et

al., 2010), Holsapple and Lee-Post (2006), and Bransford et al. (2000) were used to

inform and devise a checklist (Appendix 1) which could be applied to each of the e-

learning packages. This process was done in consultation with the project commissioner

to ensure this would meet the project requirements.

Checklist items were grouped together to inform compliance to the following

subheadings/criteria:

Category To assess Informed by Learning Techniques

Was information presented to support learning and application of knowledge?

ELSS (Dorobat, 2014)

UseLearn (Oztekin et al., 2010)Bransford et al. (2000)

Accessibility

Does the e-learning allow users to access information easily?

ELSS (Dorobat, 2014) UseLearn

System quality

Does the e-learning platform function sufficiently to support learning without distraction? (e.g. from poor design or technological error)

ELSS (Dorobat, 2014) Holsapple and Lee-

Post (2006) UseLearn (Oztekin et

al., 2010)

Usefulness & Satisfaction

Does it meet user requirements/expectation?

ELSS (Dorobat, 2014) Bransford et al.

(2000) Holsapple and Lee-

Post (2006)

Social factors Are users able to interact with others to promote learning?

ELSS (Dorobat, 2014)

Across these five categorise there were 21 items, some of which were broken down into

sub-checklist items to support decision making. User feedback was used to inform the


10

compliance rating on five items, which are identified on the checklist (e.g. “Is the

training pitched at the correct level?”).

Ethical considerations

Review by an ethics panel was not necessary for this project as no participants were

recruited. The analysis required access to the learning packages via the University of

Leeds Minerva platform and routinely gathered, anonymous, user evaluation data.

Procedure

The checklist was and applied to each of the seven e-learning packages. Each item

would be rated as “Present”, “Partial” [partially present], “Absent”, or “Not

Applicable/No Data”. The researcher and project commissioner completed and

compared the results on the HTRK package to establish agreement regarding the rating

criteria and to improve the reliability of the ratings.

Data analysis

Quantitative data analysis using descriptive statistics was used to calculate compliance

for each package. This score was expressed as a percentage of items: Present, Partially

Present, Absent, or NA/No Data in each of the pedagogical/instructional design

categories and an overall compliance score.

User feedback was available on four of the seven packages: SCD, HTRK,

Neuropsychology and Presenting Data. The routinely collected DClinPsy e-learning

evaluation questionnaire (Appendix 2) also asked users three qualitative questions

regarding: Objectives, Assessment, and Suggestions. There were not enough qualitative


11

data from the completed trainee feedback forms to conduct a robust thematic analysis

as described by Braun and Clarke (2006). Available feedback has instead been

considered during the rating of some checklist components and to generate preliminary

themes to inform the recommendations.

Results

The results shall be considered in terms of overall compliance to the checklist, followed

by a breakdown of how this was informed by each pedagogical and instructional design

category.

Overall Compliance

A breakdown of compliance across all packages can be seen in Figure 1. Overall

compliance to the checklist varied across the packages with an average ‘Present’ rating

of 50% (range = 36.36 - 63.64%). When considering both present and partially present

items overall compliance is high across the packages (mean = 81.17; range= 68.18 –

90.91%). There were very few items rated as ‘Absent’ (mean = 5.2%; range = 0 –

13.64%) however, the results are influenced by the number of items rated as N/A or

where data were unavailable (mean = 10.39%; range = 4.55 – 22.73%). The most

compliant package was SCD, followed by HTRK; the lowest rated compliance was for

Presenting Data (as shown in figure 2).


12

Figure 2: Overall Checklist Compliance across the E-Learning packages

Learning Techniques

The pedagogical and instructional design category “Learning techniques” had the

highest level of ‘present’ rated compliance across the e-learning packages (mean =

90.47%; range = 83.33 – 100%). As shown in Figure 3; Neuropsychology, SCD and

Presenting data all achieved 100% ‘present’ ratings; HTRK achieved 83.33% (100%

when present and partial were combined). For the three remaining packages there were

no available data from the user feedback component. There were no “absent’ ratings in

any category. The strong compliance suggests that the e-learning package present the

information well, using techniques which can promote successful learning.

40.91

36.36

45.45

59.09

50.00

54.55

63.64

27.27

36.36

27.27

22.73

40.91

36.36

27.27

13.64

4.55

4.55

0.00

4.55

4.55

4.55

18.18

22.73

22.73

18.18

4.55

4.55

4.55

0 10 20 30 40 50 60 70 80 90 100

Presenting Data

Intro to Statistics

Literature searching & Reviewing

Psychometrics

Neuropsychology

History taking and record keeping

Single case design

Compliance (%)

E-Le

arn

ing

Pac

kage

Overall Compliance

Present Partial Absent No Data or N/A


13

Figure 3: Compliance to “Learning Techniques” Category

System Quality

‘Present’ ratings across the e-learning packages for the System Quality category was

much lower (Figure 4), achieving a mean of 40% (range = 20-80%). There was a high

level of partially achieved components (mean = 40%), indicating that some features are

included but require improvement. Some examples of this include:

“Are files easy to upload/download or view?” - links present but no longer

functioning.

“Error prevention” – Interactions with animations did not always function.

Users were not always informed why they could not move on (e.g. if a task was

incomplete or incorrect).

83.33

83.33

83.33

83.33

100.00

100.00

100.00

0.00

0.00

0.00

16.67

0.00

0.00

0.00

16.67

16.67

16.67

0.00

0.00

0.00

0.00

0 10 20 30 40 50 60 70 80 90 100

Psychometrics

Intro to Statistics



Presenting Data

Single case design

Neuropsychology

Compliance (%)

E-Le

arn

ing

Pac

kage

Learning Techniques



14

“Is the course well organised?” – Users could not navigate to a specific page of

information if they wanted to recap this. Not all packages provided information

about where they were in the training or had multiple routes to the same

information.

When considering both present and partially achieved items as compliant, the mean

compliance increases to 80% (range = 60-100%).

Figure 4: Compliance to “System Quality” Category

The results indicate high levels of variation in the successful implementation of

components which support the e-learning environment and performance, and that

these can be improved across the e-learning packages.

Accessibility

“Accessibility” achieved a lower compliance rating across the E-Learning packages,

with a mean ‘present’ rating of 40% (range = 20-60%). When combining ‘present’ and

20

40

40

60

20

20

80

40

20

20

20

80

80

20

20

20

20

20

0

0

0

20

20

20

0

0

0

0

0 10 20 30 40 50 60 70 80 90 100

Intro to Statistics


Presenting Data

Single case design


Neuropsychology

Psychometrics

Compliance (%)

E-Le

arn

ing

Pac

kage

System Quality



15

‘partial’ this increased to a mean of 77.14% compliance (range = 40-100%). As shown

in Figure 5, SCD achieved the highest level of compliance, whereas Presenting Data

has the lowest. Presenting Data also has the highest level of ‘absent’ features on this

category than any other checklist category (40%).

Figure 5: Compliance to “Accessibility” Category

These results suggest that accessibility varied across the e-learning packages (e.g.

function across internet browsers or mobile devices). No package achieved a ‘present’

rating for the ability to adjust the display settings to user preference (e.g. colour, size)

which could be an important feature regarding access for those with different learning

needs.

Usefulness and Satisfaction

20

20

40

40

40

60

60

20

60

40

40

40

20

40

40

0

0

0

20

20

0

20

20

20

20

0

0

0

0 10 20 30 40 50 60 70 80 90 100

Presenting Data

Intro to Statistics

Psychometrics


Neuropsychology


Single case design

Compliance (%)

E-Le

arn

ing

Pac

kage

Accessibility



16

Of all the categories “Usefulness and Satisfaction” had the highest level of No Data or

N/A ratings (mean = 39.28%; range= 25-50%) due to limited user feedback to inform

the ratings. The “Motivation to learn” item was not assessed by the feedback form and

so this could not be rated on any package. Only three packages prompted the learner to

complete a feedback form about the experience (Neuropsychology, Psychometrics,

HTRK), however all packages are able to be evaluated via the Minerva platform and so

this was rated as ‘partial’ on those who did not provide a specific prompt to users.

As shown in figure 6 the highest rated package was HTRK, achieving a ‘present’ score

of 75%. The lowest was Presenting data, which did not achieve a ‘present’ rating.

Overall ‘present’ ratings were rather low (mean = 39.29%; range = 0-75%). When

combining present and partial ratings the average compliance increased to 60.71%

(range = 50-75%).

Figure 6: Compliance to “Usefulness and Satisfaction” Category

0

25

25

50

50

50

75

50

25

25

0

25

25

0

50

50

50

50

25

25

25

0 10 20 30 40 50 60 70 80 90 100

Presenting Data


Intro to Statistics

Psychometrics

Single case design

Neuropsychology


Compliance (%)

E-Le

arn

ing

Pac

kage

Usefulness and Satisfaction



17

These results demonstrate the need to increase completion of feedback measures, as it

is unclear how satisfied users are with the e-learning packages.

Social Factors

None of the packages had a fully compliant ‘social factors’ rating. All were rated as

partially present due to the wider facility on the University of Leeds “Minerva” platform

to begin forum style discussions. This was not rated as present as there were no

discussions currently listed nor were there opportunities for interaction and feedback

from staff members or tutors (such as a Frequently Asked Questions thread). It was not

always clear who to contact for further information or support however, some credited

those who contributed to the course content or design.

Qualitative feedback

The user feedback questionnaire includes free text boxes where users are able to provide

qualitative information relating to three areas: Objectives, Assessment, and

Suggestions. These questions can be seen in Table 1, along with preliminary themes

and quotes to illustrate these.

Table 1: Qualitative Analysis of evaluation data

Question Preliminary Themes Illustrative Quote Objectives: What different or extra objectives do you think should be included in the course?

Unsure/No suggestions

Concern about increased time to complete

“I’m not really sure, they are

a good length and [to include extra objectives] would also add time.”

Assessment: Is there any way that the questions or quiz could be improved?

Value of feedback and explanation

Consolidates learning

“to give feedback if you get it

wrong, rather than having to go through it a few times until getting the question right.”


18

“I thought the quiz was a

good way to consolidate learning”

Do you have any suggestions about additional e-learning resources?

To support other teaching

Clinical Academic

Interaction

Social Varied

presentation of information

“E-learning on different therapy models or [clinical] presentations”

“Maybe one for quantitative and qualitative methods to support in making choices for thesis/SEP?”

“Maybe more links to videos that can talk through ideas.” “Maybe you could have an area where you could leave questions and get answers have discussions. I'm not sure if that is in the remit of e-learning, but the more interactive the better!”

Overall the feedback suggests that users value the e-learning and recognise its utility in

supporting learning; even raising suggestions of other topics where e-learning could be

useful. Users do appear to be cautious about adding additional aims or topics due to the

additional demand on trainee time to complete these. It is also suggested that users are

unsure as to the “remit” of e-learning and what can be offered within this.

The checklist includes both a “justification” and “Other Comments” section, where the

researcher noted observations regarding the e-learning to inform the rating of each

component. The completed checklist was shared with the commissioner to consider

alongside the ratings to allow easy identification of where errors were observed and so

these can be improved. The following are examples of this type of feedback:

Error prevention a) Can tasks be completed easily?


19

o “If stuck at a question which requires a correct answer, user cannot

navigate around to find the information or move on” – Literature

searching and Review

Is the course accessible across different devices/media/platforms?

o “Worked on iPad in Safari but not Chrome, which asked users to

download a different app. On iPad Safari when more information was

displayed on one screen the scroll feature did not work” –

Neuropsychology.

Discussion

Research has warned that when developing e-resources for learning it is not sufficient

to simply transfer paper-based resources to an online environment (Yelland et al.,

2008). The process of knowledge acquisition requires learners to be supported and

involved in the active learning process through the inclusion of pedagogical and ID

factors (Mehanna, 2004). It is important that the online environment is supportive,

engaging and provides opportunity to apply their knowledge without distraction

(Daskalakis & Tselios, 2013).

Aim 1: This project aimed to explore the pedagogical and ID features of the e-learning

packages developed for use on the DClinPsy training programme using a bespoke

evaluative checklist.

Overall compliance to the checklist was high across the e-learning packages. When

assessing ‘present’ rated factors there was variation as to how strongly these features

were included however, when combining ‘present’ and ‘partial’ items no package


20

achieved a score lower than 65%. This suggests that the e-learning resources are well

developed to include sufficient pedagogical and ID components to at least some extent

to promote successful learning, however this can be improved.

Of the checklist categories “Learning Techniques” was the most highly rated category,

with no score below 80% ‘present’. This category explored features such as; the

opportunity to use knowledge and receive feedback (Mehanna, 2004), explanation of

concepts and suitably pitched information (Tavangarian et al., 2004), and varied

presentation of the information. The results suggest that each of these pedagogical

features are sufficiently well included to promote learning.

System quality explored the ID features of the e-learning packages and the findings

suggest variability across these as to how well implemented these were. This assessed

how well users were able to navigate the system without disruption from design or

technological errors (Daskalakis & Tselios, 2013). Error prevention, organisation and

function were assessed, with errors observed within the packages. There were some

difficulties observed completing tasks or navigating the system which reduced

compliance to ‘partial’, as features were present but not always functioning as expected

(e.g. navigating menus).

The accessibility category assessed how well the packages allowed users to access

resources and adjust these to user preference. All packages were accessible through

web browsers but varied in how well these could be accessed across different devices

(e.g. tablet computer). Most did not require the user to download additional software to

access, however users were advised to use Google Chrome for best functionality. Most

adapted well to different screen sizes, although some (e.g. Neuropsychology) did not


21

adjust to a smaller display meaning the user had to scroll to view the information. No

package allowed the user to configure the colour or font of the display.

Aim 2: The project will explore user satisfaction using data from evaluative feedback

forms. This objective was not fully met due to limited evaluative data from trainees.

All categories were influenced by a limited amount of user feedback, however the

‘Usefulness and satisfaction’ category had the highest amount of missing data as a result

and so the findings should be considered with caution.

The available feedback was used to inform the ratings however these should be

interpreted with caution as they cannot be generalised to the wider trainee group. Of

the packages where feedback was available satisfaction was high; more data should be

gathered to explore this component.

It would be useful to explore the trainee perspective of the “Social Factors” category,

as no package had a ‘present’ rating for this. Not all packages provided details for

contacting the course tutor, however the wider VLE system “Minerva” provides a wiki

space for trainees to use freely. This means that there is the capacity to support such a

discussion forum already in place however it would seem that this feature is scarcely

used by trainees. Ituma (2011) suggests that learners perhaps use other methods of

communication, which could be the case with trainees.

Within the qualitative feedback there was a suggestion of an area where learners can

post questions and receive answers (perhaps from the appropriate tutor/moderator or

from other trainees). It may be beneficial to explore wider trainee views on this and

whether this would be a valued resource which trainees think they would use. It could

be useful to have this open to all trainees so that past discussions can be viewed my


22

current trainees and reduce the burden on moderators/tutors to ask similarly themed

questions each year.

Recommendations:

Identified errors in current packages to be fixed/improved.

E-Learning packages to present information in a variety of methods to engage

the user.

Users should receive feedback on performance in a timely manner.

o Incorrect quiz responses should prompt a tip or direct the user to the

correct information.

E-Learning resources should be checked annually by the course tutor or learning

technician to ensure that links and documents are still accessible.

User perspective of the e-learning packages should be further explored

o All packages should have a direct link to prompt users to complete the

evaluative feedback form.

o Feedback form to include a question regarding how useful trainees

found the e-learning.

o A further service evaluation project could explore user expectations and

experience of the e-learning systems (including the demand for a

‘chat/forum’ or frequently asked questions message board).

o Course staff perspectives could also be explored as information

disseminators regarding how successful they think inclusion of an e-

learning component has been (Paechter et al., 2010).

o Both staff and trainee perspectives gathered regarding potential topics

for additional resources.


23

Critical Evaluation

The project resulted in the development of a bespoke checklist for evaluating the

DClinPsy e-learning packages. This was devised based on the literature and

consideration of what was relevant for the project based on the aims of the

commissioner. However, this means that the checklist may not be applicable to other e-

learning evaluations as there may be components which are missing or not relevant to

other projects. It could be that as the scope of e-learning changes on the DClinPsy

course, and the way information is presented develops, the checklist may need updating

to reflect that and capture other pedagogical and instructional design techniques the

university wishes to employ.

A full quantitative analysis of user feedback was not possible due to limited numbers

of feedback forms to inform a reliable analysis. There were a limited number of

completed user feedback forms, with some packages having no feedback and others

with a range of 1-8 participants. This contributed to the high levels of no-data on some

packages and reduces the reliability of the comparison of compliance across packages.

Only three packages prompted the user to complete a feedback form evaluating the e-

learning experience. This may have contributed to the low levels of user feedback, as

users may not have been aware of the form or thought this to be necessary. Therefore,

it is a recommendation from this evaluation that a prompt and link is provided at the

end of every package to increase the data available for future evaluation.

Qualitative feedback was limited due to the lack of evaluative feedback and that there

were only three qualitative questions on the evaluation form. It was considered that

trainees could be sent a questionnaire to assess their attitude toward e-learning and their

experience of use, however this was not possible within the timescales and size of this


24

project. It may be worthwhile to qualitatively explore trainee perspectives of the use of

e-learning on the DClinPsy course via a further research project.

Conclusions

Pedagogical and ID features support learners to successfully develop knowledge and

understanding. The e-learning packages developed for use on the DClinPsy training

programme were evaluated using a bespoke checklist to assess the inclusion of

pedagogical and ID features which were identified as relevant for the aims and scope

of the e-learning packages. It was found that overall ‘present’ and ‘partial’ compliance

was high and that the ‘Learning Techniques’ category was well implemented,

suggesting good inclusion of pedagogical and ID features. There were errors and areas

for improvement identified within the packages which require development to better

support learning and user needs. Some areas where there were missing data would

benefit from exploring user perspective to assess features users find helpful for learning

(e.g. a discussion forum).

This project used a largely quantitative approach and it may be useful to supplement

these findings with a further qualitative exploration. More evaluation of user

perspective is needed to assess the usability and effectiveness of the training; whilst we

can assess the presence of pedagogical and ID features, the main priority of e-learning

is that that it meets the learning requirements of the trainees.


25

References

Borreson Caruso, J., & Salaway, G. (2007). The ECAR Study of Undergraduate Students and Information Technology, 2007. Retrieved from http://www.csplacement.com/downloads/ECAR-ITSkliisstudy.pdf

Bransford, J. D., Brown, A. L., & Cocking, R. R. (2000). How People Learn: Brain, Mind, Experience and School (Expanded Edition ed.). Washington, D.C.: National Research Council.

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101.

Daskalakis, S., & Tselios, N. (2013). Evaluating e-Learning Initiatives: A Literature Review on Methods and Research Frameworks. In Web-Based and Blended Educational Tools and Innovations: IGI Global.

Dorobat, I. (2014). Models for Measuring E-Learning Success in Universities: A Literature Review. Informatica Economică, 19(3), 77-90. doi:10.12948/issn14531305/18.3.2014.07

Gordon, N. (2014). Flexible Pedagogies: preparing for the future. Retrieved from Holsapple, C. W., & Lee-Post, A. (2006). Defining, Assessing, and Promoting E-Learning

Success: An Information Systems Perspective. Decision Sciences Journal of Innovative Education, 4(1), 67 - 85.

Islam, N. A. K. M. (2013). Investigating e-learning system usage outcomes in the university context. Computers & Education, 69, 387 - 399. doi:10.1016/j.compedu.2013.07.037

Ituma, A. (2011). An evaluation of students’ perceptions and engagement with e-learning components in a campus based university. Active Learning in Higher Education, 12(1), 57-68. doi:10.1177/1469787410387722

Lim, C. J., & Lee, S. (2007). Pedagogical Usability Checklist for ESL/EFL E-learning Websites. Journal of Convergence Information Technology, 2(3), 67 - 76.

Martín-Rodrígueza, Ó., Fernández-Molinab, J. C., Montero-Alonsoc, M. Á., & González-Gómezd, F. (2015). The main components of satisfaction with e-learning. Technology, Pedagogy and Education,, 24(2), 267 - 277. doi:10.1080/1475939X.2014.888370

Mehanna, W. N. (2004). e-Pedagogy: the pedagogies of e-learning. Alt-J Research in Learning Technology, 12(3), 279-293. doi:10.3402/rlt.v12i3.11259

Merrill, D. M., Drake, L., Lacy, M. J., & Pratt, J. (1966). Reclaiming Instructional Design. Educational Technology, 36(5), 5-7.

Mödritscher, F. (2006). The Impact of an E-Learning Strategy on Pedagogical Aspects. (30/09/2019). Retrieved from https://pdfs.semanticscholar.org/b6c1/9ca093d5187491be0afb4b5527e800e17abe.pdf

Oztekin, A., Kong, Z. J., & Uysal, O. (2010). UseLearn: A novel checklist and usability evaluation method for eLearning systems by criticality metric analysis. International Journal of Industrial Ergonomics, 40, 455 - 469. doi:10.1016/j.ergon.2010.04.001

Paechter, M., Maier, B., & Macher, D. (2010). Students’ expectations of, and experiences in e-learning: Their relation to learning achievements and course satisfaction. Computers & Education, 54, 222-229. doi:10.1016/j.compedu.2009.08.005

Ruiz, J. G., Mintzer, M. J., & Leipzig, R. M. (2006). The Impact of E-Learning in Medical Education. Academic Medicine, 81(3), 207 - 212.

Smith, S. D., & Borreson Caruso, J. (2010). The ECAR Study of Undergraduate Students and Information Technology, 2010. Retrieved from http://anitacrawley.net/Resources/Reports/ECAR%20study%20highlights.pdf

Tavangarian, D., Leypold, M. E., Nölting, K., Röser, M., & Voigt, D. (2004). Is e-Learning the Solution for Individual Learning? Electronic Journal of e-learning, 2(2), 273-280.

http://www.csplacement.com/downloads/ECAR-ITSkliisstudy.pdf

https://pdfs.semanticscholar.org/b6c1/9ca093d5187491be0afb4b5527e800e17abe.pdf

https://pdfs.semanticscholar.org/b6c1/9ca093d5187491be0afb4b5527e800e17abe.pdf

http://anitacrawley.net/Resources/Reports/ECAR%20study%20highlights.pdf


26

Yelland, N., Tsembas, S., & Hall, L. (2008). E learning: issues of pedagogy and practice in the information age. In P. Kell, W. Vialle, D. Konza, & G. Vogl (Eds.), Learning and the learner: exploring learning for new times: University of Wollongong.


27

Appendix:

Appendix 1: Bespoke Checklist used to evaluate the e-learning packages:


28

Appendix 2: DClinPsy E-Learning Feedback Evaluation Form


29


30

Documents

Evaluating the E-Learning packages on the …...3 (Bransford et al., 2000; Mödritscher, 2006). The e-learning system should have an intuitive design, where users can navigate, find,