View
2
Download
0
Category
Preview:
Citation preview
Internet‐basedlearningfortraininghealthcare
professionalsin‐service
DrGeoffWongProfessorTrishaGreenhalgh
QueenMary,UniversityofLondon
Internet‐based learning for training health care professionals in service
2
Internet‐based learning for training health care professionals in‐service
Authors Dr Geoff Wong, Senior Lecturer in Primary Care* Professor Trisha Greenhalgh, Professor of Primary Health Care and Dean for Research Impact *Corresponding author Author affiliations Centre for Primary Care and Public Health Queen Mary, University of London Contact details Yvonne Carter Building 58 Turner Street, London, E1 2AB United Kingdom Tel: +44 7973818122 Email: grckwong@gmail.com Funding This project was funded by the World Health Organization Acknowledgements We want to thank Dr Mediah Ahmed for designing, piloting, refining and executing the MEDLINE database search. In addition we wish to thank Professor Khalid Khan for his invaluable advice and help at various points with this project. Conflicts of interests We declare that we have no conflicts of interests.
Internet‐based learning for training health care professionals in service
3
Contents Executive summary............................................................................................................. 5 1. Introduction .................................................................................................................... 6 2. Aim and objectives.......................................................................................................... 8 3. Methods .......................................................................................................................... 9 3.1 Justification for using realist review ......................................................................... 9 3.2 Review processes .................................................................................................... 10
4. Results ........................................................................................................................... 13 4.1 Searching................................................................................................................. 13 4.2 Study characteristics ............................................................................................... 15 4.3 Programme theory for Internet‐based learning for HCPs in‐service...................... 26
5. Discussion...................................................................................................................... 45 5.1 Comparison with exiting literature......................................................................... 47 5.2 Strengths and limitations........................................................................................ 47
6. Recommendations ........................................................................................................ 50 7. Conclusions ................................................................................................................... 52 8. Appendices.................................................................................................................... 53 9. Reference List................................................................................................................ 56
Internet‐based learning for training health care professionals in service
4
List of figures
Figure 1: Outline of searching process and document flows ……………………………………. 14 Figure 2: Refined programme theory of Internet‐based learning …………………………….. 29
List of tables
Table 1: Characteristics of included studies …………………………………………………………….. 17 Table 2: Summary of the attributes of an innovation ………………………………………………. 27
List of boxes
Box 1: Important questions for course developers to ask and address when considering and designing an Internet‐based learning course…………………………………..
51
Internet‐based learning for training health care professionals in service
5
Executive summary Internet‐based learning is increasing being used to teach a wide range of topics to many different types of health care professionals. The literature indicates that Internet‐based learning can be just as efficacious as more traditional forms of education and training (e.g. such as face to face lectures or course). The greater challenge is knowing when to use it, for whom and for which types of learning goals – in preference to traditional forms of education. In other words, the knowledge gaps in Internet‐based learning are in implementation and no longer in proving efficacy. The realist review conducted to provide evidence for this report sets out to address these very questions on implementation – ‘How, why, for whom, to what extent, in what circumstances does Internet‐based learning work for health care professionals in‐service?’. It found that engaging learners, initially to get them to start a course and then to continue to completion is a far bigger challenge than that of course design and content. The data in the studies included in this review indicate that many Internet‐based learning course developers possess the necessary expertise to develop courses. Teaching principles and theories are often combined with interactivity and useful course content to produce courses capable of meeting their objectives. Few however consider the issue of engagement and as a result the experience limited uptake and/or large numbers of dropouts from their courses. This review finds that this does not necessarily have to be the case. A key finding is that courses need to be developed in such a way as to give learners good reasons to want to start a course and complete it. These reasons may be because the learner believes or feels that Internet‐based learning offers; access to learning; cost savings; convenience; ease of use; is accessible (in language and literacy terms); fits in with their competing pressures and is compatible with their practice and values. A complex interplay exists between the aforementioned contextual influences and the decision making processes within learners. Many of these same contexts also influence their decisions to complete a course. Evidence from the realist review has been used to create a decision support resource for those wishing to develop Internet‐based learning courses. The resource consists of a series of important questions for course developers to ask and address when considering implementing and designing an Internet‐based learning course. Each of the questions in the decision support resource links back to contextual influences that have been shown to influence learner engagement. Addressing each and everyone one of the questions in the decision support resource will not guarantee that a course will always be success. This is not possible as there is always a complex myriad of other different contexts to contend with – such as learner types, their needs, their training levels, location and so on. However, judiciously addressing each of these contexts is likely to result in a course that is more likely to be liked by learners and achieves its objectives.
Internet‐based learning for training health care professionals in service
6
1. Introduction
When something is such a creative medium as the web, the limits to it are our imagination.
Sir Tim Berners‐Lee
The use of the Internet to provide education for health care professionals is growing (1). This is reflected in both the sheer number of courses available and in the published literature (2). Rationales for wanting to use the Internet ranged from convenience (3‐5), increased access to learning (6‐10), to cost savings (3;11;12). When the Internet is used to provide education for health care professionals, educators have used a range of approaches. Early examples used email communication only, as technological advances occurred, elaborations included the addition of resources online (on web‐pages or as downloadable files), sound, video, slideshows, animation and additional means for communication (live chat rooms, bulletin boards and audio). In addition to using the Internet, some courses were combined with face to face teaching approaches, such as lectures and small group learning – often referred to as blended learning. This diversity of approaches in the use of the internet for educational purposes has resulted in an explosion of terms used to describe it – web‐based learning or course, Internet‐based learning or course, e‐learning, blended learning or course and webinars (13). The only commonality between these terms is that Internet technology is somehow used in the educational process – be it to provide learning materials, enable communication between learners and tutors and to provide feedback on performance. As such there is no consensus on the preferred term (14) and in this report our preferred term is internet‐based learning, predominately as it succinctly describes the educational approaches we are interested in. The increase in use of the Internet as a means of providing education and training for health care professionals has not been without its challenges. Initial concerns revolved around the questions of efficacy – can it work in the first place? Such a question resulted in many ‘proof of concepts’ studies that compared the use of the Internet against no educational interventions. Another area of concern was whether the Internet when used for educational purposes was at least as good as more traditional teaching formats. This concern led to a range of ‘media comparison’ studies. This growth in primary research has also been mirrored by an increase in the number of evidence syntheses undertaken. The systematic review and meta‐analysis by Cook et al., Internet‐Based Learning in the Health Professions: a meta‐analysis, deserves particular mention as it covers the learner group this report is concerned with (15). This rigorously conducted review covers studies from 1990 to 2007, their findings were that compared to no intervention Internet‐based learning was better and equivalent to traditional methods.
Internet‐based learning for training health care professionals in service
7
When compared to traditional methods, Internet‐based learning was “… neither inherently superior to nor inferior…”. The conclusion they come to is that the knowledge gaps are no longer concerned with whether Internet‐base learning works. We can be confident that it can do (15;16). Where there is greater uncertain is clearly summed up by the authors of this review “How can Internet‐based learning be effectively implemented?” and “When should Internet‐based learning be used?” (15). Like many educational interventions, the outcomes that can be achieved from Internet‐based learning are highly context dependent (17). These questions set out the challenge for those wanting to use the Internet in educating health care professionals and addressing them is the focus of this report. The challenge is to understand the contexts in which Internet‐based learning is most likely to produce its desired outcomes.
Internet‐based learning for training health care professionals in service
8
2. Aim and objectives The main aim of this report is to address the question: How, why, for whom, to what extent, in what circumstances does Internet‐based learning work for health care professionals in‐service? Using a realist review approach the literature on Internet‐based learning work for health care professionals (HCPs) in‐service was collated, summarised, analysed and synthesised. It is anticipated that the knowledge generated may feed into WHO (World Health Organization) guidance on the use of Internet‐based learning for this group of learners. The focus recommendations from the realist review are focussed more on the needs of the developers of Internet‐based courses than learners. The specific objectives for the realist review that informs this report are to:
a) Collate relevant studies b) Analyse and synthesis the current state of knowledge on this topic c) Provide recommendations on how internet‐based learning might be most
effectively used for HCPs in‐service
Internet‐based learning for training health care professionals in service
9
3. Methods
3.1 Justification for using realist review Having undertaken a review in the closely related field of the use of the Internet in medical education before (18), it was anticipated that the literature would most likely have the following features:
Internet‐based learning is used to refer to a broad range of instructional strategies
be quasi‐experimental (e.g. pre‐ / post‐ evaluations)
evaluate learner satisfaction or change in knowledge, attitudes and/or skills (as opposed to patient related outcomes)
employ different outcome measures
have a broad range of learners (i.e. different professions, seniority)
cover a broad range of topics
research will mainly come from richer nations
evaluations will be small scale in nature In summary, a significant amount of heterogeneity was anticipated such that it is unlikely that a meta‐analysis will yield consistent and meaningful findings. This has been demonstrated by Cook et al.’s review (15). Making sense of such literature is challenging, especially if one of the anticipated goals of the knowledge generated is to inform the development of guidance to aid the implementation of Internet‐based learning for HCPs in‐service. In effect this review needs to generate knowledge that addresses the very broad question, “How, why, for whom, to what extent, in what circumstances does Internet‐based learning work for HCPs in‐service?” The answers to such a question have been identified as where the knowledge gaps now lie (see above) (15). Past experiences (18) and the literature (19) indicate that the best methodological approach is to undertake a realist review. Such a literature review approach is much more likely to produce knowledge that will inform implementation (20). In brief, realist review (also known as realist synthesis) is an interpretive theory‐driven approach to literature reviews which applies realist philosophy to the synthesis of findings from primary studies that have a bearing on a single research question. It uses interpretive cross‐case comparison to understand and explain how and why observed outcomes have occurred in the studies included in a review. An overview of the realist review method can be found online at www.ramesesproject.org or in the article Realist methods in medical education research: what are they and what can they contribute? (19).
Internet‐based learning for training health care professionals in service
10
3.2 Review processes The central task of the realist review was to address the question “How, why, for whom, to what extent, in what circumstances does Internet‐based learning work for HCPs in‐service?” To do so a realist programme theory of Internet‐based learning was developed and refined. The programme theory is a ‘map’ of the sequence of steps needed to achieve the final desired outcome from Internet‐based learning interventions. For each step, a realist logic of analysis was applied, so as to explain how the (intermediate) outcome for each step was achieved in realist terms – i.e. what interactions between context and mechanism(s) led to that outcome. For each step in the sequence, the mechanism(s) that cause the outcome was sought out along with relevant triggering contexts. The initial programme theory was iteratively developed and refined from that proposed by Wong et al. (18).
Searching and inclusion Studies for inclusion into the realist review were identified in the following ways. Database searching: MEDLINE (via OVID) was searched from 2007 to 2013 inclusively and restricted to English language studies only. Only one search was undertaken. The search was designed, piloted and refined by an informaticist. The search strategy may be found in Appendices. Screening: This was undertaken by one individual (GW). A 10% subsample of the citations retrieved at each screening stage was independently checked by TG. Any disagreement was resolved by discussion. In the first screen all hits returned by the MEDLINE search were screened against the inclusion and exclusion criteria below. The focus of the review was on the use of Internet‐based learning for any HCP who were already working in a post (i.e. in‐service). The brief for this report specifically wished to exclude award bearing courses and also education and training for health care professional trainees. The inclusion criteria were deliberately broad:
The Internet was used for educational / teaching purposes
HCP (any type) in service were involved The exclusion criteria were:
No Internet was used (e.g. simulators, tele‐education, only CD‐ROMs, video‐conferencing)
The Internet was not used for teaching (e.g. surveys, monitoring, administration, clinical reminder systems, audience response system)
The course/programme was award bearing
The course included only HCP trainees / students
Internet‐based learning for training health care professionals in service
11
Internet‐based education was only used to teach/train/develop educators/faculty
There was no outcome data (e.g. only descriptions of a course/intervention/programme)
Where there was any doubt the full text was obtained and screened. In the second round of screening, attempts were made to obtain the full text of all studies and these were screened against inclusion and exclusion criteria.
Data extraction, analysis and synthesis The lead realist reviewer (GW) read all the included papers and data were used if they were relevant to the realist analysis – i.e. could inform some aspect of the programme theory. Data analysis and the inferences made were double‐checked by TG. Where necessary, data analyses and the plausibility of inferences were discussed between GW and TG. Assessments of rigour (how trustworthy were the data being used) were also made (21). To illustrate, how rigour was operationalised, if data had been generated using a questionnaire, then the trustworthiness of the data would be considered to be greater if the questionnaire had been previously tested and shown to be reliable and valid and had not been altered (or if alterations had been made subsequent testing had been undertaken). Realist reviews require a degree of theorising. Theorising is an interpretive process, requiring immersion in the data and abductive reasoning (to work out, for example, what mechanisms might have been in play). All the included studies were read and re‐read by GW looking for data that informed the development and/or refinement of the programme theory. The full texts of the included papers were uploaded into NVivo (a qualitative data analysis software tool) (22). Relevant sections of texts relating to contexts, mechanisms and their relationships to outcomes were coded in NVivo. This coding was both inductive (codes created to categorise data reported in included studies) and deductive (codes created in advance of data extraction and analysis as informed by the initial programme theory). The characteristics of the studies were extracted separately into an Excel spreadsheet (23). Each new element of data was used to refine the programme theory if appropriate, and as the programme theory was refined, included studies were re‐scrutinised to search for data relevant to the revised theory that may have been missed initially.
Data analysis used a realist logic of analysis to make sense of the initial programme theory. Interpretive cross‐case comparison was used to understand and explain how and why observed outcomes have occurred, for example, by comparing interventions where Internet‐based learning has been ‘successful’ against those which have not, to understand how context has influenced reported findings. During the review, an
Internet‐based learning for training health care professionals in service
12
important process was moving iteratively between the analysis of particular examples and refinement of programme theory.
Due to timescales, the breadth and depth as well as the processes of the realist review were truncated. The implications of the changes we made are discussed in more detail in Strengths and Limitations (Section 5.2) below. Where relevant this realist reviewed followed the quality standards set out for realist reviews by the RAMESES project (24).
Internet‐based learning for training health care professionals in service
13
4. Results
4.1 Searching The MEDLINE search returned 1159 documents. After the removal of 147 duplicates, there were 1012 studies screened in the first round. Of these 1012 studies, 91 appeared to meet inclusion and exclusion criteria. Screening by GW identified 90 potentially relevant studies. The independent screening by TG of a 10% random sample (of the 1012 studies) identified a further single study (25). Discussion between GW and TG occurred over the eligibility of one study and it was decided after discussion that it did not warrant inclusion as the Internet was not used. The full text was obtained for 90 of these 91 studies. We were unable to obtain the full text for one document (26). Round two, full text screening, identified 68 studies that fulfilled the inclusion criteria. Independent screening by TG of a 10% random sample at this stage did not generate any disagreements. These 68 studies were used to develop and refine the programme theory of Internet‐based learning for HCPs. Figure 1, outlines the searching process and document flows.
Internet‐based learning for training health care professionals in service
14
Figure 1: Outline of searching process and document flows
MEDLINE search Internet-based learning + health care professionals (HCPs) + outcome data but not including medical students 1159 Hits
Duplicates removed 1012 documents
First screen (Title, Abstract, MeSH) Inclusion criteria Internet used for education/training
purposes HCP (any type) in service involved 91 documents
Second screen (full text) 68 documents included in realist review
Duplicates 147 documents
Excluded Exclusion criteria: No Internet was used (e.g. simulators, tele-
education, only CD-ROMs, video-conferencing)
The Internet was not used for teaching (e.g. surveys, monitoring, administration, clinical reminder systems, audience response system)
The course/programme was award bearing The course included only HCP trainees /
students Internet-based education was only used to
teach/train/develop educators/faculty There were no outcome (e.g. description of
a course/intervention/programme) 921 documents
Excluded Did not meet exclusion criteria 22 documents Unable to obtain full text 1 document
Internet‐based learning for training health care professionals in service
15
4.2 Study characteristics All 68 studies were in English (as expected due to the filter we used in our search strategy). In terms of year of publication, five were published in 2007, five in 2008, seven in 2009, eight in 2010, 20 in 2011, 19 in 2012 and four in 2013. The majority of the studies evaluated Internet‐based learning courses that were provided to HCPs in richer nations. The breakdown by country is as follows (% in brackets): USA 30 (45.6); Canada 10 (14.7); Multi‐national 8 (11.8); United Kingdom 7 (10.3); Brazil 1 (1.5); China 1 (1.5); Finland 1 (1.5); France 1 (1.5); Italy 1 (1.5); Japan 1 (1.5); New Zealand 1 (1.5); Peru 1 (1.5); Spain 1 (1.5); Uruguay 1 (1.5); US Affiliated Pacific Islands 1 (1.5); No data 1 (1.5). The number of learners participating in the Internet‐based learning courses was broad and ranged from 12 to 20061. The age of the learners was also broad, ranging from 20 years of age to over 70. The gender balance between learners varied form course to course. Some courses had a preponderance of females, whilst in others the gender balance was more equal. Where there were more female learners, this reflected the gender make up of the health care profession (e.g. more female learners in courses directed at nursing HCPs). A wide range of HCPs participated in Internet‐based learning courses – including dentists, dieticians, health care managers, midwives, nurses, occupational therapists, pharmacists, physical therapists, physicians, psychologists, speech and language therapists, social workers and many others. Within each HCP group, the learners ranged from trainees to sub‐specialists. The exact mix of HCP learners varied depending on the course topic area. Topic areas for the Internet‐based learning courses covered the full scope of clinical and in some cases managerial practice. In essence the Internet was being used to provide education on a broad range of topics. When the Internet was used, the course designers used it in differing ways. Thirty five (51.5%) courses used a website alone (often with multimedia content) to deliver the course. The Internet was also used to support traditional learning methods – blended learning. Other technologies used in the included studies include virtual learning environments. These are specially developed software, delivered (mainly) via a web‐browser that tries to simulate traditional face to face learning environments. Learners are given access to (for example) tests, homework, course materials and also opportunities to interact with each other and tutors via email or bulletin boards. Webinars constitute another use of the Internet reported in the included studies. In these the Internet is used to deliver a lecture, either live or recorded for viewing later. During live webinars, participants are often able to ask the questions. The breakdown of how the remainder of the included studies used the Internet for learning are as follows (% in brackets) – Webinars alone, live or recorded for viewing later 9 (13.2), Blended
Internet‐based learning for training health care professionals in service
16
learning 8 (11.8); Virtual learning environments 4 (5.9); Multiple technologies including the Internet 2 (2.9); Website and email 2 (2.9); Website with discussion board 2 (2.9); Webinars with another technology 2 (2.9); Video‐conferencing using the Internet 1 (1.5); Email alerts 1 (1.5); No data 2 (2.9). Internet‐based learning enables the learner to interact in different ways. At one level, learners can interact with the contents in a website (e.g. answer a question and get automated feedback on the answer). The technology used may also enable learners to interact with other learners and/or a tutor or lecturer (e.g. via email, a chat room or discussion board). Within the included studies levels of interaction were classified as; learner – content; learner – learner or; learner – tutor respectively. Some courses enabled more than one form of interaction. Thirty seven (54.4%) of courses enabled only learner – learner interaction only. Interaction between the learner – content, learner – learner and learner – tutor was enabled in 21 (30.9%) courses. Learner – content and learner – tutor interactions were possible in six (8.8%) courses. One course (1.5%) enabled learner to learner interactions only. There was no data available for three (4.4%) courses. The study designs used in the evaluation of the Internet‐based learning courses were broad and main reflected the research question being asked. Where the purpose of the evaluation was to demonstrate efficacy (i.e. can Internet‐based learning work for a specific HCP group on a particular topic area) either a randomised trial approach or a quasi‐experimental ‘pre / post’ course design was used. Overall there were 24 (35.8%) pre / post studies and five (7.5%) randomised controlled trials. The remainder of the evaluation designs ranged from qualitative interviews to post course evaluations on a range of outcomes. The level of change reported within included studies at which an Internet‐based learning course impacted was classified using a modified version of Kirkpatrick’s model. In brief, in this model; Level 1 evaluative data measures changes on learner satisfaction; Level 2 learning outcomes; Level 3 performance improvement in practice and; Level 4 patient or health outcomes (for more details see Appendices). Studies often measured more than one level of change. Using this system of classification there were (% in brackets); 46 (67.6%) Level 1 studies; 9 (13.2%) where Level 2 was self reported and 33 (48.5%) where change was measured; 4 (5.9%) where Level 3 was self reported and 3 (4.4%) where change was measured and 2 (2.9%) at Level 4. Thus the majority of studies evaluated the effects of Internet‐based learning courses on satisfaction and measured changes in knowledge. When classified by the direction of the outcome(s) reported within the included studies, there were 48 (70.6%) positive studies, followed by 14 (20.6%) studies with mixed outcomes, one (1.5%) negative study and five (7.4%) where such a classification was not applicable because of the type of the evaluation method used. Table 1 summarises the characteristics of the included studies.
Internet‐based learning for training health care professionals in service
17
Table 1: Characteristics of included studies
Document name Year
Country of study
No. of learners Age data
Gender breakdown
Professional mix Topic area Interaction
Use of Internet Study design Findings Outcome measures
Kirkpatrick level Summary of results
Alemagno et al. (27) 2010 USA 256 20 to 59 No data Nurses
Hand hygiene knowledge and
compliance Learner-content Website
Pre / post knowledge test. Post intervention evaluation. Positive
Knowledge and self reported change in behavior
1 & 2 (self reported)
Knowledge increased after intervention. Self reported improvement in hand hygiene awareness and practice.
Andreazzi et al. (28) 2011 Brazil 21
mean 37.1 (range 24 to 72) No data
Physicians (25%),
microbiologists (75%) Microbiology Learner - learner
Website and email
Pre / post knowledge tests. Site visit and student feedback. Positive
Self reported satisfaction, job outcome, knowledge test, observed practice 1 & 2
Liked course, would recommend it and felt improved their job prospects.
Arnella et al. (29) 2010
Global provider 20061 No data No data
Nurses (66%), physicians
(34%) Palliative care Learner - content Website Post course evaluation Positive
Self reported learning, change in practice, recommend to others and free text comments 1
High satisfaction with course and knowledge gains.
Atack et al. (30) 2012 Canada 72
25 to 61 (mean 42) Female 87%
Broad range HCPs (see
Table 3 p109) Surge capacity
Learner - content Learner - learner Learner - tutor
Blended learning
Pre / post course competency survey and post course satisfaction survey. Positive
Surge management competency and satisfaction 1 & 2
Course increase surge management competency and overall satisfaction was high for the course on a range of measures.
Bachman et al. (31) 2008 USA 19 No data No data School nurses
Diabetes in school children
Learner - content Learner - learner Learner - tutor
VLE (blackboard)
Formative end of course evaluation. Mixed
Value of course along four dimensions 1
The course was felt to be useful for learning the topic. Mixed evaluations came from preferences for online vs face to face learning.
Banks et al. (32) 2011 UK 1094 No data No data
Newly qualified nurses
Help nurses transition from
student to qualified practitioner
Learner - content Learner - learner
Website with online forum
Telephone interviews plus online survey Mixed
Questionnaire with multiple questions on a range of topics 1
133 newly qualified nurses provided free text comments. Mixed responses on a range of measures.
Battaglia et al. (33) 2012 USA 122 Broad Broad
Pharmacists and pharmacy
students Medicines therapy
management Learner - content Website Pre / post questionnaires Positive
Theory of planned behaviour changes (attitudes, controls norms) and knowledge change 1 & 2 Changes in knowledge but not attitudes.
Beynat et al. (34) 2012
France (also users
from Canada, Morocco
and others) 137 26 to 42 Female 46% Physician
residents 75%
Management of diabetic
retinopathy Learner - content Website Pre / post knowledge test Positive Change in knowledge 1 & 2
Elearning more suitable when practice is close to teaching method - i.e. clinicians look at photos of retinas in real practice and the intervention can provide a similar learning opportunity.
Blazer et al. (35) 2012 USA 96 No data
Female (86 and 89%)
Doctors, genetic counsellors and
advanced practice nurses
Genetic cancer risk assessment
Learner - content Learner - learner Webinar
Quasi-experimental study and nested qualitative component comparing pre / post knowledge of web based conferencing vs. face to face Positive
Knowledge and reflective learning worksheet 2
Only evaluated the face to face vs web conference component of this course. Results was that web conferencing was at least as good as if not better (on some measures) as face to face
Internet‐based learning for training health care professionals in service
18
Blum et al. (36) 2012 USA 222 Mean 42 Female 93% Nurses
Venous thromboembolism
(VTE) Learner - content Website Pre / post knowledge test Mixed Knowledge test of VTE 2
Knowledge changed pre vs. post but not sustained at 6 month follow up knowledge test
Boehmer et al. (37) 2011 USA 89 Broad
Female (48 to 52%)
Physician (68-69%), nurse (5
to 10%) Herbal and natural products (H/NP) Learner - content Website Pre course questionnaire. Positive
Knowledge, attitudes, and experiences with H/NP
1 & 2 (self reported)
Increased knowledge and confidence about herbal and natural products.
Bollinger et al. (38) 2011
Low-middle income
countries N/A N/A N/A N/A Range of topics -
HIV
Learner - content Learner - learner Learner - tutor
Multiple technologies
Not a study but a 'lessons learnt' overview of how first few years of their course has run. N/A N/A N/A
An overview article. Mostly 'good news stories'.
Booker et al. (39) 2011 USA 57 No data No data
No data other than school health staff
Student child health
Learner - content Learner - learner Learner - tutor Webinar
Initial visit with training via telehealth. Follow up visit from training team as requested Positive
Delivery of one of three health interventions - measured by various instruments and chart reviews 4
Changes noted in practice (as seen in chart reviews) post intervention in (for example) obesity process measures,
Borgerson et al. (6) 2012
USA, Canada, Australia,
New Zealand
and Brazil 160 Broad No data
Clinical research
professionals (25% nurses)
Adverse event reporting in
pediatric oncology trials
Learner - content Learner - learner Learner - tutor Webinar
Post course evaluation at end and 4 to 6 weeks later using questionnaires Positive
Satisfaction and value questionnaires
3 (self reported)
Participants like it. It was on a topic many struggled with. Self reported change in adverse event reporting.
Bussieres et al. (40) 2011 France 50 No data No data
Physicians, residents, nurses,
pharmacists, pharmacy
students or technicians
Identifying paediatric
medications Learner - content Website Convenience sample of 50 groups of HCPs. Positive
Time taken to identify drugs and questionnaire on usefulness 1 & 2
Found the test useful. Like to be able to compare to others. Suggested it should become part of their training.
Buxton et al. (41) 2012 USA 97 No data No data Pharmacists
Pharmacy Continuing
professional development
Learner - content Learner - learner Learner - tutor Webinar
Online data collection. Pre course demographic details. Post course questionnaire. Mixed
Satisfaction and reasons for attending webinars 1
Mainly positive evaluations. Access and convenience were important contextual influences.
Buxton et al. (42) 2013 USA 36 No data No data Pharmacists
Pharmacy - drug diversion
Learner - content Learner - learner Learner - tutor Webinar Post course evaluation Mixed
Satisfaction and reasons for wanting to use live vs. webcast 1
No differences in preferences between live and webcast. Authors suggest that learners select the format they are happy with.
Bynum et al. (43) 2010 USA 45,996 No data Female 77%
Physicians, nurses , allied
health professionals,
dental professionals,
pharmacists and manager.
A broad mixed determined by a range of needs
assessment
Learner - content Learner - learner Learner - tutor Webinar Post course evaluation Positive Satisfaction 1 High satisfaction with course reported.
Canchihuaman et al. (7) 2011 Peru 619
Average - 41.5 (physicians) 38 (midwives)
Physicians 72,1% male,
midwives 84.7% female
Physicians and midwives
Sexually Transmitted
Diseases Learner - content
Learner - tutor Website and
RCT - with pre / post knowledge test immediate and 4 months later. Self reported changes in practice - post intervention. Mixed
Feasibility, acceptability, knowledge and self reported changes in practice 1 & 2
Changes in knowledge sustained but significantly less at 4 weeks, but no change in self reported practice
Internet‐based learning for training health care professionals in service
19
Carter et al. (44) 2009 Canada 46 Mean 43 Female 98% Nurses Stroke
Learner - content Learner - learner Learner - tutor
Multiple technologies
Pre / post collection of confidence and perception of competence data. Satisfaction - post course online survey and four focus groups Positive
Confidence using technology, perception of competence and satisfaction 1 & 2
Nurses' confidence in using elearning and self reported competence increased after using course. Satisfaction with course ws generally high.
Casebeer et al. (45) 2008 USA 5261
53 vs. 51 (completed vs non - average)
73 vs 79% male
(completed vs non)
Physicians 26% (primary care)
Range of topics from 48 continuing medical education
modules Learner - content Website
Random sample of learners given a post course evaluation using vignettes to test knowledge. Positive Change in knowledge 2
Those who completed module had greater knowledge vs. those who did not.
Casebeer et al. (46) 2010 USA 17142
50.6 vs 50.8 (completed vs. non - average)
69 vs 72% male
(completed vs non) No data
Range of topics from 114
continuing medical education modules Learner - content Website
Random sample of learners given a post course evaluation using vignettes to test knowledge. Positive Change in knowledge 2
Those who completed module had greater knowledge vs. those who did not. Evidence of 'additional' benefit of "interactivity"
Chen et al. (3) 2007
US Affiliated Pacific Islands (USAPI) 59 No data
Female 64.4%
Nurses (47.5%), doctors (25.4%),
others (only a few of each -
dentists, teachers)
Diabetes and Oral Health and Metabolic Syndrome
Learner - content Learner - learner Learner - tutor Webinar
Pilot study, over 2 days of lectures. Pre and post course knowledge test (identical). Written post course evaluation form including demographic data. Verbal feedback at live sessions recorded and transcribed Positive
Change in knowledge, verbal feedback and other evaluation data 1 & 2
Participants’' knowledge improved post lectures. Generally satisfied with lectures. Liked interactivity and for recorded lectures being able to review material in their own time.
Chhabra et al. (8) 2013 International No data No data No data No data Spinal cord injury Learner - content Website
Description of elearning resource N/A N/A N/A
Describes in detail development of the e-learning resource and also that it is based on e-learning principles. Dedicated team to develop resource with input from educationalists and psychologists.
Chung et al. (47) 2012 International 369 No data No data
Physicians (41%), nurses
(22%), midlevel practitioners
(20%), midwives (5%),
pharmacists (5%), and other
HCWs (7%) HIV Learner - content
learner - tutor
Webinar and Catalyst
VLE
Pre / post course (immediate and at 4 months) assessment of HIV knowledge and course evaluation. Mixed HIV knowledge 1 & 2
Just over half of students improved their score on the post-course assessment. Satisfaction was generally high.
Comer et al. (48) 2011 USA 177 No data No data
Nurses, physician assistants, residents,
fellows, and attending physicians
Central line–associated
bloodstream infections (CLABSIs) Learner - content Website
Pre / post course (immediate and at 3 to 4 months) - CVCKAQ. Positive
CVC Knowledge and Attitude Questionnaire (CVCKAQ). Piloted but not formally tested 2
Significant increases in CVCKAQ on all 5 sites. Knowledge retained at 3 to 4 months but lower than immediate post test.
Internet‐based learning for training health care professionals in service
20
Curran et al. (49) 2007 Canada 187 No data No data Nurses (78%)
Paediatric emergency medicine
Learner - content Learner - tutor Website Network analysis Mixed
Knowledge-seeking, sharing, exchange and contextual influences on group participation N/A
Emergency Department HCPs do use a forum to share and seek information. Number of postings were low. Unclear why.
De Rivas et al. (50) 2010 Spain 266 No data No data
Primary care physicians
Blood Pressure control in patients
at high risk of heart failure Learner - content Website
Pre / post performance data - physicians collected data form 10 consecutive eligible patients. Positive Blood pressure control 4
Cardiovascular risk stratification by physicians was poor. BP did drop significantly but by only 1.1 mmHg.
Desai et al. (51) 2013 No data 232 No data No data
Nephrologists (67%) and other
HCPs (33%) Nephrology Learner - content Website Post course satisfaction survey. Positive Satisfaction 1
Overall satisfaction - YouTube channel highly accurate, current and objective.
Fleet et al. (52) 2011 Canada 125 No data female 50.8%
Family physicians
(54.7%), other MDs (11.8%),
Residents 8.8%, nurses (5.5%) +
others Asthma Learner - content Website
Pre / post knowledge assessment and post course satisfaction survey. Positive
Knowledge and satisfaction 1 & 2
Post course participants' knowledge significantly increased. Satisfaction with course was high.
Gerkin et al. (4) 2009 USA 16 No data No data Nurses
Kidney transplant care Learner - content Website
Post course evaluation of outcomes, educational practices, faculty support, learner support, and the use of technology. Positive
Web-based course outcome assessment framework 1 & 2
Nurses liked course and reported it helped them in their work.
Gill (53) 2007 UK No data 36 to 55 No data Nurses
Accessing information on
databases No data VLE
(webCT)
Post course questionnaire. Convenience sub-sample interviewed by phone. Positive
Knowledge, skills, clinical effectiveness and patient care (self reported)
1, 2 and 3 (self reported
2 & 3)
Overall satisfaction (course met expectations of students) and also self reported changes in knowledge and practice.
Harrington et al. (54) 2009 USA 59
53.77 mean (27-79)
Female 81.4%
Owners, operators, and staff of small
residential care facilities for the
elderly Fire safety Learner - content Website
RCT - with pre and post intervention test of knowledge, attitudes and self reported practice. Mixed
Knowledge, attitudes and self reported practice
1, 2 and 3 (self
reported)
Intervention group showed significant changes in knowledge and self reported practice. No changes in attitudes. Participants very positive re the intervention
Harris et al. (55) 2011 UK 451
46.3 mean (21-72)
Female 42.8% Dentists Child protection Learner - content Website
Anonymous questionnaire survey after distribution of resource. Random sample of 1000 dental practices. Mixed
Received booklet, used it, found it useful and self reported change in practice
1 and 3 (self reported)
63.4% remember receiving booklet / website and of this group 92.5% recall using it. Increase in self reported knowledge, changes in attitudes and practice.
Hon et al. (56) 2008 Canada 117 No data No data
Nurses, care aides, and allied
health
Personal Protective
Equipment (PPE) Learner - content Website
Pre / post intervention observation of HCPs selection of personal protective equipment. Positive PPE selection 3 (observed)
Significant improvements in selection and sequencing of PPE and also In hand washing.
Internet‐based learning for training health care professionals in service
21
Iley et al. (57) 2010 UK 29 No data No data Nurses Cardiac care
Learner - content Learner - learner Learner - tutor
Blended learning
Knowledge test at end of onsite induction. Assessment of cardiac competency in practice by mentor. Course evaluation with mixed methods (focus groups and mentors interviews). Mixed N/A
1,2 and 3 (observed)
Learners were happy with the course and felt they learnt a lot and it gave them confidence. Mentors noted increase in competence. Issues around need for more interactivity and some intimidation from seeing how much some knew. Being on a online course can make it harder for some learners to find protected time.
Irvine et al. (58) 2012 USA 172 21 to 65
Female approx 85%
Nurses, occupational therapists,
social workers and physical
therapists
Working with residents in long
term care facilities with mental illness Learner - content Website
RCT - pre and post course assessment. Positive
Changes in a) behaviour and communication, b) knowledge c) attitudes, self efficacy, empathy and stigmatizing beliefs, d) acceptability 1 & 2
Positive changes on most outcome measures and sustained for up to 8 weeks.
Karpa et al. (59) 2012 USA 226 No data
No data (but differences in response
rate between M vs Female
reported) Nurses (61%) Herbal and natural products (H/NP) Learner - content Website
Web-based resources made available and advertised for 1 year. At the end of the year, HCPs asked them to take part in a survey. Positive
Use, satisfaction, self reported change in knowledge
1 & 2 (self reported)
Satisfaction with online resource, self reported increase in confidence and knowledge of H/NP. Most would recommend resources to others.
Kettles et al. (60) 2008 UK 387 No data See Table 1
Broad mix of mental health
care professionals
The Mental Health (Care and Treatment)
(Scotland) Act 2003 Learner - content Website
Mixed methods evaluation: 1) Questionnaire survey 2) focus groups and 3) telephone interviews. Mixed
Recognise resource, use it, satisfaction, opinions about format preferences
1 & 2 (self reported)
Resource was recognised and used by some. No clear preferences for different format. General satisfaction expressed about the quality and value of the learning resources. Some self reported change in practice.
Kobak et al. (61) 2007 USA 12 Mean 26
Female 83.3% Researchers
Administration of the Positive and
Negative Symptom Scale (PANSS)
Learner - content Learner - tutor
Video-conferencing
via the internet
Pre / post intervention assessment of PANSS rating competency by blinded rater. Positive
Competence in using PANSS
1,2 and 3 (observed)
Increased knowledge of PANSS rating scale and observed improvement in rating competence. Generally satisfied with training package on offer.
Kobak et al. (62) 2011 USA 30 No data Female 96.6
Broad mix of HCPs involved
in research.
Screening Tool for Autism in Toddlers
and Young Children (STAT) Learner - content Website
Pre / post intervention knowledge test. Positive
Knowledge and satisfaction 1 & 2
Knowledge improved after intervention and was comparable to that of the face to face group. Generally satisfaction on a range of dimensions were high.
Lauder et al. (63) 2011 UK 98 No data No data
Newly qualified nurses
Help nurses transition from
student to qualified practitioner
Learner - content Learner - learner
Website with online forum
Mixed methods: 1) Gricean analysis of on-line communication and 2) qualitative analysis of communication content. Mixed
Gricean rating (quantity, quality, relevance, manner) and qualitative analysis N/A
Little use of asynchronous online discussion as expected. Issues with nurses having to find time to use Flying Start - duplication of work.
Lefebvre et al. (64) 2007 Canada 17 Broad No data
Physiotherapy (2), psychology
(1), nursing (13), and social
work (1) Traumatic brain
injury (TBI) No data No data
Qualitative study - interviews before and after training. N/A N/A
1 & 2 (self reported)
Learners had better understanding of how the TBI affected the family and develop reflective practice.
Internet‐based learning for training health care professionals in service
22
Legris et al. (65) 2011 Canada 78 No data
Intervention Female
73%,control 61% Pharmacists
Chronic Kidney Disease Learner - content Website
RCT - Pre and post intervention knowledge and skills tests. Telephone interviews with 8 pharmacists. Satisfaction data gathered with online questionnaire. Positive
Knowledge, skills, satisfaction and opinions of elearning 1 & 2
Pharmacists skill and knowledge improved after intervention. Satisfaction was high. Clear request for opportunity to ask experts about more difficult cases.
Leikola et al. (66) 2009 Finland 42 No data No data Pharmacists
Comprehensive medication reviews
(CMR)
Learner - content Learner - learner Learner - tutor
VLE (moodle)
Continuous assessment on course by tutors - assignments, learning diaries and e-portfolio. Positive
Theoretical and practical skills to be able to provide CMR services 1 & 2
Overall positive about course and what they learnt. Almost all said they felt more able to do CMRs. Strong desire for interaction.
Lewin et al. (67) 2012
International (81% USA) 18522 Broad No data Broad mix Breastfeeding Learner - content Website
Mixed methods evaluation: 1) expert opinion of intervention 2) knowledge acquisition by users (pre / post intervention) 3) faculty opinion on value of course Positive
Competencies matched, knowledge gain, faculty opinion 2
BreastfeedingBasucs resource of high quality. Learners using it gained knowledge. Faculty generally liked it.
Llambi et al. (68) 2011 Uruguay 335 No data No data
Physicians (81.2%), nurses
(13.4%), psychologists
and social workers(5.4%) Smoking cessation
Learner - content Learner - learner Learner - tutor
Blended learning
Pre / post course knowledge tests and commitments to change (CTC) statements Positive
Commitments to change and knowledge 2
Learners completing course increased knowledge. Most took part in online activities and provided CTC statements. Most reported they implemented or partially implemented CTCs.
Macdonald et al. (5) 2008 Canada 59 Broad
Female 59.3%
Physicians, nurses, nurse practitioners,
and pharmacists
enhance collaborative
practice Learner - content Website
Pre / post course questionnaires. Interviews at end of project with learners and managers. Mixed
Satisfaction, changes in - knowledge, attitudes, skills care delivery. Transfer of learning to workplace. Improvements in residents' well-being
1, 2 and 3 (self
reported)
On the whole the learners like the resource. Especially as it gave some access to learning they would otherwise not be able to attend. Self reported changes in most of the outcome measures.
Marsh-Tootle et al. (69) 2010 USA 136 No data
Female 52.8%
(intervention) 57.1%
(control) Primary care physicians
Strabismus and amblyopia Learner - content Website
RCT - clustered. Pre / post intervention knowledge test points. Positive
Knowledge and self reported increase in use of preschool vision screening and diagnosis of strabismus and ampblyopia
1 & 2 (self reported)
In the intervention arm, knowledge increased as did self reported change in practice. Both levels fell after delay, but remained higher than baseline or control group
Mazzoleni et al. (70) 2012 Italy 2876 No data No data
Physicians, nurses,
technicians, therapists and psychologists
Neurological bladder, obese
patient handling, respiratory
pathologies, clinical trials
Learner - content Learner - tutor
VLE (moodle)
One year evaluation of use - using online data collection instruments. Positive
Attitudes, use and costs N/A
High usage by staff (especially nurses), authors report that the courses are cheaper to run than face to face for the amount of educational credits awarded.
Internet‐based learning for training health care professionals in service
23
Mitchell et al. (11) 2011 USA 13 No data
Female 84.6%
Family physicians
Motivational interviewing (MI)
Learner - content Learner - learner Learner - tutor
Blended learning
Post course evaluation with questionnaire and motivational interviewing skills proficiency measured pre and post intervention - blinded, independent coder. Positive
Self reported changes in confidence and clinical practice, satisfaction and acceptability 1 & 2
Participants were skilled internet users. Not all though Second Life was better than face to face for learning. After course self reported increase in confidence and ability. Blinded rating showed increase in proficiency after course.
O'Connor et al. (9) 2011 USA 3456 Broad
See Table 2 page 424 (mostly female)
Midwife, nurse practitioner, physician
assistant and physicians Breastfeeding Learner - content Website
Pre / post course evaluation of knowledge - using 10 years worth of usage and knowledge score data. Three versions of knowledge test in that time. Positive Change in knowledge 2
Knowledge scores increased post intervention.
Olson et al. (71) 2011 USA 24 No data No data
Family physicians
Tobacco cessation practice
Learner - content Learner - learner Learner - tutor
Blended learning
Post course evaluation with telephone interviews. N/A
Oklahoma Sooner Care Health Management Program Provider Incentive Plan audits 1 and 2
A range of findings that throw light on the processes that are involved implementing practice guidelines.
Palmer et al. (72) 2011 USA 132 35 to 55
Female (89.9%)
Nurses (32%), physicians
(20.4%) and nurse
practitioners (14.4%). Breast cancer Learner - content Website
Pre / post course knowledge tests. Positive
Knowledge and satisfaction 1 and 2
Overall high satisfaction and on seven of the knowledge areas tested there was significant change post course.
Pluye et al. (73) 2012 Canada 127
49.6 average
Female 49.6% Physicians
Use of evidence from Cochrane
reviews Learner - content Email alerts
Longitudinal evaluation study. Courriels Cochrane sent weekly and rated by readers using the Information Assessment Method (IAM) Positive IAM domains
1 & 2 (self reported)
Physicians read and rated the Courriels Cochrane. They reported that the informational content was valuable.
Sherman et al.(25) 2012 USA 68
33.7 average
Female 82.9% Nurses
Critical care pharmacology
Learner - content Learner - learner Learner - tutor
Blended learning
Mixed methods evaluation: RCT - Pre and post intervention knowledge tests. Post intervention focused group to discuss satisfaction Positive
Change in knowledge Learner satisfaction 1 and 2
No differences in knowledge and satisfaction scores between blended learning and lectures.
Sweet et al. (10) 2009 USA 29 No data No Pharmacists
Diabetes - new medications Learner - content Website
Pre / post intervention knowledge tests. Post course evaluation Positive
Knowledge and satisfaction 1 and 2
Post course increase in knowledge and generally high satisfaction with course.
Tian et al. (74) 2007 China approx 900+
30 (mean - villagers)
Female 66% (villagers)
Villagers and students
Public's knowledge of sexual health
issues Learner - content
Learner - tutor Website Pre / post evaluation with questionnaires. Positive
Sexual health knowledge 2
Health workers, villagers and students in Nanhua experienced the greatest increase in knowledge of HIV/AIDS and sexually transmitted infections, and Dayao the lowest. The improved knowledge was substantial on some topics (e.g. hepatitis B transmission).
Internet‐based learning for training health care professionals in service
24
Toledo et al. (75) 2012 USA 52 No data No data
Anesthetists (46%),
Obstetrician (25%) Nurse
(29%) Postpartum
haemorrhage Learner - content Website
Pre / post intervention measure of knowledge using 5 blood loss simulation stations. Negative Blood loss estimation 2
Decay in knowledge of almost all participants.
Tsugihashi et al. (76) 2011 Japan 176 Broad Female 46%
Doctor (35%), pharmacist
(24%, Nurse (13%),
physiotherapist (5%) dietician
(5%), engineer(5%) Clinical research
Learner - content Learner - learner Learner - tutor Webinar
Post course knowledge test and measure of attendance and in lecture counting of interaction. Positive
Lecture attendance rate, interaction in lectures, end of course examination scores 2
Course was well attended, participants used the interactive features regularly, no knowledge differences between professional groups.
Valaitis et al. (77) 2007 Canada 36 No data No data
Medical residents (39%), nursing students
(31%), health sciences faculty (25%) and staff
(3%) Web conferencing
Learner - content Learner - learner Learner - tutor Webinar
Q methodology used to identify unique and common viewpoints of participants with exposure to Web conferencing. N/A Q sort scores analysis 1
Learners classified in to a range of typologies - pragmatists, positive communicators and shy enthusiasts. These groups on the whole had preferences that were specific to them though there was some cross-over.
Varjavand et al. (78) 2012 USA 36
55 (median, range 38-66) Female 19% Physicians
Physician Reentry / Refresher course
Learner - content Learner - learner Learner - tutor
Blended learning
Mixed methods evaluation: 1) Daily formative assessment and feedback; 2) 360 evaluation of professionalism; 3) Assessments of Web-based exercises; 4) standardized patient examinations; 5) Pre / post course examination. Positive Examination results
1,2 and 3 (observed)
Demonstrated change in knowledge, most reported goal gained (80%+), high satisfaction with course.
Wallen et al. (79) 2011 USA 129 No data Female 91%
Nurses and Allied Health Professionals Genomics
Learner - content Learner - learner Learner - tutor
Blended learning
Pre / post knowledge test and post course evaluation. Positive
Knowledge and WEBLEI 1 and 2
Statistically significant change in knowledge.
Walsh et al. (80) 2009 UK 449 No data No data
Family Physicians Consultation styles Learner - content Website
Thematic analysis of free text feedback to the resource Positive Perceptions 1
Of feedback received the majority gave positive feedback. Many users liked the case histories and commented that these gave them a framework around which to base their reflections.
Walters et al. (81) 2012
New Zealand 190
approx 40% <30
Female 65.8% Pharmacists
Opioid substitution treatment (OST) Learner - content Website
Pre / post course evaluation and convenience sample of telephone interviews. Positive
Attitudes, skills and knowledge around opioid dependence 1
OST training was well received and successful in improving the knowledge, skills and attitudes of the pharmacists who participated. The online format was found to be highly acceptable, and appears to be a feasible mode of training.
Williams et al. (12) 2009 USA 103 No data No data
Family physicians
Diagnosis of acanthosis nigricans Learner - content Website
Pre / post knowledge test and post course evaluation. Feasibility evaluation. Positive Knowledge 2
Statistically significant change in knowledge.
Internet‐based learning for training health care professionals in service
25
Yank et al. (82) 2012 USA 57 No data No data
Medical residents, physicians
(23%), nurses (12%), Allied
Health Professionals
(14%)
Teaching self management
support to patients
Learner - content Learner - learner Learner - tutor
Webinar and website
Pre / post course evaluation. Webinar attendance recorded and focus groups. Positive
Survey results, webinar attendance and focus group data
1 & 2 (self reported)
Study participants demonstrated significant changes in the primary outcomes of beliefs and confidence.
Young et al. (83) 2013
Australia and
International 2244 No data No data Nurses and
midwives (91%)
Sudden Infant Death Syndrome
(SIDS) and sleeping
recommendations No data No data Pre / post course knowledge test Positive Knowledge 2
Statistically significant change in knowledge.
Internet‐based learning for training health care professionals in service
26
4.3 Programme theory for Internet-based learning for HCPs in-service
Initial programme theory for refinement The programme theory outlined by Wong et al. formed the initial starting point for the realist review. This programme theory was selected as it was developed to explain the influence of context on the use of Internet‐based learning in the closely related field of medical education (18). This programme theory identifies that there are two distinct challenges in the implementation of Internet‐based learning courses. Firstly, learners need to engage with the course. This would be equivalent to the real‐world challenge of getting learners to attend (for example) a lecture, seminar or workshop. Secondly, once learners engage, the course design and teaching approaches and content used in the course needs to be capable of helping the learners meet the course’s objectives. These two distinct challenges are explained and analysed in the sections below. Engaging learners Engaging learners can be a significant problem in Internet‐based learning. An indication of the scale of this problem can be found in the included studies which report such data. Course completion rates reported were; 16.7% (37); 18% (29); 20% (34), 38% (52); 58% (67); 60% (50) and; 81% (47). Some caution needs to be exercised when interpreting these data as these included studies recruited learners on a voluntary basis. However, a reasonable inference would be that those expecting to use Internet‐based learning to deliver education and training should not necessarily expect high course completion rates. An explanation of why this might be the case is provided below. In terms of course engagement, the programme theory developed by Wong et al. uses existing substantive theory by Moore and Benbasat on technological acceptance to make sense of the decision making processes underlying engagement by learners with Internet‐based learning courses (84). Using this theory, Internet‐based learning is conceptualised as a type of technological innovation – a new way of teaching and learning that the learner may or may not have tried before. Moore and Benbasat propose that there are eight properties of a technological innovation that influences decision making by a potential adopter (of the innovation), these are: Compatibility, Ease of use, Relative advantage, Trialability, Results demonstrability, Image, Visibility, Voluntariness. More details on these properties (or attributes as they are referred to by Moore and Benbasat) can be seen in Table 2. This theory by Moore and Benbasat is adapted from Roger’s work on the diffusion of innovations (85). One of the included studies drew on work of Roger’s as a means of making sense of their data (31).
Internet‐based learning for training health care professionals in service
27
Table 2: Summary of the attributes of an innovation
Attributes Compatibility ‐ with existing practices and values
Ease of use ‐ the degree to which the innovation is expected to be free of effort
Relative advantage ‐ split into the degree to which it is perceived as better than its precursor and the degree to which it is perceived as useful ‐ implicitly, for doing one's job
Trialability ‐ can be tried out on a limited basis
Result demonstrability ‐ the degree to which it is perceived as amenable to demonstration
Image ‐ the degree to which it is seen as adding to the user's social approval
Visibility ‐ the degree to which the innovation is seen to be used by others
Voluntariness ‐ the degree to which use of the innovation is controlled by the potential user's free will
In the realist review the attributes proposed by Moore and Benbasat were specified for Internet‐based learning and a realist logic of analysis applied. This realist logic of analysis conceptualises these attributes using a realist ‘lens’ – either conceptualising them as mechanism, context or outcome. Context and mechanism are not ‘free floating’ and should be linked to an outcome. Making sense of the data from included studies thus needed the development of a programme theory that incorporated not only the final desired outcome(s), but also (where necessary) all the intermediate outcomes. In addition, for each of these outcomes, data would be needed from included studies to support any inferences made about what the causal mechanisms might be and the contexts when it would most likely be triggered. Whilst the programme theory developed by Wong et al. was a useful starting point, analysis of the data from included studies indicated that additional disaggregation of intermediate outcomes was needed. Wong et al.’s programme theory did not specifically make a distinction between mechanisms for initial engagement (i.e. starting the course in the first place) and continuing engagement to completion. The programme theory developed and refined in this review makes such a distinction. For the intermediate outcome of initial engagement in Internet‐based learning, the single most important mechanism appears to be relative advantage. To want to undertake a course, learners need to be able to decide for themselves that an Internet‐based learning course is not only better than precursor but also useful for doing their job. The precursors in this instance are other ways of learning, such as textbooks, face to face courses, lectures and so on – what might be termed traditional learning approaches. Data from included studies indicate that a wide range of contexts can trigger within a learner the response that Internet‐based learning has relative advantage. These are
Internet‐based learning for training health care professionals in service
28
listed and discussed in detail below along with supporting data and illustrated in Figure 2. Once a learner has decided to engage in Internet‐based learning, their decision making processes can begin to take into account actual experiences. Prior to actually participating in Internet‐based learning, learners rely on their beliefs and feelings (based on previous experiences or other sources – e.g. advice and/or recommendations from friends and /or colleagues, what they have read etc.). In some cases, learners will have undertaken Internet‐based learning before and so have some basis on which to judge relative advantage. However, they will still not know for sure that their judgement was correct until they have actually started the course. For the intermediate outcome of completes Internet‐based learning, more mechanisms appear to become relevant. Along with relative advantage, ease of use, compatibility and trialability emerge as important mechanisms in causing continuing engagement, finally resulting in course completion. Data from included studies indicate that the range of contexts that can trigger these mechanisms has a degree of overlap with those that are relevant for the learner’s initial decisions about relative advantage. These are listed and discussed in detail below along with supporting data and illustrated in Figure 2.
Internet‐based learning for training health care professionals in service
29
Figure 2: Refined programme theory of Internet‐based learning
Course design and content Once a learner has engaged in
Internet-based learning the following become relevant
contextual influences
Previous experiences of Internet-based learning inform these contextual influences, if none, then they are perceived. Once learners have
engaged, they are able to ascertain for themselves if and how each contextual influence is relevant to them.
Engages in Internet-based
learning
Relative advantage
Compatibility
Access to learning
Competing pressures
Convenience Cost saving
Ease of use
Language & literacy
Ease of use
Pedagogic principles
Usefulness of content
Interactivity
Learner offered Internet-based
learning
Trialability
Completes Internet-based
learning
Changes in desired outcome(s) of Internet based learning:
Satisfaction
Knowledge
Patient health outcome(s)
Practice
Compatibility
Mechanisms
Intermediate outcome Intermediate outcome
* There is much less data in the included studies to support if Internet-based learning is able to change observed day to day practice or patient outcomes.
*
Internet‐based learning for training health care professionals in service
30
Access to learning Access to learning is an important influence on the decision making processes for learners who are considering engaging with Internet‐based learning. The data below have been used to illustrate and support this inference.
Dentist: “First time using this format. This is more accessible to staff … ” (3) Most of the participants (68%) disclosed that there were no other HIV training opportunities in their area. (47) Several participants noted that the program was a great idea and offered staff access to continuing‐education programs without having to juggle coverage or negotiate whose turn it was to attend a live program. (10) “Although face‐to‐face meetings are better than Web conferencing, for those people who can’t be there, Web conferencing is useful” (77) … the number of visits to this website quickly rose to more than 500,000 hits/month from more than 100 different countries, including many countries in Africa, Asia and Latin America (Figure 1). This rapid demonstration of interest and demand for clinical training content … (38) The attendance of eLearning courses has been unexpectedly high. In all the FSM [Fondazione Salvatore Maugeri] medical centres there has been an active participation. ... FSM educational activities can hence guarantee also to people not easily reached, assuring educational facilities to all the healthcare personnel and uniformity in terms of skills. (70) These results may be explained by the limited access to continuing education programs among participating health professionals in rural communities and minority ethnic groups. Reduced travel time and costs and avoiding missed days of work for education might have affected the high levels of satisfaction among health professionals in rural communities. Additionally, the increased communication among health professionals in rural areas during program sessions may have impacted their high levels of satisfaction. (43)
Compatibility Compatibility (or perceptions of it) appears to have an influence at both the point of deciding to engage as well as to continue to completion in Internet‐based courses. At the point of deciding to engage in Internet‐based learning, data on compatibility appears to act via its influence as a contextual influence on the relative advantage mechanism.
For those who preferred the face‐to‐face learning style, difficulty with the online format was evident: “I do prefer face‐to‐face over online,’’ ‘‘I learn better in a classroom setting,” (31)
Once learners have started an Internet‐based learning, their experiences on the course starts to inform their initial perceptions of compatibility and this may change for the better or for the worse. Compatibility changes from being a contextual influence on the
Internet‐based learning for training health care professionals in service
31
relative advantage mechanism (when a learner is deciding on whether to engage) to being a mechanism in itself once engagement has taken place.
A minority of participants commented that they did not find the resource useful and some felt that they received all the necessary information they required from attending courses or from existing policies/literature on the subject of child protection: “Our PCT [Primary Care Trust] has its own comprehensive safeguarding children policy and procedures, training etc. For our service using this Child Protection and the Dental Team package would have been a confusing duplication” “I find going to courses more helpful” (55) “I prefer hard copies . . . I like to hold the document and read on the train or bus.” (Interview 5, advocacy worker) (60)
I don’t think it [web version] should be compulsory because a lot of people are not comfortable with e‐learning. It was very interactive . . . there were exercises and you can self‐test, which was very useful, and it should be part of a range of options. (Interview 3, social worker) (60) Even the majority of learners who reported preferring face‐to‐face learning realized that due to their busy lives, the conflicting schedules of the team members, and not all the professionals being employed full‐time on‐site, the reality of face‐to‐face collaborative practice training is a challenging one. (5)
Within the included studies, there was data to indicate that Internet‐based learning course developers should undertake a needs assessment that included ascertaining the training modality preferences of their target learners.
The development process began with a needs assessment. ... The interview revealed a wide range of experiences with disaster management (DM) and online learning and indicated that some sites preferred an online only approach whilst others wanted a mix of online and face to face activities. Institutions were provided with the course option that was considered most desirable and feasible by their senior administration team. (30) … our formative research suggested that web‐based was the preferred method of our target audience. (72)
Competing pressures
There were substantial data to indicate that competing pressures on learners’ time was
an important contextual influence – not only at the point of deciding whether to engage
but also when a learner had engaged in Internet‐based learning.
When questioned regarding organizational barriers that may have impacted the learners’ ability to participate in the resource, the main issue that emerged was time. Related to this, it was a huge challenge for many of the teams to get together to complete the team meetings. Scheduling of these meetings was complicated by the fact that not all of the team members worked at the homes full‐time. (5)
Internet‐based learning for training health care professionals in service
32
A limiting factor was SBHC [school‐based health centers] staff availability because of factors
including ongoing heavy clinical responsibilities, lack of support for meeting time, pressure to
pro‐duce billable hours, etc. (39)
Once a learner had engaged, the context of competing pressures influenced relative
advantage and/or compatibility – potentially to the point that they would disengage or
not participate in the future.
“I am not convinced that Flying Start is the best way to facilitate my development. For one thing the
prohibitive amount to work through”
“I like most could do without the extra work FS causes” “I’m a newly qualified midwife in (name of region). I have ended up with piles of duplication.. There’s
not enough time in the day to keep going over all this self education malarkey e let me do my job!”
“Where in earth do we find the time, but really a lot of the Flying Start stuff (as much as I find it
annoying to do) is CPD. I did a CPD diary now and then before I started FS [Flying Start] and I find a lot
of the stuff are similar”
The high demands of undertaking Flying Start NHS alongside local CPD are seen by many as a
duplication of effort. Much of this can be attributed alongside the practitioner learning to manage their
time but also from an organisational perspective whereby for many no official protected time has been
offered to participants to undertake Flying Start. The result of these deficits is a feeling of stress and
over‐burden on newly qualified practitioners. (63)
While respondents reported that they theoretically had protected time to enable them to complete
Flying Start NHS, four out of five were not always able to take it. Difficulties were associated with
making time on a busy ward, lack of time available to mentors to fulfil their role and shift patterns
being incompatible. (32)
There may be several reasons why the learners did not score in the 85% range or higher on the
knowledge post test. … some learners commented that they were not able to attend all the lectures or
finish the modules due to limited amounts of time to participate in the learning activities and
competing priorities of clinical area patient care needs. (79)
The most commonly mentioned preventive or hampering factors were ‘‘lack of time’’ and ‘‘busyness’’ (40%, n=14) and job constraints (37%, n=13) such as busyness or lack of support at the workplace. (66)
Time was an issue for many staff. Sometimes it was about the timing of training when staff could not
be released, and sometimes it was the lack of time on a busy shift. (60)
Data in included studies indicated that it was possible, to an extent, to counter the
contextual influence of competing pressures on relative advantage. The obvious
‘solution’ was to create protected time for the learning activity.
Internet‐based learning for training health care professionals in service
33
Important point here that protected time is important‐ as on e face to face course you get to leave work to go on the course, but when the learning is Internet‐based, the time commitment can seem to be invisible and starts to happen in the learner’s own time. … it is important to note that some course members were awarded almost no study time at all, something that would be unlikely to occur if they were being supported through a classroom based course where university attendance was mandatory. (57)
Offering protected time for the Internet‐based learning appeared to have two functions.
It (obviously) provided the time to do the learning but also showed that the activity was
valued.
This evaluation has shown that newly qualified practitioners are more likely to complete Flying Start NHS if they are supported to do so. Thus investing time in newly qualified practitioners during their first year in employment is likely to pay dividends for the NHS, patients and carers, as well as the individuals concerned. [Banks 2011] However, it became clear in the interviews that allocation of time in the form of study leave was somewhat inequitable. Although every course member was supported by their manager to undertake the course, the level of support varied enormously between hospitals. This is significant given that course participants expressed concerns about the amount of material presented to them and the time available to peruse it. [Iley 2011]
Convenience As a contextual influence, convenience appeared to be both important and relevant to decisions about whether to engage in Internet‐based learning and continue with it to completion. It mainly influenced decisions on relative advantage but also, to a lesser extent compatibility.
For those who preferred online education, convenience was the main theme: ‘‘Online is so nice a person can fit the course into their personal schedule,’’ ‘‘I can fit it into my schedule,’’ ‘‘It’s easier than driving and missing work for education,’’ ‘‘Better than taking time off,’’ ‘‘I was able to build on my previous knowledge at my convenience and speed,’’ and ‘‘I don’t have to travel to access it.’’ (31) In addition, the distance learning format permitted a more flexible schedule for students, who were able to participate in this course without interrupting their normal work schedule. (38) “I liked that it was online and could be done at my own pace” (52)
‘‘liked the use of the computer and eLearning because it can be done at a time that is convenient for me.” (4) Many course members identified the ease of accessing the online course whenever they liked as
being very helpful when working round shifts and families. Having control over when they
engaged in the learning process was a prominent and valued feature of the course. (57)
The two main reasons for enrollment of students were the need for scheduling flexibility and the
Internet‐based learning for training health care professionals in service
34
fact that the course would not require the absence from their work places. (28)
One student had not studied at this level for 36 years, but was attracted to the course by its flexibility, being able to work at home and not having to go to university when she felt she would be out of her depth. (53)
Cost saving
There were limited data on the influence of cost savings as a contextual influence on. It mainly appeared to act on relative advantage at the point when a learner is deciding whether to engage with Internet‐based learning.
“I would prefer to attend seminars online rather than face‐to‐face for cost savings.” (77) Nurse: “Presentations like this are very cost effective [for me] and would be good for the Pacific [region]. We need more of them.” (3)
Ease of use
When deciding whether to engage in Internet‐based learning, the learner’s perceptions
(real or imagined) of how easy the course is to use is important. This acts as a
contextual influence on relative advantage.
Lack of computer access and insufficient computer skills are the most frequently cited barriers [to participation] (4) Nurse: “Have not used in Chuuk [Islands]. Don’t have power all the time. I think it makes me scared but I want to learn how to use the computer. We have a computer lab but I haven’t learned how to use it. But I would like to learn.” (3) In some cases, students lack the experience and training required to use the technologies. (38)
Once users have engaged in Internet‐based learning, in a similar fashion to
compatibility, ease of use itself becomes a mechanism that appears to cause continuing
engagement or not.
Participants were also invited to respond to two open‐ended items on the course evaluation. The first item was "What did you like best about the program? " The most common responses were: "using a computer for training” (n = 15), "everything" (n = 5), "easy to use and understand" (n = 4) and “working at your own pace” (n = 3). (54) "I just love its flexibility, and being able to pick it up, put in down, and go back to sections. It was so friendly ‐ when is the next one coming?" (53)
Internet‐based learning for training health care professionals in service
35
For those who preferred the face‐to‐face learning style, difficulty with the online format was evident: … “I had difficulty getting into the site every time I tried to do a lesson.” (31) The reason for a participation of only 65% of staff in this study was that some nurses who took the pre‐test and completed the educational material were unable to complete the posttest because the computer program ‘‘timed out’’ when nurses left to complete emergent patient care. Unfortunately, these nurses could not then reaccess the program as test access was only allowed one time. (36) Seven of the 10 respondents did not listen to the webinar … because of technical difficulties (57.1%) … (41) The vast majority of learners interviewed reported that they did not have any problems with navigation; nor did they have trouble accessing the resource online. … However, a few did face roadblocks accessing the resource, or aspects of it (e.g. audio, video), and these related to security settings, filters, pop‐up blockers, firewalls, and the screen resolution on their computers. Several learners reported that they needed the technical support at their organization to download Flash Player (software necessary to view the resource) onto their computers as they did not have network permission to do so themselves. There were a couple of programming oversights that were a source of frustration for the learners. They included not being able to save their responses in activities if they wanted to go back in the resource to review something and not being able to print the content. (5)
Ease of use as a contextual trigger for relative advantage is closely related to another context, that of the availability of the Internet. This context is particularly important when Internet‐based learning courses are being delivered to low and middle income countries. As can be seen in the data below, where a HIV course is being delivered in resource‐limited settings, assumptions cannot be made that Internet access is available and/or reliable.
Some sites were challenged by the inconsistency and speed of internet access, the availability of electricity, and the presence and functioning of computer equipment. Some volunteer coordinators were less computer and internet literate than others and required more technical assistance from Seattle. (47)
Additional data to support the importance of taking this context into account include:
Despite great efforts by collaborating organizations in Zambia, it was initially very difficult to engage students willing to participate in this first course. Reasons for this reluctance to participate in a distance learning course included a healthy skepticism about the reliability of internet access in Zambia, required for the low‐bandwidth live faculty chats, course registration and knowledge testing. (38) Nurse: “Have not used in Chuuk [Islands]. Don’t have power all the time. (3)
Possible solutions to this problem of Internet availability and reliability included providing printed materials, CD‐ROMS and course content suitable for low and high bandwidths (38).
Internet‐based learning for training health care professionals in service
36
To try to address the issue of ease of use, some Internet‐based learning course designers have deliberately provided training and technical support to potential learners.
Nurse: “We need to have good technical support, and we need to have both a tech person and a medical person familiar with the content to answer questions present for any such presentations.” (3) Because many … physicians identify computer skills and evidence‐based medicine searches as areas of deficiency, the DUCOM [Drexel University College of Medicine] program offers information technology skills sessions to enhance computer‐based medical literature research skills. (78)
Some of the training and technical support was deliberately provided face to face due to the familiarity of learners with such and approach.
There are inherent problems when using this type of course. Feelings of isolation are common, but the use of WebCT and mail tool has attempted to reduce these feelings. Two open days were arranged to encourage students to network and meet other students in order to discuss common problems and identify solutions. This enabled students to resolve further issues such as misunderstandings and technical problems, as tutors were also in attendance at these open days. (53) Course members felt the initial attendance day on the course gave them a satisfactory introduction to using the Blackboard site and felt sufficiently confident in using this software. (57)
Language and literacy
The programme theory developed by Wong et al. did not mention the contextual issue of language and literacy. This was likely to be due to the learner group that programme theory was developed for – namely medical students, trainees and qualified doctors, had high levels of language and/or literacy skills. However, if an Internet‐based course is to be offered to a wider range of HCPs, then assumptions cannot be made about the language skills and literacy levels of the learners. Data within the included studies was sparse on this contextual influence. There was little information on its exact influence on the mechanisms identified in the programme theory developed in this realist review. Andreazzi et al. report that when a course on microbiology was being offered to physicians and microbiologists, learners “…had difficulties with the English language.” (28), but it is unclear what effect this had on initial and continuing engagement. A possible solution to this challenge is outlined by Irvine et al. who suggest that when the target learner group is “… typically less educated…” then modifications should be made to the course content “…on‐screen text elements were short and written at the second–sixth grade reading level.” (58).
Internet‐based learning for training health care professionals in service
37
Trialability Trialability appears to be a mechanism that is relevant once learners have engaged with Internet‐based learning. Having the ability to ‘try and see’ this form of learning is important to learners. Positive experiences seem to; influence learners to continue on a course; engage in further Internet‐based learning and; recommend it to others. The most important contextual influences seem to be related to course design and content (which is discussed below in more detail).
The participants found SL [Second Life] to be an effective method of learning and reported they would like to take other courses in SL. … The majority reported they would recommend SL to a colleague, strongly agreed that SL is effective, and exhibited interest in taking more courses with SL. (11) The early case discussions were notable for the US faculty providing much of the discussion and input. However, within a very short period of time, our colleagues in Ethiopia took ownership and leadership of this program. Now, 4 years after a skeptical and difficult start to the program, there have been more than 200 case discussions recorded and distributed throughout Ethiopia and clinicians throughout the world have benefited by viewing these excellent discussions on line. The program was such a success that the AAU has now built its own state‐of‐the‐art video conferencing facility in their medical school auditorium, expanding participation to many more faculty and students. In addition, two more Ethiopian medical schools have build similar facilities and are now actively participating in these case conferences, which continue to be broad cast throughout the world. (38) All mentors agreed they would mentor future students again and had other staff working with them who wanted to undertake the course themselves. (57) Additionally, nearly one fourth of the physicians (5 of 21) and nearly half of the nursing staff (16 of 39) NMCD users indicated that as a result of having the electronic tool, they had personally begun using an H/NP. (59) Almost all participants would have done an additional module had it been available (“the more the merrier”) and this extra work would not have impacted on their decision to undertake the training program. (81) The majority (91.6%) of respondents had recommended the webinar to colleagues (6)
Course design and content The contents of a course and how this content is delivered in an Internet‐based learning course are important to learners. Within the included studies, there is sparse data on how course content and design (as context) influences learners’ decisions about whether to engage in Internet‐based learning. Some data indicates that branding, reputation and/or endorsement by professional bodies may be important (29;38;71;79). Once a learner has decided to engage and has started using Internet‐based learning, then data within the included studies indicate that (at least) three aspects of course
Internet‐based learning for training health care professionals in service
38
content and design are important – pedagogic principles, usefulness of the content for the learner’s job and interactivity. These are all discussed in more detail below. Pedagogic principles Many of the Internet‐based learning courses were designed using specific pedagogic principles or educational theories. A broad range of these principles or theories were used and some courses used more than one (4;29;38;62;63;69;71). The range included; Roger’s theory of diffusion of innovations (31); Theory of planned behaviour (33); Adult learning theory (4;35;62;68); Knowles’ self directed learning theory (36); Constructivist learning (4;6); Community of Inquiry (41;42); situated learning theory (7); community of practice (49;57;63); principles of self‐directed learning (52); cognitive theory of multi‐media learning (58); instructional design (62); Kolb’s theory of experiential learning (66) and; knowledge‐to‐action framework (73). As reported earlier, 70.1% of the studies reported positive findings. Coupled with the finding that the included studies also used a broad range of pedagogical principles and theories to inform their course design, it would be reasonable to infer that there is no single superior way to design Internet‐based learning courses. The judicious and coherent application of pedagogical principles and theories in course design appears to be more important. Such a finding is of no great surprise and has been previously reported in the literature (15;86). Course design does not appear to be a major challenge for Internet‐based learning course developers. Despite the broad range of pedagogical principles and theories used to inform course design, the included studies appear to demonstrate that learners can be satisfied and knowledge change can occur. An important point to note is the processes by which course design has occurred. Many Internet‐based learning courses included in this review specifically reported that the course was developed by a dedicated project team that include educationalists and content experts, often with specific technological input (3;4;7‐9;11;31;43;44;57;65;66;68;72;73;79). To ensure that the learning content is useful and wanted by their target learner group, some course developers report that they deliberately undertook needs assessments beforehand (30;43;72). Appropriately and well designed courses act as a contextual influence on relative advantage and compatibility. Some of this data has already been presented above but additional detail is provided below on course content and interactivity. In brief summary, a well designed course provides a ‘positive’ learning experience, one that is useful, interactive, easy to use all of which provides reasons for the learner to want to carry on (relative advantage) and potentially changes their learning preferences (compatibility).
Internet‐based learning for training health care professionals in service
39
Course content Course content that was useful and relevant to the day to day work of learners was appreciated and formed one of the building blocks of a well designed course. When a learner is making decisions about whether to participate in Internet‐based learning, the usefulness of the course content for doing their job can only be perceived or ‘guessed at’. A learner cannot know for sure until they have engaged with the course. The included studies were trials or evaluations of an Internet‐based learning course. Whilst participants were actively recruited, none of the evaluations/trials specially asked about individual’s perceptions of the importance of usefulness as a contextual influence on their motivation to participate. There is thus an absence of evidence from included studies to inform this contextual influence and we have to rely on extrapolating from Moore and Benbasat’s theory on its importance. In other words, that usefulness of a course contents is an important context influence on relative advantage when a learner is deciding on whether to engage with a course. However usefulness is relevant for course design, as once the learner has engaged their expectation is to encounter learning content that is useful for their job. The utility or usefulness of the course content to a learner’s everyday job is one of the important contextual influences on learners’ decision making (relative advantage).
All of the nurses also reported that the content level was appropriate for their job activities, and most nurses (87.5%) found the lesson very helpful for enhancing their job performance related to the care of the kidney transplant recipient. (4) Participants felt that the practical information included within the resource (for example, signs of child abuse and neglect) was particularly useful. Studies in the ‘Resources’ section, which could be photocopied or downloaded, also attracted favourable comments. Among these, the ‘flowchart for action’, which summarises the steps that should be taken when there are concerns about the welfare of a child… Participants also commented on the usefulness of including con‐tact information for further information and advice: “Flowchart for referral – outlining the procedures in a straightforward and unambiguous manner” “Guidance for observing signs of abuse’ ‘Most relevant was who/where/how to contact professionals for ‘outside’ help”. (55) The training areas that participants identified as particularly useful were the topics covered in Module 3: Practical aspects of administration and dispensing OST [opioid substitution treatment] and management of common clinical issues. Participants commented that the content was “most useful since it is very practical” and “relates closely to everyday practice”. (81) Most people found the resource practically based, comprehensive, easy to use and understand. “I think it’s more user‐friendly. It addresses all of the questions that people have that might be a useful way to think about the legislation, the practical things.” (Inter‐view 10, nursing lecturer) (60)
Internet‐based learning for training health care professionals in service
40
A few participants thought the resource was too broad and lacked detail. Unit 4 was seen as complicated. There was not enough information relating to tribunals and detention (also poorly addressed in the Act). Little attention was given to learning disabilities or personality disorder. (60)
A couple of learners interviewed did not feel that the content always reflected the reality of LTC [long term care]. … One pharmacist explained it this way “I think that the way collaborative practice was portrayed [in the learning resource] only happens in an ideal world; that you actually get to sit down at a table and discuss patients like that. It certainly doesn’t happen in our world here. (5)
A few studies used specific strategies to ensure that the course contents would be useful. One strategy (already mentioned above) was to specifically undertake a needs assessment:
Topics for the programs were offered in response to specific requests by hospital staff and health providers as determined by surveys, key informants, and focus groups. (43)
Another was to focus the content for learners who most need to learn the knowledge or
skill:
The 5 hospitals integrated the course into hospital‐specific training activities targeted at those clinicians who might perform or assist in CVC [central venous catheter] placement. (48)
It would have seemed reasonable to infer that content that is relevant to the day to day job of the learner would have a positive impact on learning. However, this is not an association that was specifically explored in any of the included studies. In addition, the importance of useful content as a contextual influence on continuing engagement with Internet‐based learning was not explored within the included studies. Both these issues have not been addressed in this review and would be suitable for further work, initially by searching for relevant studies in the broader educational literature. Interactivity Interactivity was something that was frequently designed into Internet‐based courses that were included in this review. Interactivity was also mentioned by learners as something they valued. None of the included studies or learners defined exactly what they meant by interactivity, but analysis of the included studies indicate that there were a combination of processes that learners wanted. For example, they wanted to be able to ask questions and receive answers in webinars, to clarify their understanding of a topic (through discussion with a fellow learner and/or an expert) and/or to get feedback on performance (such as on multiple choice questions). In many of the included studies, course designers often obliged, for example, by providing feedback on answers to
Internet‐based learning for training health care professionals in service
41
multiple choice questions, building in email, chat room or bulletin boards into the course or enabling audio or typed questions to speakers in webinars. The value place on interactivity Internet‐based courses is well recognized in the literature (18) and there is an association between degree of interactivity and learning outcomes (13;87). One explanation for this comes from existing substantive theory on how learning takes place – Laurillard’s Conversational Framework (88). This framework was developed to try to explain how it is that learners gain academic knowledge in higher education. It outlines the activities necessary to complete the learning process. Specifically, it focuses on the need for the tutor (face‐to‐face or virtual) to try to understand how and what a learner understands about topic and then use dialogue and practice with intrinsic feedback to help the learner change their understanding. In other words, learners learn by having the opportunity to clarify their understanding of a topic through discussion with others. Data within the included studies strongly support the value placed by learners on the importance to them of interaction.
“This was very user friendly & being able to come back & understand why my answers were wrong was most helpful.” (29) Participants were particularly satisfied with the … interactivity (91.5%) of the webinar. (6) Doctor: “We just do the audio with the slides, but this one is better because you can actually see the presenter and you can really interact. This is perfect.” Nurse: “Liked audio lecture since I can ask questions. Also liked computer training, but could not ask questions.” “With audio you need more of the microphones, so that you can interrupt and ask questions when the time is right.” (3) “It was very interactive . . . there were exercises and you can self‐test, which was very useful, and it should be part of a range of options.” (Interview 3, social worker) (60) Interviewed pharmacists were generally satisfied with the Web‐based training. They liked the dynamic and interactive aspects: “It’s interesting, different, lively, and stimulating.” (65) … interactivity between participants and instructors was encouraged by the Internet‐based feedback system. A total of 176 participants accessed the web site 3564 times and asked 284 questions to the instructors using the system. (76)
Where interaction was lacking, learners specifically ask for more.
Doctor: “First time exposed to this kind of training. It was very informative. I liked using the computer. I liked everything about it except that I would like to have more interaction.” (3) Much of the course content was viewed by the course members and mentors as being of a high calibre, for example clinical guidelines and peer reviewed papers. Some course members said
Internet‐based learning for training health care professionals in service
42
that they would have particularly benefited from being able to converse online with the specialist who had actually put the course content together for each particular session. … much of the learning that would normally take place as a result of class discussion did not actually occur online and therefore the course member she mentored had to look further afield to gain different perspectives. This problem could be alleviated to a degree through facilitated online forums but this would require a much greater online presence from the course team. (57) Many pharmacists found that this type of training raises many questions but does not allow for feedback: “My head is full of questions, and I’d like to be able to consult people with expertise in the field to help me judge specific situations properly.” The addition of a Web discussion forum was suggested to provide opportunity to ask questions. (65)
Not really interactive. My mind wandered off at one point.” (3)
Interactivity appears to act as a contextual influence on relative advantage. As with usefulness and the judicious and coherent application of pedagogic principles and theories, it contributes to making a course a ‘positive’ experience. A reasonable inference that can be drawn is that this is more likely to lead to course completion.
A refined realist programme theory Internet-based learning courses: putting it all together The refined realist programme theory illustrated in Figure 2 is attempts to capture the complexities of trying to make sense why, how, for who, in what contexts and to what extent Internet‐based learning might work in the real world. Whether or not learners achieve the desired outcome when they engage in Internet‐based learning depends on a complex interplay of contexts and mechanisms at various points within the programme theory. This complex interplay was recognised within the included studies and is illustrated below by this text from Kettles et al.:
Strengths and weaknesses of the different formats and the way in which they were disseminated, familiarity with and preferences for the different versions of the resource varied according to circumstances and preferred learning styles. There was no clear preference for one version over the others. This is not surprising, given the widely varied contexts in which participants were involved with mental health. In some cases, the most important factor was the availability of access, e.g. whether there was web access at work or online access at home. There was value in all three formats being available, which permitted wider access than would be possible with a single format. (60)
However, it is possible to make some sense of this complexity using the programme theory refined in the realist review. To illustrate the explanatory potential of this programme theory, two of the included studies are examined in more detail – Harrington et al. (54) and Macdonald et al. (5). These two studies have been selected as they both used Internet‐based learning to train long term elderly care home staff – Harrington et al. provided training on fire safety and Macdonald et al. on collaborative learning. Despite the different topic areas, country the studies took place in and
Internet‐based learning for training health care professionals in service
43
educational delivery methods, both demonstrated high satisfaction from learners, changes in knowledge after the course and self reported changes in behaviour. Both courses reported high levels of engagement and completion by learners. The programme theory helps to make sense of these two studies by throwing light on why these findings might have occurred. The key lies in exploring the contexts in which these studies occurred and how these have influenced the mechanisms identified in the programme theory. Both settings were ones where HCPs had few convenient, cheap and flexible high quality educational options. Many of the HCPs worked across geographically dispersed sites. Additionally, staff turn over was often high and new staff need to be trained as soon as possible (as opposed to have to wait till the next available face to face training course). In the care home setting, many of the HCPs were unable to take time off to study as they need to prioritise patient care. Staffing levels were often very tight and there was often no one to ‘cover’ any time taken off. Also the employers had limited (if no) budget for training. Yet training was either wanted by the HCPs or compulsory – such as fire training in the case of Harrington et al. Furthermore, these Internet‐based courses were developed after a needs assessment (as in Harrington et al.) and by experienced multidisciplinary course developers. Using the programme theory, we can see that the contexts described above can all influence the mechanisms of relative advantage to encourage HCPs to want to use Internet‐based learning. For these learners, Internet‐based learning offered convenient anywhere anytime learning – provided access to learning. The flexibility of access, whilst not addressing competing pressures, does at least give the learner choice on when to engage, something not available if the course was at a fixed time and place and the HCP had to take time off work to attend. These courses were offered free as part of the studies so it is not possible to comment on the influence of this context on decisions about relative advantage. Once learners had engaged, data within these studies indicated that on the whole the HCPs were highly satisfied with the design and content of the course. The courses were interactive enough, useful for their day to day work and easy to use. This is likely to be because they were developed by experienced course developers who had a clear idea of what the need was in their target learner group, pitched it at the right level of literacy, with relevant content and made it easy to use. The majority of the learners responded to the positive learning experience by going on complete the course and to evaluate it positively. In summary, what the above, hopefully, illustrates is that the programme theory can be used to make sense of the outcomes observed in Internet‐based learning courses for HCPs. The programme theory tells us that learners need to have a reason to want to engage with Internet‐based learning. The Internet‐based learning has to be at least as good as if not better than the alternative educational options – it has to have relative
Internet‐based learning for training health care professionals in service
44
advantage in the mind of the learner. Courses may be perceived to have relative advantage because they; offer (the contextual influences of) access to learning; cost savings; convenience; ease of use; are accessible (in language and literacy terms); fits in with competing pressures and; are compatible with the learners practice and values. Once a learner has decided to engage with Internet‐based learning, rather than perceptions, the learner now has a chance to find out for themselves if their initial perceptions were correct. A complex interplay and balancing act occurs. The learner can now see if Internet‐based learning is compatible and easy to use by engaging and learning on the course – trialability. Compatibility and ease of use, become part of the decision making process (as mechanisms) along with relative advantage. These mechanisms continue to be influenced by actual experiences that informs and modifies (potentially both positively and negatively) their initial perceptions. Learners find out if an Internet‐based course actually does offer; access to the learning they need; cost savings; convenience, ease to use; compatibility; ease of understand (in terms of language and literacy terms; fit with competing pressures; and well designed, work relevant content that is interactive. The learner balances their experiences and then decides to continue or not. If the course has been well designed then it is a reasonable expectation that (at the very least) learners will be satisfied and their knowledge increased. This understanding of the contexts and their complex interactions with mechanisms has been used to develop a decision support resource which is discussed in detail the Recommendations section below.
Internet‐based learning for training health care professionals in service
45
5. Discussion
The review in this report set out to answer ‘How, why, for whom, to what extent, in what circumstances does Internet‐based learning work for health care professionals in‐service?’. This focus addresses more the issues on the implementation of Internet‐based learning as the current body of literature in this topic area suggests that this where the knowledge gaps are. Findings are that Internet‐based learning courses have been used to train or educate a broad range of health care professionals (HCPs) on a diverse range of topics. Both men and women from a wide spread of ages appear to be able to learn using Internet‐based learning. The most likely changes to result are satisfaction levels with the courses and knowledge change. There is less confidence that Internet‐based courses are able to change the day to day practice of HCPs and/or patients’ health outcomes. The Internet‐based learning courses were designed by course designers to use Internet technology differently and there does not appear to be one ‘ideal’ course design approach. Data from the included studies indicate that, as expected, the outcomes achieved from Internet‐based learning vary by context. A complex interplay exists between these contexts and outcomes. The biggest challenge in the use of Internet‐based learning appears not to be course design and content, but that of getting learners to want to engage with the course in the first place and then continue to completion. This may be compared to the face to face educational challenge of getting students to turn up to an educational event (such as a workshop), participate and to stay to the end. As can be imagined a complex interplay between a range of contexts will determine whether the aforementioned happens. It is possible to make sense of this complex interplay and to do so this review has developed and refined a realist programme theory specifically for the use of Internet‐based learning in HCP training. This programme theory ‘maps’ out the sequence of steps needed to achieve the final desired outcome from Internet‐based learning course. Internet‐based learning courses are conceptualised as innovations – something new that many (but not all) learners will not have used before. Such an approach enables the programme theory to draw on existing substantive theory to increased understanding – in this case using diffusion of innovation theory as refined for technological innovations by Moore and Benbasat. To get learners to engage in Internet‐based learning, they need to first see that there are good reason to do so – i.e. there is relative advantage, especially compared to the alternative educational or training available to them. These reasons may be because they believe or feel that Internet‐based learning offers; access to learning; cost savings;
Internet‐based learning for training health care professionals in service
46
convenience; ease of use; are accessible (in language and literacy terms); fits in with their competing pressures and are compatible with their practice and values. Once the learner has started the course, by being able to actually try out the course (trialability) they are able to determine if their beliefs and feelings about the course are as initially perceived. Does the course really offer what they want in terms of ;access to learning; cost savings; convenience; ease of use; accessibility (in language and literacy terms); fit with their competing pressures and compatibility? Course design and content play an increasing important part at this stage of the programme theory. Decisions about relative advantage will include the learner’s experiences of doing the course. A course that has judiciously and coherently used pedagogical principles and theories will be one that is easy to use and interactive – both course contexts that are valued by learners. Learners will expect that the course content is relevant to their day to day job. To optimise course design and content, some of the included studies’ course developers undertook initial needs assessments, provided technical support (face to face or via the Internet) and worked in multidisciplinary project teams consisting of educational, content and technical expertise. The programme theory also illustrates that there is a ‘feedback loop’ that comes from learners engaging with an Internet‐based learning course. With use, learners may (or may not) change their beliefs and feelings about Internet‐based learning and so influence either their continuing engagement with a course they are taking or their future engagement with other courses. What hopefully the refined programme theory illustrates is that it is possible to make sense of how Internet‐based courses may be implemented to make it more likely that they are used and completed by learners. Two important caveats need to be mentioned when Internet‐based learning courses are used to educate or train HCPs in‐service. Firstly that there is a paucity of data to show that Internet‐based courses can cause learners to measurably or observably change their day to day clinical practice. This is not a unique finding and it is well recognised that education or training often only makes up part of any intervention that is designed to change clinical practice (89). What is well demonstrated in the included studies is that Internet‐based learning courses can be expected to change learner’s satisfaction and knowledge levels. Secondly, where it has been reported in the included studies (7;28;36;54;69;75), it is not realistic to expect that any gains in knowledge will necessarily be sustained for long. Again this is not an unexpected finding (86) and discussion on why this may be case is beyond the scope of this review.
Internet‐based learning for training health care professionals in service
47
5.1 Comparison with exiting literature This realist review further refines and builds on the programme theory developed by Wong et al. (18). The programme theory in that review was developed and refined for Internet‐based learning courses for medical students and doctors only. In this review, the scope of the programme theory was broadened to include all HCPs. Further refinements were made in order to throw more light on the issues on engagement. The two main refinements were to disaggregate engagement into initial engagement and continuing engagement. In addition, additional context issues were identified in this review – namely accessibility (in language and literacy terms) and fit with the learner’s competing pressures. This broader and more refined programme theory is a novel contribution to the existing literature. Many of the findings from this review are coherent with the existing literature. Similar to that reported by Cook et al. the majority of included studies in this review used a before and after design and found that knowledge changes were not sustained. The other characteristics of the included studies in this review also closely match that Cook et al. (15). Both Cook et al. and Philips et al. (90) also report that most Internet‐based learning courses mainly measured satisfaction and knowledge changes. On the issue of the incorporation of interactivity into Internet‐based learning courses, Cook et al. (15), Carol et al. (91) and Mansouri et al. (92) report evidence that this has an important influence on learning outcomes. This review throws further light on this issue by illustrating that this may be because the ability to interact to clarify understanding and gain feedback is an integral part of the learning process, as argued by Laurillard (88).
5.2 Strengths and limitations In common with all literature reviews this realist review has a number of strengths and weaknesses. Strengths The topic area covered by this review is one that is generally considered to be a complex one. In addition the knowledge gaps in this topic are in implementation – i.e. any new knowledge produced needs to be able to address the questions of in what circumstances, how and for whom should Internet‐based learning be used. The nature of the topic area under investigation and knowledge needed both need to be taken into account when choosing a literature review method (93). Realist review was chosen as the literature review method precisely because it can not only deal with complex issues, but also provide knowledge on implementation. This choice of review approach constitutes an important strength.
Internet‐based learning for training health care professionals in service
48
This realist review produced many findings that were largely coherent with the existing literature on Internet‐based learning. Whilst there have been some additional insights gleaned from this review (such as a refined and broader programme theory), the fact that many of its findings build on and are coherent with the existing literature adds additional plausibility. This needs to be borne in mind when reading the limitation section below. Finally this realist review has been conducted in a transparent way. Detail has been provided on the processes used in this review and where possible a link between data to inferences made has been provided. Limitations This realist review was carried out within a limited budget and time envelope (six months). As a result a number of review processes were truncated or omitted. Only a single database (MEDLINE) search with language limitations and a narrow date range was undertaken. There was not enough time to search other databases. Within the screening process, there was only one screener but additional quality checks were undertaken. In addition, contrary to common practice in other realist review, no additional searches were undertaken to look for additional data to develop and refine sections of the programme theory. Thus it is possible that relevant studies and documents will have been missed. However it must be borne in mind that no realist review will ever claim to be exhaustive, comprehensive and definitive. The nature of the complex topic areas examined in realist reviews preclude this – there is just too much complexity, too many contexts interacting with mechanisms to generate outcomes (94). What instead a realist review builds are coherent and plausible explanations in the form of programme theories that are tested against empirical data from included studies and documents. The explanations are always going to be partial, as new data will always emerge, some of which will modify any existing explanatory programme theory. Within this realist review, sense has been made of the Internet‐based learning for training HCPs in‐service. The included studies, whilst not exhaustive and comprehensive do reflect the characteristics of this body of literature. Additionally the realist programme theory has been developed and tested against empirical data and is coherent with existing substantive theory. The scope of this realist review is limited in that it developed a refined a programme theory that covers almost exclusively the initial and continuing engagement of learners with Internet‐based learning courses. What this review has only touched on is the area of how to design an Internet‐based learning course. The scope has also not included the cost of setting up and running Internet‐based learning courses and also the needs learning and technical needs of faculty for running such courses. The limited searching undertaken has meant that there is the issue of the absence of data on some aspects of the programme theory. For example this review did not find
Internet‐based learning for training health care professionals in service
49
any data from within the included studies on four of the attributes outlined by Moore and Benbasat ‐ namely demonstrability, image, visibility and voluntariness. Potentially, voluntariness is a very important attribute to understand further as it would be reasonable to infer that where an Internet‐based learning course is not voluntary, this context would have influences on a learner’s assessment of relative advantage. This absent data appears to be an artefact of the type of studies included in this review. All participants in the included studies were volunteers. Time and resources permitting, additional searching could have been undertaken to make more sense of this important attribute. Regardless, this absence of data does not necessarily negate the remaining explanatory powers of the programme theory developed and refined in this review. For example an Internet‐based course may be compulsory, but this would still only be one contextual influence on a learner’s decision making about relative advantage. A learner may still drop out if other aspects of the course did not ‘work’ for them – e.g. the course is not easy of use, there are language and literacy problems, and the course content and design are poor. This review makes no claims to be comprehensive and exhaustive. No realist review should make such claims. Instead, what this review has done is to transparently shed light on some of the most prominent and seemingly important contextual influences that influence learners’ likelihood of initially engaging with and continuing on an Internet‐based course. As new data emerges from more primary evaluations of Internet‐based learning, further contexts may emerge as important. As such, with time, the programme theory presented in Figure 2 is likely to need further refinement.
Internet‐based learning for training health care professionals in service
50
6. Recommendations
The goal of these recommendations is to provide Internet‐based learning course developers with a decision support resource. These recommendations are based on the findings of the realist review. These findings have been translated into a series of questions that focus on important aspects of course design. For example, the findings indicate that before learner will consider engaging with Internet‐based learning they must believe or feel that it has relative advantage – i.e. is better than alternative educational or training available to them. In the recommendations this issue of relative advantage has been converted into the following question – ‘Will Internet‐based learning offer relative advantage (i.e. will it offer something that is better in some way than currently available educational/training options)?’ The contexts that influence this initial process are then listed below the aforementioned question as further questions. The idea being to prompt course developers to reflect on the need, in the first place, for Internet‐based learning and if needed, how they might wish to consider how they would optimise the design and content of their course. The initial questions focus more on prompting course developers to consider the learning needs of their target group and reasons for why the Internet may (or may not) be the best delivery medium. The subsequent questions in the decision support resource are deliberately focussed on getting; a) learners to engage with an Internet‐based learning course in the first instance (as this appears to be a significant challenge) and; b) learners to then complete the course. Once a learner has engaged, a broad range of teaching approaches appear to work and do help course participants to learn. As such the decision support resource is less prescriptive on how learners should be taught – i.e. on course design itself. The decision support questions are not a checklist or a cook‐book that, if followed will invariably lead to the desired learning outcomes delivered by an Internet‐based learning course. No course works for everyone all of the time as for every course there will be a myriad of contextual differences. Instead thoughtfully addressing each question will mean that each of the more important contextual influences (identified in this review) are considered, thus making it more likely that a course will lead to the desired learning. The decision support questions may be found in Box 1. The decision support questions do not address the issues of the cost of setting up and maintaining an Internet‐based course. It also does not tackle the issue of training and technical support for faculty involved in delivering a course. These two areas are beyond the scope of this report.
Internet‐based learning for training health care professionals in service
51
Box 1: Important questions for course developers to ask and address when considering and designing an Internet‐based learning course.
Needs assessments
Has a needs assessment been undertaken to understand the needs of the target learners?
How has your needs assessment informed course rationale, design and content (see below for more guidance)?
Rational for use of Internet‐based learning
Why is the Internet being considered as a medium for delivering education / training?
Does your target learner group have reliable Internet access?
What are the courses’ objectives? Are these achievable using Internet‐based learning?
Course design and content
Will Internet‐based learning offer relative advantage (i.e. will it offer something that is better in some way than currently available educational/training options)? For example will your Internet‐based learning course:
o Increase access to learning? o Save time and/or money for the learner? o Be convenient and easy to use? o Be understandable (in terms of language and literacy) to the learner? o Fit in with the competing pressures of your learners? o Be compatible with your learners’ practices and values?
Does your course project team possess the necessary educational, content and technical skills? If not, how will you obtain such expertise?
Which pedagogical principles and/or theories will you base your course on? Why? o Through which processes will learning take place? o Will any interactivity be built into the course? How?
How will you ensure the course content will be relevant and useful to your learners’ day to day job?
Technical support
Will you offer technical support to learner? If yes, how?
Internet‐based learning for training health care professionals in service
52
7. Conclusions
Internet‐based learning is increasing being used to teach a wide range of topics to many different types of health care professionals. The literature indicates that Internet‐based learning can be just as efficacious as more traditional forms of education and training (e.g. such as face to face lectures or course). The greater challenge with Internet‐based learning is knowing when to use it, for whom and for which types of learning goals – in preference to traditional forms of education. In other words, the knowledge gaps in Internet‐based learning are in implementation and no longer in proving efficacy. The realist review conducted in this report set out to address these very questions on implementation. Building on existing understanding of Internet‐based learning by health care professionals, it has found that engaging learners, both initially to get them to start a course and then to continue to completion is a far bigger challenge that than course design and content. The data in the studies included in this review indicate that many Internet‐based learning course developers possess the necessary expertise to develop courses that are satisfactory to learners and can change their knowledge levels. Few however consider the issue of engagement and as a result many experience poor uptake and large numbers of dropouts from their courses. This review finds that this does not necessarily have to be the case. A key finding is that courses need to be developed in such a way as to give learners good reasons to want to start a course and complete it. A complex interplay of context influences the decision making processes within learners. Many of these contexts also influence their decisions to complete a course. Whilst there is no ‘magic’ combination of contexts that will always ensure that an Internet‐based course will turn out to be a run‐away success, judiciously addressing each of these contexts is likely to result in a course that is liked by learners and achieves its objectives.
Internet‐based learning for training health care professionals in service
53
8. Appendices
List of Appendices 8.1 Search strategy for MEDLINE 8.2 Modified version of Kirkpatrick’s model for evaluation
Internet‐based learning for training health care professionals in service
54
8.1 Search strategy for MEDLINE
1 Learning/ or Computer‐Assisted Instruction/ or Education, Medical, Continuing/ or Internet/ or e‐learning.mp. or Education, Distance/
127148
2 world wide web.mp. 2943
3 evaluation studies.mp. or Evaluation Studies/ 327054
4 Educational measurement.mp. or Educational Measurement/
28120
5 (assess* or evaluat* or apprais*).mp. [mp=title, abstract, original title, name of substance word, subject heading word, keyword heading word, protocol supplementary concept, rare disease supplementary concept, unique identifier]
4047807
6 health personnel/ or allied health personnel/ or "coroners and medical examiners"/ or dental staff/ or dentists/ or faculty, dental/ or faculty, medical/ or faculty, nursing/ or health educators/ or health facility administrators/ or infection control practitioners/ or medical laboratory personnel/ or medical staff/ or nurses/ or nursing staff/ or personnel, hospital/ or pharmacists/ or physicians/
201446
7 1 or 2 128546
8 3 or 4 or 5 4061689
9 6 and 7 and 8 2347
10 9 not medical students.mp. [mp=title, abstract, original title, name of substance word, subject heading word, keyword heading word, protocol supplementary concept, rare disease supplementary concept, unique identifier]
2229
11 10 2229
12 limit 11 to (english language and yr="2007 ‐ 2013") 1159
Internet‐based learning for training health care professionals in service
55
8.2 Modified version of Kirkpatrick’s model for evaluation*
Level of evaluative data Detailed description of criteria for each level of evaluative data
1: Learner Satisfaction (reaction)
Evaluates how well participants liked a programme (e.g. participants’ perceptions, satisfaction with programme objectives, content, instruction, delivery, and /or instructors).
2: Learning outcomes (learning)
Assesses changes in skills, knowledge, or attitudes among learners (most commonly conducted through pre and post‐test study designs).
3: Performance improvement (behaviour)
Extent to which learning has influenced the post‐learning behaviour or performance of a learner in their practice setting. Attempts to answer the question: are the newly acquired skills, knowledge, or attitudes being used in the everyday environment of the learner?
4: Patient / health outcomes (results)
Measures tangible results which are influenced by the performance of the learner as a result of participation in the continuing education activity (e.g. improving patient health or improving efficiencies). Evaluation at this level is challenging given the variety of uncontrollable variables a learner encounters when he or she leaves the boundaries of a programme.
*from: Curran V, Fleet L. A review of evaluation outcomes of web‐based continuing medical education. Medical Education 2005;39:561‐7
Internet‐based learning for training health care professionals in service
56
9. Reference List
(1) Bateman J, Davies D. The challenge of disruptive innovation in learning
technology. Medical Education 2014; 48:227‐228.
(2) Cheng Bo, Wang M, Morch A, Chen N‐S, Kinshuk, Spector JM. Research on e‐learning in the workplace 2000–2012: A bibliometric analysis of the literature. Educational Research Review 2014; 11:56‐72.
(3) Chen TH, Buenconsejo‐Lum LE, Braun KL, Higa C, Maskarinec GG. A pilot evaluation of distance education modalities for health workers in the U.S.‐Affiliated Pacific Islands. Developing Human Resources in the Pacific 2007; 14:22‐30.
(4) Gerkin KL, Taylor TH, Weatherby FM. The perception of learning and satisfaction of nurses in the online environment. Journal for Nurses in Staff Development 2009; 25:E8‐E13.
(5) Macdonald CJ, Stodel EJ, Chambers LW. An online interprofessional learning resource for physicians, pharmacists, nurse practitioners, and nurses in long‐term care: benefits, barriers, and lessons learned. Informatics for Health & Social Care 2008; 33:21‐38.
(6) Borgerson D, Dino J. The feasibility, perceived satisfaction, and value of using synchronous webinars to educate clinical research professionals on reporting adverse events in clinical trials: a report from the Children's Oncology Group. Journal of Pediatric Oncology Nursing 2012; 29:316‐322.
(7) Canchihuaman FA, Garcia PJ, Gloyd SS, Holmes KK. An interactive internet‐based continuing education course on sexually transmitted diseases for physicians and midwives in Peru. PLOS One 2011; 6:e19318.
(8) Chhabra HS, Harvey LA, Muldoon S, Chaudhary S, Arora M, Brown DJ et al. www.elearnSCI.org: a global educational initiative of ISCoS. Spinal Cord 2013; 51:176‐182.
(9) O'Connor ME, Brown EW, Lewin LO. An Internet‐based education program improves breastfeeding knowledge of maternal‐child healthcare providers. Breastfeeding Medicine 2011; 6:421‐427.
(10) Sweet BV, Welage LS, Johnston JP. Effect of a Web‐based continuing‐education program on pharmacist learning. American Journal of Health‐Systems Pharmacists 2009; 66:1902‐1903.
Internet‐based learning for training health care professionals in service
57
(11) Mitchell S, Heyden R, Heyden N, Schroy P, Andrew S, Sadikova E et al. A pilot study of motivational interviewing training in a virtual world. Journal of Medical Internet Research 2011; 13:e77.
(12) Williams RL, McPherson L, Kong A, Skipper B, Weller N, clinicians PN. Internet‐based training in a practice‐based research network consortium: a report from the Primary Care Multiethnic Network (PRIME Net). Journal of the American Board of Family Medicine 2009; 22:446‐452.
(13) Cook D, Garside.S., Levinson A, Dupras D, Montori V. What do we mean by web‐based learning? A systematic review of the variability of interventions. Medical Education 2010; 44:765‐774.
(14) Cook D. Web‐based learning: pros, cons and controversies. Clinical Medicine 2007; 7:37‐42.
(15) Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet‐based learning in the health professions: a meta‐analysis. 2008; 300:1181‐1196.
(16) Lam‐Antoniades M, Ratnapalan S, Tait G. Electronic continuing education in the health professions: an update on evidence from RCTs. 2009; 29:44‐51.
(17) Ellaway R, Pusic M, Yavner S, Kalet A. Context matters: emergent variability in an effectiveness trial of online teaching modules. Medical Education 2014; 48:386‐396.
(18) Wong G, Greenhalgh T, Pawson R. Internet‐based medical education: a realist review of what works, for whom and in what circumstances. BMC Medical Education 2010; 10:12.
(19) Wong G, Greenhalgh T, Westhorp G, Pawson D. Realist methods in medical education research: what are they and what can they contribute? Med Educ 2012; 46:89‐96.
(20) Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review‐‐a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy 2005; 10 Suppl 1:21‐34.
(21) Pawson R. Evidence‐based policy: A realist perspective. London: Sage, 2006.
(22) NVivo. 2014.
(23) Excel. 2014.
(24) Wong G, Greenhalgh T, Westhorp G, Pawson R. RAMESES publication standards: realist syntheses. Journal of Advanced Nursing 2013; 69(5):1005‐1022.
Internet‐based learning for training health care professionals in service
58
(25) Sherman H, Comer L, Putnam L, Freeman H. Blended versus lecture learning: outcomes for staff development. 2012; 28:186‐190.
(26) Ferreira CC, Souza SO. LevRad software as a tool to learn how to proceed with an evaluation of barriers. 2011; 57 Suppl:OL1500‐OL1504.
(27) Alemagno SA, Guten SM, Warthman S, Young E, Mackay DS. Online learning to improve hand hygiene knowledge and compliance among health care workers. Journal of Continuing Education in Nursing 2010; 41:463‐471.
(28) Andreazzi DB, Rossi F, Wen CL. Interactive tele‐education applied to a distant clinical microbiology specialization university course. 2011; 17:524‐529.
(29) Arenella C, Yox S, Eckstein DS, Ousley A. Expanding the reach of a cancer palliative care curriculum through Web‐based dissemination: a public‐private collaboration. 2010; 25:418‐421.
(30) Atack L, Bull E, Dryden T, Maher J, Rocchi M. An evaluation of learner perception of competency and satisfaction with three models of an interdisciplinary surge capacity course. Journal of Allied Health 2012; 41:106‐112.
(31) Bachman JA, Hsueh KH. Evaluation of online education about diabetes management in the school setting. The Journal of School Nursing 2008; 24:151‐157.
(32) Banks P, Roxburgh M, Kane H, Lauder W, Jones M, Kydd A et al. Flying Start NHSTM: easing the transition from student to registered health professional. 2011; 20:3567‐3576.
(33) Battaglia JN, Kieser MA, Bruskiewitz RH, Pitterle ME, Thorpe JM. An online virtual‐patient program to teach pharmacists and pharmacy students how to provide diabetes‐specific medication therapy management. American Journal of Pharmaceutical Education 2012; 76:131.
(34) Beynat J, Ben Mehidi A, Aubert JP, Bron AM, Massin P, Creuzot‐Garcher C. RETIDIAB: assessment of a continuing medical education website for the improvement of diabetic retinopathy management. Daibetes & Metabolism 2011; 37:118‐123.
(35) Blazer KR, Christie C, Uman G, Weitzel JN. Impact of web‐based case conferencing on cancer genetics training outcomes for community‐based clinicians. Journal of Cancer Education 2012; 27:217‐225.
(36) Blum CA, McCaffrey RG, Bishop M, Singh R, Swan A, Neese R et al. Educating nurses about veno‐thrombolytic events (VTE). Journal for Nurses in Staff Development 2012; 28:173‐176.
Internet‐based learning for training health care professionals in service
59
(37) Boehmer S, Karpa K. Evaluating the value of a web-based natural medicine clinical decision tool at an academic medical center. BMC Health Services Research 2011; 11:279.
(38) Bollinger RC, McKenzie‐White J, Gupta A. Building a global health education network for clinical care and research. The benefits and challenges of distance learning tools. Lessons learned from the Hopkins Center for Clinical Global Health Education. Infectious Diseases Clinics of North America 2011; 25:385‐398.
(39) Booker JM, Schluter JA, Carrillo K, McGrath J. Quality improvement initiative in school‐based health centers across new Mexico. Journal of School Health 2011; 81:42‐48.
(40) Bussieres JF, Lebel D, Voytenko S, Marquis C, Bailey B. A pilot study to assess an online training module to quickly identify drugs on resuscitation trays. Annales Francaises d'Anesthesie et de Reanimation 2011; 30:31‐36.
(41) Buxton EC, Burns EC, De Muth JE. Professional development webinars for pharmacists. American Journal of Pharmaceutical Education 2012; 76:155.
(42) Buxton EC, De Muth JE. Pharmacists' perceptions of a live continuing education program comparing distance learning versus local learning. Research in Social & Administrative Pharmacy 2013; 9:230‐235.
(43) Bynum AB, Irwin CA, Cohen B. Satisfaction with a distance continuing education program for health professionals. Telemedicine and e‐Health 2010; 16:776‐786.
(44) Carter L, Rukholm E, Kelloway L. Stroke education for nurses through a technology‐enabled program. Journal of Neuroscience Nursing 2009; 41:336‐343.
(45) Casebeer L, Engler S, Bennett N, Irvine M, Sulkes D, DesLauriers M et al. A controlled trial of the effectiveness of internet continuing medical education. BMC Medicine 2008; 6:37.
(46) Casebeer L, Brown J, Roepke N, Grimes C, Henson B, Palmore R et al. Evidence‐based choices of physicians: a comparative analysis of physicians participating in Internet CME and non‐participants. BMC Medical Education 2010; 10:42.
(47) Chung MH, Severynen AO, Hals MP, Harrington RD, Spach DH, Kim HN. Offering an American graduate medical HIV course to health care workers in resource‐limited settings via the Internet. PLOS One 2012; 7:e52663.
(48) Comer A, Harris AD, Shardell M, Braun B, Belton BM, Wolfsthal SD et al. Web‐based training improves knowledge about central line bloodstream infections. Infection Control and Hospital Epidemiology 2011; 32:1219‐1222.
Internet‐based learning for training health care professionals in service
60
(49) Curran JA, Abidi SSR. Evaluation of an online discussion forum for emergency practitioners. Health Informatics Journal 2007; 13:255‐266.
(50) De Rivas B, Barrios V, Redon J, Calderon A. Effectiveness of an Interventional Program to Improve Blood Pressure Control in Hypertensive Patients at High Risk for Developing Heart Failure: HEROIC study. Journal of Clinical Hypertension 2010; 12:335‐344.
(51) Desai T, Sanghani V, Fang X, Christiano C, Ferris M. Assessing a nephrology‐focused YouTube channel's potential to educate health care providers. Journal of Nephrology 2013; 26:81‐85.
(52) Fleet LJ, Fox G, Kirby F, Whitton C, McIvor A. Evaluation outcomes resulting from an internet‐based continuing professional development (CPD) asthma program: its impact on participants' knowledge and satisfaction. Journal of Asthma 2011; 48:400‐404.
(53) Gill A. E‐learning and professional development‐‐never too old to learn. British Journal of Nursing 2007; 16:1084‐1088.
(54) Harrington SS, Walker BL. The effects of computer‐based fire safety training on the knowledge, attitudes, and practices of caregivers. Journal of Continuing Education in Nursing 2009; 40:79‐86.
(55) Harris JC, Bradbury J, Porritt J, Nilchian F, Franklin CD. NHS dental professionals' evaluation of a child protection learning resource. British Dental Journal 2011; 210:75‐79.
(56) Hon CY, Gamage B, Bryce EA, LoChang J, Yassi A, Maultsaid D et al. Personal protective equipment in health care: can online infection control courses transfer knowledge and improve proper selection and use? American Journal of Infection Control 2008; 36:e33‐e37.
(57) Iley K, McInulty L, Jones I, Yorke J, Johnson M. Developing competence in cardiac care through the use of blended learning: course members' and mentors' accounts. Nurse Education Today 2011; 31:323‐327.
(58) Irvine AB, Billow MB, Bourgeois M, Seeley JR. Mental illness training for long term care staff. Issues in Mental Health Nursing 2012; 13:81.
(59) Karpa KD, Boehmer S. Examining utilization of a natural medicine Web‐based tool among non‐primary care physicians, academic faculty, and nurses at one institution. The Annals of Pharmacotherapy 2012; 46:513‐520.
Internet‐based learning for training health care professionals in service
61
(60) Kettles AM, Kiger AM, Philip K, Caldow J. An evaluation of a Mental Health Act educational resource. Journal of Psychiatric and Mental Health Nursing 2008; 15:662‐669.
(61) Kobak KA, Opler MGA, Engelhardt N. PANSS rater training using Internet and videoconference: results from a pilot study. Schizophrenia Research 2007; 92:63‐67.
(62) Kobak KA, Stone WL, Ousley OY, Swanson A. Web‐based training in early autism screening: results from a pilot study. Telemedicine and e‐Health 2011; 17:640‐644.
(63) Lauder W, Roxburgh M, Atkinson J, Banks P, Kane H. The quality of on‐line communication in a national learning programme for newly qualified nurses, midwives and allied health professionals. Nurse Education in Practice 2011; 11:206‐210.
(64) Lefebvre H, Pelchat D, Levert MJ. Interdisciplinary family intervention program: a partnership among health professionals, traumatic brain injury patients, and caregiving relatives. Journal of Trauma Nursing 2007; 14:100‐113.
(65) Legris ME, Seguin NC, Desforges K, Sauve P, Lord A, Bell R et al. Pharmacist Web‐based training program on medication use in chronic kidney disease patients: impact on knowledge, skills, and satisfaction. Journal of Continuing Education in the Health Professions 2011; 31:140‐150.
(66) Leikola SNS, Tuomainen L, Ovaskainen H, Peura S, Sevon‐Vilkman N, Tanskanen P et al. Continuing education course to attain collaborative comprehensive medication review competencies. American Journal of Pharmaceutical Education 2009; 73:108.
(67) Lewin LO, O'Connor ME. "BreastfeedingBasics": web‐based education that meets current knowledge competencies. Journal of Human Lactation 2012; 28:407‐413.
(68) Llambi L, Esteves E, Martinez E, Forster T, Garcia S, Miranda N et al. Teaching tobacco cessation skills to Uruguayan physicians using information and communication technologies. Journal of Continuing Education in the Health Professions 2011; 31:43‐48.
(69) Marsh‐Tootle WL, McGwin G, Kohler CL, Kristofco RE, Datla RV, Wall TC. Efficacy of a web‐based intervention to improve and sustain knowledge and screening for amblyopia in primary care settings. investigative Ophthalmology & Visual Science 2011; 52:7160‐7167.
Internet‐based learning for training health care professionals in service
62
(70) Mazzoleni MC, Rognoni C, Pagani M, Imbriani M. Online education in a large scale rehabilitation institution. Studies in Health Technology and Informatics 2012; 174:67‐71.
(71) Olson CA, Shershneva MB, Brownstein MH. Peering inside the clock: using success case method to determine how and why practice‐based educational interventions succeed. Journal of Continuing Education in the Health Professions 2011; 31 Suppl 1:S50‐S59.
(72) Palmer RC, Samson R, Triantis M, Mullan ID. Development and evaluation of a web‐based breast cancer cultural competency course for primary healthcare providers. BMC Medical Education 2011; 11:59.
(73) Pluye P, Grad R, Granikov V, Theriault G, Fremont P, Burnand B et al. Feasibility of a knowledge translation CME program: Courriels Cochrane. Journal of Continuing Education in the Health Professions 2012; 32:134‐141.
(74) Tian L, Tang S, Cao W, Zhang K, Li V, Detels R. Evaluation of a web‐based intervention for improving HIV/AIDS knowledge in rural Yunnan, China. AIDS 2007; 21 Suppl 8:S137‐S142.
(75) Toledo P, Eosakul ST, Goetz K, Wong CA, Grobman WA. Decay in blood loss estimation skills after web‐based didactic training. Simulation in Healthcare 2012; 7:18‐21.
(76) Tsugihashi Y, Kakudate N, Yokoyama Y, Yamamoto Y, Mishina H, Fukumori N et al. A novel Internet‐based blended learning programme providing core competency in clinical research. Journal of Evaluation in Clinical Practice 2013; 19:250‐255.
(77) Valaitis R, Akhtar‐Danesh N, Eva K, Levinson A, Wainman B. Pragmatists, positive communicators, and shy enthusiasts: three viewpoints on Web conferencing in health sciences education. Journal of Medical Internet Research 2007; 9:e39.
(78) Varjavand N, Greco M, Novack DH, Schindler BA. Assessment of an innovative instructional program to return non‐practicing physicians to the workforce. Medical Teacher 2012; 34:285‐291.
(79) Wallen GR, Cusack G, Parada S, Miller‐Davis C, Cartledge T, Yates J. Evaluating a hybrid web‐based basic genetics course for health professionals. Nurse Education Today 2011; 31:638‐642.
(80) Walsh K, Treasure W. Reflective online learning resource on functional symptoms: feedback on effectiveness. Education for Primary Care 2009; 20:390‐392.
Internet‐based learning for training health care professionals in service
63
(81) Walters C, Raymont A, Galea S, Wheeler A. Evaluation of online training for the provision of opioid substitution treatment by community pharmacists in New Zealand. Drug and Alcohol Review 2012; 31:903‐910.
(82) Yank V, Laurent D, Plant K, Lorig K. Web‐based self‐management support training for health professionals: a pilot study. Patient Education and Counseling 2013; 90:29‐37.
(83) Young J, Higgins N, Raven L, Watson M. Supporting nurses and midwives to promote a Safe Infant Sleeping eLearning Program. Australian Nursing Journal 2013; 21:41.
(84) Moore G, Benbasat I. Development of an instrument to measure the perceptions of adopting an information technology innovation. Inform Sys Res 1991; 2:192‐222.
(85) Rogers E. The Diffusion of Innovations. Fourth ed. New York: Free Press, 1995.
(86) Lau V. Computer‐based teaching modules design: principles derived from learning theories. Medical Education 2014; 48:247‐254.
(87) Cook D, Levinson A, Garside S, Dupras D, Erwin P, Montori V. Instructional design variations in internet‐based learning for health professions education: a systematicreview and meta‐analysis. Academic Medcine 2010; 85:909‐922.
(88) Laurillard D. Rethinking University Teaching: a conversational framework for the effective use of learning technologies. Second ed. London: RoutledgeFalmer, 2002.
(89) Forsetlund L, Bjørndal A, Rashidian A, Jamtvedt G, O'Brien M, Wolf F et al. Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews 2009;(2):CD003030. DOI: 10.1002/14651858.CD003030.pub2.
(90) Phillips JL, Piza M, Ingham J. Continuing professional development programmes for rural nurses involved in palliative care delivery: an integrative review. 2012; 32:385‐392.
(91) Carroll C, Booth A, Papaioannou D, Sutton A, Wong R. UK health‐care professionals' experience of on‐line learning techniques: a systematic review of qualitative data. 2009; 29:235‐241.
(92) Mansouri M, Lockyer J. A meta‐analysis of continuing medical education effectiveness. 2007; 27:6‐15.
Internet‐based learning for training health care professionals in service
64
(93) Wong G. Is complexity just too complex? Journal of Clinical Epidemiology 2013; 66:1199‐1201.
(94) Pawson R. The Science of Evaluation: A Realist Manifesto. London: Sage, 2013.
Recommended