Upload
doankhuong
View
216
Download
4
Embed Size (px)
Citation preview
1
Return on
Investment for
the All Wales
Academy
Juste Abramovaite,
Skills for Justice
August, 2015
2
Contents
Introduction to the All Wales Academy and Return on Investment 3
ROI and eLearning – other studies 4
Methodology 6
Findings 7
Training on Data Protection and Information Governance – current picture 7
Additional feedback from local authority representatives 10
Conclusion and recommendations 13
Appendices 15
3
Introduction to the All Wales Academy and Return on Investment
The All Wales Academy (AWA) is a new eLearning platform being launched in Wales which is
planning to provide over 60 eLearning training modules for staff across all 22 Welsh local
authorities. Its vision is to introduce a new way of working by up-skilling employees and
increasing their employability. In addition the academy will reduce training budgets and
make more efficient use of employee time. It is also hoped that the collaboration model put
in place by the project will facilitate collaboration between local authorities in the long-
term.
As it is important for the developed platform to be sustainable and leave a robust legacy
once the project has concluded it was decided that a Return on Investment evaluation was
needed to investigate and identify the benefits of the platform and the collaborative
foundation. This method of evaluation was deemed particularly appropriate as Return on
Investment’s (ROI) aim is to measure and account financial return in real time as well as
taking into account future savings and wider benefits. ROI is therefore not purely focusing
on financial aspects of the project but considers the value in both quantitative and
qualitative terms. Barker (2001)1 has outlined the usefulness of ROI analysis for any learning
products and services:
1 Return on Training Investment: An Environmental Scan and Literature Review (Barker, 2001)
Enables learning to be viewed as an investment rather than expenditure.
Enables informed choices between learning options.
Encourages employers and employees to take learning more seriously.
Compares the cost of training/retraining to new hires.
Contains and/or maximizes learning budgets.
Makes course objectives and content more relevant.
Refines and revises, or eliminates, weak learning programs.
Improve the financial worth of human capital assets.
Attracts attention to a particular problems, e.g. with productivity.
Projects future teaching/learning costs
Improves the efficiency of resource utilization.
Encourages transfer of training.
Measure the effectiveness of education/training and more.
4
It is important to recognise that ROI is different to a simple cost saving – instead of
measuring a reduction in expense (cost saving) ROI is the rate of return on investment so it
takes into consideration how much was spent on investment initially.
ROI and eLearning – other studies
John L. Setaro in his How E-Learning Can Increase ROI for Training2 states that while the
most obvious impact of e-Learning on ROI is the significant cost savings it can produce over
traditional training; well constructed eLearning is not only faster and less expensive than
classroom training but also more effective. Numerous studies have shown that people
learn faster with multimedia training; they more accurately recall what they learned over a
longer period of time and they are better able to transfer what they learned to actual
performance. Studies conducted by the military, education and industry cite a 15-25%
increase in learning with significant increases in retention and transfer of training.
Brandon Hall in his Multimedia Training Return on Investment3 reviewed a number of
studies on multimedia training and ROI. The studies involved companies such as Intel,
American Airlines, Pacific Gas and Electric, IBM, Bethlehem Steel, Bell South, Steelcase, and
Pizza Hut. In all studies reviewed the results indicated significant increases in the quality of
learning when computer based training was compared with traditional classroom
instruction.
However, this does not mean that face to face training is without use; research suggests
that it can be very effective when used as a combined way of learning - blending eLearning
together with classroom learning. Online Training ROI: Five Features That Positively Impact
Your Bottom Line 4states that online training can stand alone as an efficient solution, but
Web based programs deliver optimum returns when used in conjunction with other
techniques to create a blended learning program. In fact, one study cited in above
publication, found that online students had more peer contact with others in the class,
enjoyed it more, understood the material better and performed, on average, 20 per cent
better than those taught in a traditional classroom.
A case study from Northumberland County Council describes how the Council recently
introduced an online Learning Pool in order to provide effective and efficient learning and
development solutions for 6,000 staff across the County. The results were outstanding and
in just one year, the Council had reduced the cost of delivery of training by 80%. With the
2 http://www.swslim.org.uk/downloads/sl1718.pdf
3 Hall, Brandon (1995, July/August). "Multimedia Training's Return on Investment" Workforce Training News
4 https://www.peoplematter.com/system/files/resources/PeopleMatter-White_Paper-Training.pdf
5
increased efficiencies provided, training cost on average £4 per employee per hour. Because
of this overall 80% reduction in costs, the Council is now able to provide even more training
to its staff.
Savings for Northumberland County Council were achieved through a combination of using5:
5 https://www.totaralms.com/case-studies/northumberland-county-council
Online training (over 10,000 hours of classroom delivery
saved).
Self-service functionality for course bookings (over 1,200
hours of administration time saved).
Bespoke content creation (over 20 hours of content
developed).
Reduction in CO2 omissions and fuel savings, all whilst
improving consistency and scalability.
6
Methodology
In order to collect data on the current picture of how much the training is costing at the
moment and how much it is likely to change once the AWA is live, an online questionnaire
was prepared and sent out to 11 local authorities (LAs) in early June6. The questionnaire
asked respondents to provide details on the following:
Unfortunately, potential respondents were reluctant to take part as many felt that they did
not have the answers to the questions that were asked. A change of tactics was required
and an amended questionnaire was sent out in early July7. Questions in this latter
questionnaire focused on Data Protection and Information Governance Courses delivered in
the last financial year. 13 LAs were asked to take part and eight responded to the
questionnaire. In addition, representatives from two LAs were keen to engage in more
detailed discussions following the questionnaire so in-depth phone interviews were held
with these individuals.
6 Letter which was sent out to the LAs with a list of questions is in Appendix 1.
7 An amended letter with a full set of questions is in Appendix 2.
The training costs LAs are facing.
How much of training costs were for face to face training
and how much was for e-learning modules.
The number of people attending training sessions.
The length of each training course.
How training costs will change once the AWA is in place.
If training was accredited.
How satisfied people were with the training they
received.
7
Findings
Initial feedback from most respondents in regards to the questions asked in the
questionnaire was that these were asked too prematurely and should instead be posed in a
year’s time when the AWA is fully operating.
None of the LAs could commit to providing an estimate for future costs once the AWA is
live; most of them stated that it is too early to discuss and that these costs have not yet
been determined. Thus establishing an exact figure of how much money will be saved once
the AWA is fully operating is difficult. However, LAs did provide a picture of what their
training on Data Protection and Information Governance looks like at the moment. All data
collected is represented in Appendix 3.
Training on Data Protection and Information Governance – current picture
From the information provided by LAs it is evident that training differs in both numbers of
people who attend and the costs involved.
Graph 1 below outlines how many people attended the training in each Local Authority
(who provided such information) in the last financial year for Data Protection and
Information Governance Courses. Cardiff had the highest number of attendees overall
(5,126) while Wrexham had the lowest number with just 40 people attending the training.
These figures show a combined number of people attending e-Learning and face to face
training.
“...the ROI may be premature as the system isn't live at
the moment”
(Email from a Local Authority representative)
8
Graph 1
1055
328
5126
40
1700
4591
90
0 1000 2000 3000 4000 5000 6000
Blaenau Gwent
Gweynedd
Cardiff
Wrexham
Denbighshire
Caerphilly
Vale of Glamorgan
Total number of people attended the training
(Data taken from Local Authorities responses to ROI questionnaire (Appendix 2 and 3))
Courses delivered also varied in length. Graph 2 below shows the average length of the
courses for e-Learning and face to face training in all LAs (who provided such information).
On average, the shortest face to face training was delivered in Gwynedd (1.5 hours) and the
longest in Wrexham (6 hours). E-learning was significantly shorter in all LAs and ranged
between 0.14 hours in Cardiff to 1 hour in Caerphilly. Understandably, not all face to face
training can be replaced with a version of an e-Learning module; however partially moving
towards more e-Learning courses being available and delivered would significantly reduce
the time people need to spend away from their desks working. That would not directly
affect the costs of the training; however indirect savings would be substantial. For example,
3 hours per course delivered could be saved by changing it to e-Learning; if 5,000 people
undertook that course each year that would be equivalent to 15,000 hours more of working
time for the Local Authority. If on average, employees are paid £20,000 per annum that
would lead to indirect savings of £153,846 a year8.
8 3 hours*5000 people = 15,000 hours, each year one full time employee works 1950 hours (37.5 hours a week
* 52 weeks = 1950 hours). Then 15,000 hours divided by one full time employee's yearly hours: 15,000/1,950 = 7.69 full time employees. If you take £20,000 as an average pay for one employee, you get 7.69*£20,000 = £153,846 savings a year.
9
Graph 2
(Data taken from Local Authorities responses to ROI questionnaire (Appendix 2 and 3))
The costs of training vary widely across all LAs. On average, to train one person in Blaenau
Gwent costs less than a pound – this Council offers a range of eLearning modules which are
short and very cost effective. In comparison, the average cost of face to face training paid in
Wrexham is £146 per person. The graph below shows the average cost of face to face
training is just under £60 while the average cost of e-Learning is less than £6 which
suggests that there is a potential for substantial savings across LAs once more e-learning is
implemented. Although LAs were unable to provide estimates on how costs will change
once the AWA is live, this suggest that introducing e-Learning across all LAs will be cost
effective in the long run.
10
Graph 3
(Data taken from Local Authorities responses to ROI questionnaire (Appendix 2 and 3))
Additional feedback from local authority representatives
After discussing their views on the AWA, LA representatives raised some interesting points
which would not be captured for some time in the LAs’ data (due to this project being at
such an early stage), however, they could be checked in a year or two:
Duplication – Health and Safety training courses were given as one of the examples
where there is too much choice which often leads to duplication in courses undertaken by
the staff. This in turn leads to extra costs to the LAs – unifying the training courses, while of
course, still keeping needed differences according to each LA, would be the way forward.
For that reason the AWA could be a great opportunity not just to directly reduce the costs
to the LAs but also to make a selection of training courses more unified. This means that
fewer courses would be duplicated, staff would need to undertake fewer courses, overall
direct costs would be lower and time off work taken to attend various courses would
decrease.
“..AWA is a chance to unify things so we avoid all
duplications in training we are facing now... there is too
much choice”
(Email from a Local Authority representative)
11
Welsh language
The LA representatives generally viewed courses in Welsh positively however as there are
not many cases when courses in Welsh language are needed, translation costs can be a
burden to the LAs. Therefore, the AWA which offers training in Welsh would reduce the
total cost.
Implicit training costs
The LA representatives felt that face to face training is more expensive not only because of
how much each training package costs but also how much time staff members need to take
off from work to attend including travel and subsistence cost. The AWA providing eLearning
modules would save staff time in this respect, as staff could undertake training courses
from their computers, tablets or even smart phones.
Accessibility
Some LAs indicated that not all of their staff were previously able to access the training; in
some cases up to one third of all staff working for the LAs in question, would not have
access to a computer. The AWA with its new approach allowing staff to access training on
smart phones and tablets would cater for a more accessible and flexible way to learn.
Additionally, employees with disabilities who would not always be able to attend face to
face training (if it is held outside the office in a non-accessible building) would benefit by
training being delivered via eLearning.
Accreditation
None of the LAs spoken to had accredited their employees for the training courses
attended. Two main reasons given were the extra cost involved and that it was not a
requirement in staff job descriptions. Having collected data on the number of learners
registered and the annual costs of Approved Centre Status for Leadership & Management
and Health & Social Care, it can be concluded that the cost per learner varied a lot between
those LAs that provided data9.
Some LA representatives commented on how expensive it is for them to pay for Approved
Centre Statuses and since there are a lot of duplications across training courses on offer, the
AWA is seen by them as a way forward as it will help LAs to save money on training, reduce
training times by reducing duplications and allow accreditation of employees at much lower
9 All of the data is available in Appendix 4.
12
cost. Furthermore, the Institute of Customer Service suggests that employees’ job
satisfaction could increase once training is accredited as people would feel more satisfied,
committed and motivated10.
Additionally, Steven Schmidt11 found in his study on the relationship between satisfaction
with workplace training and overall job satisfaction, a significant relationship between the
two. Components of job training, including time spent in training, training methodologies,
and content, were determined to be significant in their relationship to job training
satisfaction and trainees were significantly more satisfied with the training they received
when the methodology employed was their preferred one. Therefore, it is possible that the
AWA could not only increase efficiency at work as more staff would be trained but could
also increase employees’ job satisfaction.
10
https://www.instituteofcustomerservice.com/1723/Training-accreditation---TrainingMark.html 11
Steven W. Schmidt, 2007, The relationship between satisfaction with workplace training and overall job satisfaction, Human Resource Development Quarterly, Volume 18, Issue 4, pages 481-498.
13
Conclusion and recommendations
Research findings show that there is a definite need for the AWA and that it could bring
substantial improvements on how training is delivered in the future. It is however worth
noting that not much of the quantitative analysis can be made at this point due to the early
stage of the project. The direct cost of training to all LAs is likely to decrease once all AWA
modules are in use since eLearning has a proven record of being more cost effective than
classroom learning (see the section ROI and eLearning – other studies). Additionally, the
AWA could bring increased accessibility, lower travelling costs, further introduction of
Welsh language courses and reduced duplication across all courses thus enhancing the
overall efficiency of the training provided.
It is recommended that a further financial analysis is conducted in 12-24 months time in
order to fully establish the success of the project. It will therefore be crucial to collect data
for the next year or two in all participating LAs on the following:
- The number of people attending different training courses
- Difference in numbers of those courses being face to face
training and e-Learning
o compare whether more LAs shifted towards more e-
Learning/less face to face training since AWA was
introduced
- All aspects of costs involved including implicit costs such as:
o time taken off work to attend training
o how many people chose to access the training via
smart phone or a tablet
o how many people were accredited for the courses
they attended
o was there a difference in numbers of people with
moving disabilities attending training courses since
AWA was introduced
o was there any improvement in employees work
satisfaction ratings
14
To be able to use future data for financial analysis it is very important that data collection is
standardized across all LAs. At this point, it is possible to observe the positive feedback and
expectations from all LAs that took part in the research. Respondents clearly see the
potential of the AWA and look forward to the positive changes they expect this new e-
learning platform to bring. For the sustainability and success of this project, conversations
and collaboration need to continue between all parties involved and feedback from those
organizing and attending training is important to collect as to ensure that training is
reviewed and improved on a regular basis.
15
Appendices
Appendix 1 – email with the ROI questionnaire sent LA representatives in early June
All Wales Academy (AWA) e-learning project: data for the Return on Investment research
Dear {insert name},
I am following up on an email from Rachel Searle regarding the All Wales Academy (AWA) e-
learning project. As you will be aware, Skills for Justice are currently working with partners
to develop and deliver the AWA and as part of this project, I am responsible for conducting
a Return on Investment (ROI) exercise.
In order to do this, I need to explore in detail the types of training (and the costs involved)
that were in place before the implementation of the AWA and to compare these with the
actual (or expected) training/costs going forward.
Consequently, I would kindly ask if you could provide any data you have available on the
following:
- How many people were involved in training courses before AWA was implemented?
(please give us a figure which only includes the training which is transferable to AWA)
- And/or how many hours of training that was?
- What was the cost of that training?
- How much does it cost to train people now using AWA?
- How were people accredited for the training they undertook?
- Has accreditation changed since AWA?
- Any data available on employees’ satisfaction with their work place?
- Any other information which you think would be of the importance to us to know.
Collecting this information is crucially important in order to see the effectiveness and future
sustainability of AWA, hence, your corporation would be very much appreciated.
16
Appendix 2 – email with the amended ROI questionnaire sent to LA representatives in early
July
All Wales Academy (AWA) Return on Investment Questionnaire
Skills for Justice are currently working with partners to develop and deliver the All Wales
Academy e-learning site and part of the project requires a Return on Investment (ROI)
exercise.
This involves exploring the types of training (and the costs involved) that were in place
before the implementation of the AWA and comparing these with the actual (or expected)
training/costs going forward. This questionnaire focuses on Data Protection and Information
Governance training rather than all training.
Consequently, I would kindly ask if you could provide any data you have available on the
following:
1. Can you list the different Data Protection and Information Governance Courses
offered during the period 1.4.14-31.3.15
2. Using the list of Data Protection and Information Governance training courses you
provided in question 1. Can you list how many people undertook each course between
1.4.14 and 31.3.15?
3. For each course listed in question 1, how long (in hours) was the course?
4. What was the cost of each course listed in question 1, related to Data Protection and
Information Governance?
5. Can you identify if any of the courses listed in question 1 were accredited?
6. All Wales Academy is developing a selection of courses covering Data Protection and
Information Governance, as listed below. We anticipate that some local authorities will use
AWA as part of a blended training approach for Data Protection and Information
Governance. As such, they may reduce their face to face contact training time in this area
or cease to purchase specific Data Protection training e.g. Bob’s Business. Based on your
potential plans for blended learning with AWA in Data Protection and Information
Governance, can you forecast how much it will cost your organisation to train people in this
area over a 12 month period (excluding any contribution to AWA)?
• Sharing Personal Information (already available via AWA)
• Data Protection Awareness (available on AWA w/c 20.7.15)
17
• Freedom of Information - An Introduction (available on AWA w/c 20.7.15)
• Protection of Electronic Communication Regulations (available on AWA w/c 3.8.15)
• Introduction to Information Governance (available on AWA w/c 31.8.15)
7. The AWA board are currently considering the benefits of developing an Assessment
Centre through AWA, allowing Local Authorities to make savings from their individual
assessment centre status (if they wished) and instead operate as satellite centres through
the AWA assessment centre registration. This would allow Local Authorities to still manage
their learners and assessments individually but make economies of scale for centre
approval, registration and certification. In order to calculate projected return on investment
of such an approach, we have decided to focus on the two main areas accredited by local
authorities – Leadership & Management and Health & Social Care. Therefore, can you
please provide:
i. Name of the awarding body your local authority is currently registered with for
Leadership & Management (e.g. ILM or other awarding bodies) ________________ &
annual cost for their Approved Centre Status __________________
ii. Name of the awarding body your local authority is currently registered with for
Health & Social Care (e.g. City & Guilds or other awarding bodies) ________________ &
annual cost for their Approved Centre Status __________________
iii. Number of learners registered & accredited through your centre (1.4.14-31.3.15) for
ILM qualifications _____________ and cost of this (including VAT)_____________________
iv. Number of learners registered & accredited through your centre (1.4.14-31.3.15) for
Health & Social Care qualifications______________ and cost of this_______________
Many thanks for your time. The information you have provided will be used to provide
projected return on investment figures for AWA.
18
Appendix 3 – data on the Data Protection and Information Governance courses collected
from LAs using the ROI questionnaire (Appendix 2)
Area Type of training Number
of
people
attended
per
course
Total
number
of
people
attended
Length of
training
session
Total cost Accredited
(Yes/No)
Blaenau
Gwent
Advanced Fee Frauds 373
Backing up Data 467
Business Continuity 293
Carefully Classified Bus. 540
Data Protection 726
EMail Errors 701
Email Etiquette 570
Freedom of Information 648
GCSX 545
Identity Theft 533
ISO 27001 386
Keeping It Clear 692
Mobile Working 348
PCI DSS 682
Perfect Passwords 748
Phishing Fears 441
Protected Premises 649
Risk Management V2 291
Virus Vigilance 516
19
Web Woes 406
1055 0.2 hours
(all
eLearning)
£9000 No
Gwynedd
DATA PROTECTION
(FACE TO FACE
TRAINING) 29 courses
304 1.5 hours No training
costs
identified
No
DATA PROTECTION (e-
LEARNING MODULE) –
recently produced
24 0.5 hours Learning Pool
essential
service
£16,000 per
annum (excl
VAT)
Shibboleth
Active Sync
Directory
£3,000 per
annum (excl
VAT)
No
328
Cardiff CCTV
Data Protection
The Principles of Data Protection
Email Errors
Email Etiquette
Freedom of Information
Keeping it Clear
PCI
20
Perfect Passwords
Web Woes
Virus Vigilance
Secure Printing
Protected Premises
Phishing Fears
Mobile Working
Identity Theft
5126 0.14 hours
(all
eLearning)
£12,375 BIS
ISO27001
compliant
Wrexham Information Security Law
for Public Sector
Managers, Auditing and
Data Protection
13
6 hours £1,900 No
National Archives
Training
27 6 hours 0 No
40
Denbighshire
Data Protection
eLearning
1300 0.5hours £18,000 No
Data Protection
(bespoke - classroom
training)
200
4 hours £1,200 No
FoI and EiR eLearning 200 0.33 hours Cost not
identified
No
1700
21
Neath-Port
Talbot
No training provided
Caerphilly Protection Information
eLearning - mandatory
every year for all PC
users
2800 1.5hours No
Sharing Personal
Information eLearning
200 0.5hours No
Information Governance
3 courses
73 3 hours No
Protecting Information 809 1.5hours No
Freedom of Information 209 1 hour No
Records management 27 7.5 hours No
Record management
shorter course
263 2 hours No
Data Protection and
CCTV
50 1.5 hours No
Data Protection training 80
FOI training 80
4591
£37,700
Vale of
Glamorgan
Data Protection - face to
face
48 4 hours Cost not
identified
No
Data Protection -
eLearning
42 0.5 hours Cost not
identified
No
90
22
Appendix 4 – data on the Leadership & Management and Health & Social Care courses
collected from LAs using the ROI questionnaire (Appendix 2)
Area and course Number
of
people
£ per
award
Total
cost
Cardiff - City and Guilds
Awards Registered for 2014-15
HSC Diploma Level 2 (Adults) 26 £92 £2,392
HSC Diploma Level 3 (Adults) 19 £110 £2,090
HSC Diploma Level 3 (CYP) 3 £110 £330
HSC Leadership Diploma (Adult Advanced Practice) Level 5 8 £149 £1,192
HSC Leadership Diploma (Adult Manager) Level 5 3 £149 £447
HSC Leadership Diploma (Adult Residential) Level 5 3 £149 £447
HSC Leadership Diploma (CYP Advanced Practice) Level 5 4 £149 £596
Cardiff - ILM 10 £75 £750
Gwynedd - ILM 28 £68 £1,908
Gwynedd - City and Guilds 34 £112 £3,808
Wrexham - City and Guilds 126 £76 £9,582
Wrexham - TMDP Oxford Brookes 3 £2,750 £8,250
Wrexham - ILM 20 £450 £9,000
23
Denbighshire - ILM
Level 3 16 £106 £1,696
Level 5 5 £170 £850
Denbighshire - Health & Social Care qualifications (Anterth Mon in collaboration with
Conwy, Mon and Gwynedd
approx 30-50, at £350-£500 per head 40 £425 £17,000
(costs are likely to rise in the future for age over 25 for level 3 could be £2000 per head)
Neath-Porttalbot - City and Guilds 100 £108.80 £10,180
Caerphilly - City and Guilds 45 £189 £8,500