Upload
nguyendiep
View
217
Download
0
Embed Size (px)
Citation preview
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 2 | Coralesce Ltd wwwcoralescecom
Review of Learning Technology Self-Assessment
Baseline Report
July 2014
Contents
Page
1 Introduction 3
2 About Us 3
3 Policy Context 4
4 Executive Summary 5
5 Research Findings 11
51 Organisational Self-Assessment Frameworks 11
52 Individual Self-Assessment Competency Frameworks 25
6 Comparative Analysis and Evaluation of Organisational and Individual Self-Assessment Competency Frameworks 29
7 Conclusions 33
8 Next Steps 35
Appendix 1 e-Learning Quality Frameworks 37
Appendix 2 Comparative Overview of Framework Themes 38
Appendix 3 Glossary 39
Appendix 4 Bibliography 40
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 3 | Coralesce Ltd wwwcoralescecom
1 INTRODUCTION This baseline report provides an analysis of existing learning technology self-assessment
tools used in industry schools further education and higher education both nationally and
internationally The research was undertaken to inform the Learning Technologies Self-
Assessment Project undertaken by Coralesce Ltd on behalf of the Education and Training
Foundation The aims of this project are to
produce a self-assessment tool for the FE and Learning and Skills sector which
builds on the work supported by the Learning and Skills Improvement Service
(LSIS) to create the Review and Planning for Technology in Action RAPTA tool
create a self-assessment framework that will lsquoprogress an organisationrsquos use of
learning technologies and help education and training providers respond to the
FELTAG recommendationsrsquo
(Tender Specification Learning Technologies Self-Assessment Education and Training Foundation
April 2014)
The research took place between 16th June and 13th July 2014 and involved desk-based research analysis and evaluation of a number of self-assessment tools and digital literacy ICT competence frameworks The frameworks analysed fell into three broad categories
Organisational self-assessment
Individual self-assessment
Quality assurance frameworks for e-learning
The focus of this report is on the analysis and evaluation of the organisational and
individual self-assessment tools However the information on the quality assurance
frameworks for e-learning provides a useful resource for the sector and details of these are
documented in Appendix 1
The self-assessment and competency frameworks examined have been evaluated by sector
and by criteria using five main criteria suitability for the sector impact on teaching and
learning suitability for self-assessment quality improvement and sustainability In
addition a comparative analysis has been undertaken on assessment areas used within each
tool and methods for structuring questions This analysis has been designed to inform the
next stage of the Project which seeks to consult with the sector on trialling a revised self-
assessment tool As a resource to inform the next stage of consultation a compilation of
questions has been collated under each theme into a question bank
2 ABOUT US Coralesce Ltd has been commissioned by the Education and Training Foundation to
undertake the Learning Technologies Self-Assessment Project from July 2014 ndash January
2015 Coralesce Ltd is a curriculum development and project management agency that
works with awarding organisations educational providers and funding agencies to develop
enterprising and innovative new learning journeys to inspire a 21st century workforce At
Coralesce Ltd our aims are to
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 4 | Coralesce Ltd wwwcoralescecom
bring people together to grow ideas to improve education development and training
undertake research on practice to inform developments
provide innovative solutions to transform learning journeys
advance the promotion of equal opportunities
enable people to use technology to share learning
provide a high-quality shared curriculum management service
The Learning Technologies Self-Assessment Project builds on research work already
undertaken by Coralesce Ltd for the Education and Training Foundation from January ndash
March 2014 Coralesce Ltd undertook a strategic consultation on technology in teaching and
learning to inform the Foundation on the sector priorities on learning technologies
3 POLICY CONTEXT This development of a new self-assessment tool for the sector has taken place against a
backdrop of an emerging national debate about the future use of technology in learning with
the work of FELTAG This was set up in January 2013 by Matthew Hancock Minister of
State for Skills and Enterprise in BIS as a sector group to make practical recommendations
aimed at ensuring the effective use of digital technology in learning teaching and
assessment in Further Education and Skills On 28th February 2014 the FELTAG Group
published its recommendations across six work streams
Horizon-scanning
Investment and Capital Infrastructure
Regulation and Funding
Workforce capacity
Employers
Learners
In its workforce stream the FELTAG report highlighted the need to develop the whole
workforce calling for lsquosignificant investment in the knowledge skills and understanding of
the learning technologyrsquos potential among policy-makers governors principals senior and
middle management teachers and support staffrsquo One of the key workforce
recommendations was the need to develop a revised self-assessment tool to enable the
workforce to evaluate progress as well as a call for benchmarks to be established
The Government Response (BIS 2014) to FELTAG has endorsed the recommendation that
the entire workforce needs to be brought up to speed to fully understand the potential of
learning technology The recently published Governmentrsquos FE Workforce Strategy BIS
(2014) sets one of its key priorities as the effective use of technology in teaching and
learning ldquoThe objective is that FE providers universities and industry should
collaboratively put in place up-to-date and relevant professional development and initial
training for managers teachers and trainers that is aimed specifically at improving their
knowledge of and confidence in using learning technologyrdquo The development of an
effective tool to support individuals and institutions to measure their performance on the
use of technology in learning has therefore never been so crucial
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 5 | Coralesce Ltd wwwcoralescecom
4 EXECUTIVE SUMMARY This research has been undertaken on behalf of the Education and Training Foundation to
inform its Learning Technologies Self-Assessment project The project aims to produce a
self-assessment tool for the education and training sector which builds on the most recent
work undertaken by the precursor organisation LSIS (Learning and Skills Improvement
Service) This work resulted in the production of a self-assessment tool entitled the Review
and Planning for Technology in Action (RAPTA) tool
The revised tool also needs to build on best practice in other sectors and this baseline report
seeks to explore the range of practice and approaches adopted in other frameworks This
analysis is not an exhaustive list of all the tools available but rather a practical overview to
inform the next steps in developing an appropriate and contemporary Learning
Technologies Self-Assessment tool for the sector
The frameworks analysed fall into three broad categories Organisational self-assessment
Individual self-assessment and Quality assurance frameworks for e-learning This report
provides an analysis of the organisational and individual self-assessment and competency
frameworks from industry schools and further and higher education These are then
evaluated against the following criteria by sector and by criteria
Suitability for the Sector
Impact on Teaching and Learning
Suitability for Self-assessment
Quality improvement and
Sustainability
1 Suitability for whole sector
This criterion focuses on whether the self-assessment tool is suitable for all types of
education and training providers in the sector The state funded education and training
sector is a lsquobroad churchrsquo made up of Further Education Colleges 6th Form Colleges Private
training providers Adult and Community Learning providers Prison Education Service and
Unionlearn
2 Impact on Teaching and Learning
This criterion focuses on whether the tool measures the effective use of learning
technologies in teaching and learning and addresses outcomes for learners
3 Suitability for Self-assessment
This criterion considers whether the tool is robust for the purposes of self-assessment and
considers whether it is informed by relevant standards such as
New Professional Standards for teachers
Ofsted Common Inspection Framework
Digital literacy expectations for practitioners and students
Suitability for self-assessment also involves enabling all relevant parties to self-assess and
plan for improvement whatever their competence confidence level such as
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 6 | Coralesce Ltd wwwcoralescecom
Practitioners
Curriculum managers
ICT managers
Quality managers
Senior managers
Governors or equivalent
Employers
The tool needs to be able to be integrated successfully with the collegersquos existing self-
assessment and quality improvement processes Accessibility ease of use and length of the
tool also impacts on its suitability for self-assessment
4 Quality improvement
Effective self-assessment should lead to quality improvement This criterion considers
whether the tool effectively supports quality improvement by either supporting
development via easy to use links to resources ideas best practice exemplars training
materials and or eliciting quantitative data for benchmarking To ensure quality
improvement is appropriate and effective the tool needs to ensure that the data collected is
accurate and this may include strategies to ensure rigour
5 Sustainable
This criterion considers whether the tool is sustainable and capable of supporting
continuous quality improvement in the use of learning technologies for the sector in the
years to come
SECTOR ANALYSIS
The 12 tools were evaluated by sector as follows
School Sector Self-Assessment Frameworks
The Naace Self Review Framework (SRF) provides a framework for schools which is
designed to support them to review the use of technology in a structured way and to support
school improvement through reflective practice The framework links to the schools Self
Evaluation Framework (SEF) and therefore captures evidence for the annual self-evaluation
and for Ofsted inspections The tool focuses on ICT as well as Learning Technology but
also includes a set of statements in the section on developing pupilsrsquo digital literacy
The Washington State Technology Essential Conditions Rubric evaluates
institution progress against the Washington State Educational Standards the Essential
Academic Learning Requirements The rubric is relatively short and grades institution as
beginning progressing or proficient The rubric looks at five key aspects Vision planning
and policy technical literacy for learners integration standards for teachers technical
competencies of administrators communications and connections
DigiLit Leicester is a digital literacy framework developed as a result of lsquoBuilding Schools
for the Futurersquo ICT priorities It assesses competency against six strands of digital literacy
in the secondary school sector It puts learning and learners at the centre with a very clear
focus on outcomes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 7 | Coralesce Ltd wwwcoralescecom
FE and Skills Self-Assessment Frameworks
The Development Self Audit (DSA) was undertaken by Scottish Colleges as part of a
baseline study of e-activity in 2007 It is based on practical activities and adapted from
research by Dr Stephen Marshall who developed the eLearning Maturity Model for higher
education institutions in New Zealand The self-audit reviews five aspects Learning
Development Support Evaluation and Organisation As it is based on the eMM it also
maps the aspects against five dimensions of capability Policy Planning Delivery
Management Improvement These are then mapped to produce a management level e-
maturity grid
eQSF A Quality Standards Framework for ICT in Learning was developed using
European funding and piloted in North Ireland in 2009 10 It provides a comprehensive
evaluation and action planning tool consisting of thirteen questions covering three pillars
Values Pedagogy and Practice eStructure of the Learning Process Leadership and
Management The online tool facilitates self-assessment and results in action planning for
improvement
ACL e-Learning Positioning Statement was developed by the National Institute for
Adult and Continuing Education (NIACE) and the Centre for Excellence in Leadership
(CEL) in 2005 It was designed to support institutional strategic planning in e-learning and
institutional quality improvement processes It focused on five core strands Vision and
Strategic Planning Teaching and Learning Staff Development Infrastructure and
Managing ILT and e-Learning
Generator was a self-assessment tool developed through a partnership between LSN Jisc
and BECTA in 2008-2011 Whilst complex it enabled learning providers to identify the
impact that technology had in 9 key business processes against seven enablers of
technology The tool was evaluated against a set of outcome based questions and was
capable of reviewing two or more curriculum areas within an organisation
Review and Plan Technology in Action (RAPTA) was developed by Jisc in 2012-13
with support from Learning and Skills Improvement Service linked to the Ofsted Common
Inspection Framework It aimed to provide a structured framework which can be integrated
in to an organisationrsquos self-assessment process The tool was lengthy with questions
focused on teaching and learning and leadership and management A shorter quick review
version was also available
Higher Education Self-Assessment Frameworks
e-Learning Maturity Model is a mature evaluation framework developed by Dr Stephen Marshall and provides a quality improvement framework for HE institutions in New Zealand It is a complex system focused on online and blended learning The tool benchmarks e-learning capability using 35 inter-related processes It supports external assessment rather than self-assessment as it is externally validated
Industry Self-Assessment Frameworks
Toward Maturity is a mature self-assessment and benchmarking tool which is straightforward but lengthy to use Designed to promote reflection it uses a set of KPIs and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 8 | Coralesce Ltd wwwcoralescecom
identification of best practice to provide a sophisticated benchmarking tool Benchmarking reports are based on 9 KPIs and measure against three levels beginning progressing and proficient
Cross Sector Frameworks
The Open University Digital Literacy Framework identifies the competencies for
learners at HE levels 0123 to M (Level 0 equates to FE Level 3) across five aspects of
competence It provides a self-assessment tool which is presented in short sections The
framework could be used as a basis for individual self-assessment for both learners and
practitioners
The UNESCO Competency Framework for Teachers is an international framework
that supports teachers to develop students to become collaborative problem solving
creative learners It has three aspects Technology Literacy Knowledge Deepening
Knowledge Creation
ANALYSIS BY CRITERIA
The 12 tools were evaluated by criteria as follows
1 Teaching and Learning The school frameworks reviewed have a strong focus on outcomes for learners They use a similar model and focus on classroom based delivery but do not reference online or workplace learning In some cases the positioning of teaching and learning is overshadowed by a focus on leadership and management Apart from the DigiLit Leicester framework the school based tools omit to review the use of technology to support assessment monitoring and feedback The most recent FE and Learning and Skills frameworks eQSF and RAPTA sought to have teaching as a central theme RAPTA has a twin focus on inspiring and challenging learners and encouraging independent learning The eQSF framework however describes processes rather than outcomes The UNESCO framework applies to all sectors focuses on the development of teachers and has desired outcomes for learners as a central feature The Washington State Technology Essential Conditions Rubric has a similar focus Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners Whilst the HE framework eMM focuses on online learning it has a narrow focus on teaching and learning
2 Suitability for Self-assessment The scope of the varying frameworks is similar generally covering core questions on pedagogy outcomes for learners strategic planning and professional development for practitioners The extent to which the frameworks engage with the full range of stakeholders is varied some encouraging completion by one manager within the organisation as in Towards Maturity which can reduce the rigour of the findings In contrast others encourage involvement of learners teachers and managers in the process eg RAPTA and eQSF thus providing a more robust approach to self-assessment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 9 | Coralesce Ltd wwwcoralescecom
3 Quality Improvement A number of organisational frameworks focus on quality improvement such as the Naace and QSF frameworks which provide an on-line action planning tool Other frameworks including Naace provide sources of high quality support materials as does the Open University Digital Literacy Framework
4 Suitability for the whole Sector The tools that link to the Common Inspection Framework could be adapted for use across the sector however they would need adapting to encompass the range of settings represented across the sector particularly workplace learning Frameworks which focus on individual teacher competency such as UNESCO ICT Framework and the DigiLit Leicester would require limited adaptation for use in the FE and Skills Sector
5 Sustainability The tools that have demonstrated the greatest sustainability are the Towards Maturity tool and the Naace tool which have either attracted industry sponsorship to support its annual use over a 10 year period (Towards Maturity) or incorporate small charges for registration and use (Naace)
THEMATIC ANALYSIS
A thematic analysis highlights the question areas that are considered within each tool
covering strategic leadership infrastructure teaching and learning learner experience
staff development financial management and quality improvement All the 12 tools use
structured questions some have dichotomous questions (YES NO) whilst others are based
on levels of measurement (Likert Response Scaling) (Strongly Agree to Strongly Disagree)
or Cumulative Scaling (1-4 or Descriptive Names that indicate a Journey)
In reviewing the frameworks the following conclusions have been drawn which will be
tested at the next stage of consultation within the Learning Technologies Self-Assessment
Project to enable the development of a robust and valued self-assessment tool that builds on
previous work Our conclusions from this desk research are that a self-assessment tool
should
Be explicitly linked to how learning technologies support teaching and learning
Facilitate access by a range of stakeholders by being concise and designed to support
differing approaches to self-assessment
Use structured questions on a meaningful cumulative scale
Model guide and support good self-assessment practice
Provide effective support for quality improvement through for example providing
automatically generated action plans
Be sustainable for the sector once grant funding has ceased
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 10 | Coralesce Ltd wwwcoralescecom
Comparison of the themes used in Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 11 | Coralesce Ltd wwwcoralescecom
5 RESEARCH FINDINGS
A range of frameworks were identified as part of the research and these broadly fell into three categories
Organisational self-assessment
Individual self-assessment
Quality assurance frameworks for e-learning
The major focus of the Education and Training Foundationrsquos Learning Technology Self-Assessment Project is to develop an organisational tool with the potential to integrate individual practitioner self-assessment and therefore the summary below focuses on the first two types of framework Links to quality assurance framework for e-learning are provided in Appendix 1
METHODOLOGY
The methodology used in this Baseline Report was desk-based research to identify a range of self-assessment tools and quality frameworks covering schools further education and learning and skills higher education and industry Whilst the research provided a valuable and comprehensive overview of organisational and individual self-assessment tools and quality frameworks for e-learning it has not always been possible to access full question banks for all tools This is due to the fact that once an organisation responsible for a tool is closed not all archives are kept In the case of BECTA their self-assessment tool Generator was web-based and much of the material including the full question bank was not fully archived at their demise In addition the full set of questions used by Towards Maturity has been difficult to access due to the commercial nature of the organisation Where access has been impeded the research team have sourced a sample of questions through secondary literature
51 Organisational Self-Assessment Frameworks
Naace Self Review Framework (SRF) Sector Schools
Country England
Year 2014
A self-review framework designed to support schools to review their use of technology in a structured way and to support school improvement through reflective practice Developed against a well-researched and evidence based set of criteria Self-Review Framework (SRF) is a mature system originally developed by BECTA and now managed and updated annually by Naace
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 12 | Coralesce Ltd wwwcoralescecom
Naace is a membership organisation and provides an online tool which is accessible only to
member organisations (costs pound100 to join Naace and a further pound50 to access the online
tool) Members are well supported with training in the use of SRF and a useful website A
PDF version of the criteria is free to use
The SRF focuses on supporting pupils to become digitally literate and able to express
themselves and develop ideas through the use of ICT The tool is broad in its approach and
has less of an overall focus on teaching learning and assessment compared to the RAPTA
tool As this is a school based system it has a classroom based focus and does not reflect
the use of technology to support assessment monitoring and target setting
The framework links to the schools Self Evaluation Framework (SEF) and therefore
captures evidence for the annual self-evaluation and for Ofsted inspections The tool
focuses on ICT as well as Learning Technology but has a set of statements in the section on
developing digital literacy which could be adapted The framework consists of
Leadership and Management
Teaching and Learning
Professional Development
Resources
Each section is expanded into a number statements (54 in total) each of which is graded 1
to 4 against a clear set of grade descriptor (figure 1) Whilst the tool is long it is easy to
understand and designed to be used by different staff roles
The online tool provides clear guidance on each aspect and an opportunity to record
evidence and identify actions to improve
The online tool also provides a clear graphical representation of organisational capability
(figure 2) and although not currently benchmarked could be adapted to provide benchmark
data This process would lead to production of a detailed and specific plan for
improvement
The standards section links to an ICT Mark which recognises e-maturity The Mark is
externally assessed and verifies the judgements made in the SRF Following on from the
ICT Mark schools have an opportunity to apply for a Third Millennium Learning Award a
peer-referenced award which demonstrates that schools are providing education fit for the
21st century
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 13 | Coralesce Ltd wwwcoralescecom
Figure 1 Example Question from online SRF Tool
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 14 | Coralesce Ltd wwwcoralescecom
Figure 2 Graphical summary of outputs from online tool
Level Aiming For Enhancing Learning through ICT - level 3
1 Leadership and Management 1a-
1
1a-
2
1b-
1
1b-
2
1b-
3
1b-
4
1b-
5
1b-
6
1c-1 1c-2 1c-3 1c-4
3 4 X 4 3 4 3 3 X X X X
2 Use of ICT in the curriculum 2a-
1
2a-
2
2a-3 2a-4 2a-5 2a-6 2a-7 2b-
1
2b-
2
2b-
3
2b-
4
X X X X X X X X X X X
3 Teaching and Learning 3a-
1
3a-
2
3a-3 3a-4 3a-5 3b-
1
3b-
2
3b-
3
3b-
4
3b-
5
3b-
6
3b-
7
3b-
8
3b-
9
X X X X X X X X X X X X X X
4 Assessment of digital
capability
4a-
1
4a-
2
4a-3
X X X
5 Professional Development 5a-
1
5a-
2
5a-3 5a-4
X X X X
6 Resources 6a-
1
6a-
2
6a-3 6a-4 6a-5 6a-6 6b-
1
6b-
2
6b-
3
6b-
4
X X X X X X X X X X
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 15 | Coralesce Ltd wwwcoralescecom
Washington State Educational Technology Standards and Technology Essential
Conditions Rubric
Sector School
Country USA
Year 2008
A very well constructed model based on research on the benefits and dangers of technology
The standards put the learner at the heart of the system focusing on their expectations
Developed for schools as with the Naace system in England it is classroom focused The
Washington State materials consist of standards for learners and for teachers and are
supplemented with a Technology Essential Rubric for schools to evaluate progress at an
institutional level
The standards emanated from state legislature (1997) which established common learning
goals for all students in Washington State which provided a foundation for the
development of content called Essential Academic Learning Requirements (EALR)
Educational Technology was added to the EALR in 2007 and articulates high expectations
for what students should be able to do across all subject areas The standards for
educational technology were categorised as
Technology Integration
Digital Citizenship (technical literacy moving to technical fluency)
The standards are detailed albeit a little out of date with each EALR broken down into
components each with a grade level expectation for each year of education up to K12 (see
figure 3) Each grade level expectation has a descriptor of evidence of learning As the
standard requires use across subject areas it also provides examples across different subject
areas and levels including basic and 21st century learning environment The standards
provide detailed and useful materials to support teachers and curriculum areas to evaluate
and improve practice They could also be adapted for learner self-evaluation
The self-assessment tool the Technology Essential Conditions Rubric for Schools is
relatively short (8 pages) but would support effective self-assessment and has the potential
for benchmarking It uses a simple grading system beginning progressing proficient and
looks at five key areas
Vision planning and policy
Technical Literacy for learners ndash related to EALRs
Integration standards for teachers ndashrelated to EALRs
Technical competencies of administrators
Communication and Connections
Integration standards for teachers are described in three tiers Teacher Focus on
Productivity Instructional Presentation and Student Productivity Powerful Student-
Centred 21st century Learning Environment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 16 | Coralesce Ltd wwwcoralescecom
EALR 2 mdash DIGITAL CITIZENSHIP
Students demonstrate a clear understanding
of technology systems and operations and
practice safe legal and ethical behavior
Component 21 Practice Safety
Demonstrate safe legal and ethical behavior
in the use of information and technology
GLE Grade 6
212 Practice ethical and respectful behavior
Evidence
of
Learning
Comply with district Acceptable Use Policy (AUP)
Practice ethical and appropriate use of all media and comply with copyright law
Participate and engage with the global community within expected norms of behavior and positive interaction
Recognize report and
counteract cyberbullying
Examples
Recognize different types of illegal and unethical technology use
Identify and cite Creative Commons images in online presentations
Cite all sources properly
Observable indicators are provided for each tier together with helpful examples these could
be useful for practitioner level self-assessment Teachers are supported with extensive
materials available from the Washington State Office of the Superintendent of Public
Instruction wwwk12waus curriculuminstruct
Figure 3 An example of EALR for digital citizenship is shown below
Required
An Essential Academic Learning Requirement (EALR) is a broad statement of the learning that applies to Grades Kndash12 The Component is a statement that further defines and provides more specific information about the EALR There is at least one component for each EALR The Grade Level Expectation (GLE) is a statement containing the essential content or process to be learned and the cognitive demand required to learn it A revision of Bloomrsquos Taxonomy was used to categorize the cognitive demand required of the student Each GLE includes evidence of learning statements which are considered essential to the GLE
The GLE Numbering System identifies the EALR the component and the GLE in that order In the example at right the number 2 indicates the EALR the number 21 indicates the component and the number212 indicates the GLE Note Grade levels are not referenced in the numbering system The Evidence of Learning is a bulleted list of ways students can demonstrate learning considered essential to the GLE Educators are encouraged to identify additional ways in which the student can show proficiency as the educational technology GLEs are integrated across the curriculum Suggested The Examples provide specific illustrations of the learning However these examples are not exhaustive and educators are encouraged to find multiple ways by which learners can demonstrate what they know
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 17 | Coralesce Ltd wwwcoralescecom
Development Self Audit (DSA)
Sector Further Education
Country Scotland
Year 2007
Undertaken by Scottish Colleges as part of a baseline study of e-activity in 2007 the DSA
assessed e-maturity on a broad basis The development was funded by the Scottish Funding
Council and was informed by the practical activities and research of Dr Stephen Marshall
who developed an ideal model of e-maturity (eLearning Maturity Model)
The questions in the DSA cover five aspects
Learning (processes that impact on pedagogical aspects of e-learning)
Development (processes that surround the creation and maintenance of resources)
Support (processes surrounding oversight and management)
Evaluation (processes surrounding the evaluation and quality control)
Organisation (processes associated with planning and management)
As the DSA is based on the eMM developed in New Zealand it also maps the aspects against
the five dimensions of capability
Policy
Planning
Delivery
Management
Improvement
Using external assessment this process leads to the development of a grid which provides a
management level view of the e-maturity of a college an example is shown in figure 6
Figure 6 Management Level E-Maturity Grid
In addition to the completion of the grid the Scottish Funding Council also gathered
feedback on specific issues through the use of a questionnaire completed by institutions and
individuals which looked at the following themes
DSA
Po
licy
Pla
nn
ing
Delivery
Man
ag
em
en
t
Imp
rovem
en
t
Learning
L1 e-Learning is designed according to pedagogically sound principles N P L F P
L2 Learning skills are developed explicitly amp incrementally during delivery P F P P P
L3 Peer amp tutor interactions are provided explicitly during e-learning delivery P P N P P
L4 Courses take account of student workload and e-assessment demands P P N L P
L5 e-learning actively engage students P P P P N
L6 College defined (optional) na na na na na
Development
D1 Course development is directed and supported P P L F P
D2 Resources are maintained and re-used P F P P P
D3 Courses meet the needs of all students P P N P P
D4 The e-learning infrastructure is reliable robust and sufficient P P N L P
D5 The e-learning infrastructure is standardised and integrated P P P P N
D6 College defined (optional) na na na na na
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 18 | Coralesce Ltd wwwcoralescecom
Infrastructure
Teaching and Learning
Organisation
Strategic Leadership
e-Learning coverage by department subject area
The questions covered the extent to which e-learning was used across levels type of
learning and to what extent it was used in Work based Learning An important aspect that
the questionnaire covered under the heading of strategic leadership was the financial
aspects including affordability of software implementation and cost benefit analysis
eQSF A Quality Standards Framework for ICT in Learning
Sector Learning and Skills
Country Northern Ireland
Year 2009 10
eQSF was developed using European funding and provides a comprehensive evaluation and
action planning package to assess practice in the use of existing and new technologies The
framework consists of three pillars each with a series of standards and examples of
evidence
Values Pedagogy and Practice (see figure 4)
Structure of the Learning Process
Leadership and Management
The pillars break down into 13 quality standards each with a separate on-line template to
complete Each standard is graded on a scale of attainment from Absent to Advanced
However the statements tend to describe processes rather than outcomes and are quite
broad particularly in the section on Teaching and Learning
The online resource facilitates the self-assessment process provides opportunities to
identify evidence and results in an action plan for improvement The implementation
process involves establishing leadership and ownership reviewing practice action
planning implementing action plan evaluating progress and feedback
Positive feedback is available from the three pilot case studies that were developed
including North West College in NI which indicate the tool has been used at teacher
department and organisational level
http wwweqsfeu
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 19 | Coralesce Ltd wwwcoralescecom
Figure 4 Values Pedagogy amp Practice Standards
Standard Addressed
Yes No
Actions
Completed Current Attainment Level Signature
11
The organisation supports inclusiveness and accessibility through the use of ICT and sees ICT as promoting equality and diversity in the learning process
Absent Acquiring Achieved Advanced
out of
12
The organisation facilitates the adaptation of learning materials to meet a range of learner requirements and sees ICT as a mechanism for delivering such flexibility for the practitioner and learner
Absent Acquiring Achieved Advanced
out of
13
The organisation uses ICT to extend learning beyond the classroom and supports progression to further learning and career opportunities
Absent Acquiring Achieved Advanced
out of
14
The organisation is committed to the learnersrsquo outcomes being positively impacted by ICT
Absent Acquiring Achieved Advanced
out of
Comment
Adult and Community Learning e-Learning Positioning Statement
Sector Adult and Community Learning
Country England
Date 2005-2007
The ACL eLPs were designed by CEL (Centre for Excellence in Leadership) and NIACE
(National Institute for Adult amp Continuing Education) as an aid to both institutional
strategic planning in e-learning and to support institutional quality improvement processes
As well as its use within NIACEs e-learning management and leadership courses the tool
was also embedded in individual consultancies and within the NIACE Digital Learning
teams capital investment funding work The ACL eLPs tool was superseded by BECTAs
Technology Improvement Leadership Tool - Generator NIACE formed part of the
development team for Generator and continued to contribute to its development and
piloting with community-based adult learning providers The use of the Generator tool was
integrated into NIACEs e-learning capital funding projects (CAMEL) The e-Learning
positioning statement comprised 31 elements grouped under five themes Organisations
graded themselves against a five point scale ranging from not yet started to embedded The
ACL eLearning Positioning Statement was designed as an aid to both institutional strategic
planning in e-learning and to support institutional quality improvement processes
The framework reviewed Vision and Strategic Planning Teaching and Learning Staff
Development Infrastructure and Managing ILT and e-Learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 3 | Coralesce Ltd wwwcoralescecom
1 INTRODUCTION This baseline report provides an analysis of existing learning technology self-assessment
tools used in industry schools further education and higher education both nationally and
internationally The research was undertaken to inform the Learning Technologies Self-
Assessment Project undertaken by Coralesce Ltd on behalf of the Education and Training
Foundation The aims of this project are to
produce a self-assessment tool for the FE and Learning and Skills sector which
builds on the work supported by the Learning and Skills Improvement Service
(LSIS) to create the Review and Planning for Technology in Action RAPTA tool
create a self-assessment framework that will lsquoprogress an organisationrsquos use of
learning technologies and help education and training providers respond to the
FELTAG recommendationsrsquo
(Tender Specification Learning Technologies Self-Assessment Education and Training Foundation
April 2014)
The research took place between 16th June and 13th July 2014 and involved desk-based research analysis and evaluation of a number of self-assessment tools and digital literacy ICT competence frameworks The frameworks analysed fell into three broad categories
Organisational self-assessment
Individual self-assessment
Quality assurance frameworks for e-learning
The focus of this report is on the analysis and evaluation of the organisational and
individual self-assessment tools However the information on the quality assurance
frameworks for e-learning provides a useful resource for the sector and details of these are
documented in Appendix 1
The self-assessment and competency frameworks examined have been evaluated by sector
and by criteria using five main criteria suitability for the sector impact on teaching and
learning suitability for self-assessment quality improvement and sustainability In
addition a comparative analysis has been undertaken on assessment areas used within each
tool and methods for structuring questions This analysis has been designed to inform the
next stage of the Project which seeks to consult with the sector on trialling a revised self-
assessment tool As a resource to inform the next stage of consultation a compilation of
questions has been collated under each theme into a question bank
2 ABOUT US Coralesce Ltd has been commissioned by the Education and Training Foundation to
undertake the Learning Technologies Self-Assessment Project from July 2014 ndash January
2015 Coralesce Ltd is a curriculum development and project management agency that
works with awarding organisations educational providers and funding agencies to develop
enterprising and innovative new learning journeys to inspire a 21st century workforce At
Coralesce Ltd our aims are to
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 4 | Coralesce Ltd wwwcoralescecom
bring people together to grow ideas to improve education development and training
undertake research on practice to inform developments
provide innovative solutions to transform learning journeys
advance the promotion of equal opportunities
enable people to use technology to share learning
provide a high-quality shared curriculum management service
The Learning Technologies Self-Assessment Project builds on research work already
undertaken by Coralesce Ltd for the Education and Training Foundation from January ndash
March 2014 Coralesce Ltd undertook a strategic consultation on technology in teaching and
learning to inform the Foundation on the sector priorities on learning technologies
3 POLICY CONTEXT This development of a new self-assessment tool for the sector has taken place against a
backdrop of an emerging national debate about the future use of technology in learning with
the work of FELTAG This was set up in January 2013 by Matthew Hancock Minister of
State for Skills and Enterprise in BIS as a sector group to make practical recommendations
aimed at ensuring the effective use of digital technology in learning teaching and
assessment in Further Education and Skills On 28th February 2014 the FELTAG Group
published its recommendations across six work streams
Horizon-scanning
Investment and Capital Infrastructure
Regulation and Funding
Workforce capacity
Employers
Learners
In its workforce stream the FELTAG report highlighted the need to develop the whole
workforce calling for lsquosignificant investment in the knowledge skills and understanding of
the learning technologyrsquos potential among policy-makers governors principals senior and
middle management teachers and support staffrsquo One of the key workforce
recommendations was the need to develop a revised self-assessment tool to enable the
workforce to evaluate progress as well as a call for benchmarks to be established
The Government Response (BIS 2014) to FELTAG has endorsed the recommendation that
the entire workforce needs to be brought up to speed to fully understand the potential of
learning technology The recently published Governmentrsquos FE Workforce Strategy BIS
(2014) sets one of its key priorities as the effective use of technology in teaching and
learning ldquoThe objective is that FE providers universities and industry should
collaboratively put in place up-to-date and relevant professional development and initial
training for managers teachers and trainers that is aimed specifically at improving their
knowledge of and confidence in using learning technologyrdquo The development of an
effective tool to support individuals and institutions to measure their performance on the
use of technology in learning has therefore never been so crucial
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 5 | Coralesce Ltd wwwcoralescecom
4 EXECUTIVE SUMMARY This research has been undertaken on behalf of the Education and Training Foundation to
inform its Learning Technologies Self-Assessment project The project aims to produce a
self-assessment tool for the education and training sector which builds on the most recent
work undertaken by the precursor organisation LSIS (Learning and Skills Improvement
Service) This work resulted in the production of a self-assessment tool entitled the Review
and Planning for Technology in Action (RAPTA) tool
The revised tool also needs to build on best practice in other sectors and this baseline report
seeks to explore the range of practice and approaches adopted in other frameworks This
analysis is not an exhaustive list of all the tools available but rather a practical overview to
inform the next steps in developing an appropriate and contemporary Learning
Technologies Self-Assessment tool for the sector
The frameworks analysed fall into three broad categories Organisational self-assessment
Individual self-assessment and Quality assurance frameworks for e-learning This report
provides an analysis of the organisational and individual self-assessment and competency
frameworks from industry schools and further and higher education These are then
evaluated against the following criteria by sector and by criteria
Suitability for the Sector
Impact on Teaching and Learning
Suitability for Self-assessment
Quality improvement and
Sustainability
1 Suitability for whole sector
This criterion focuses on whether the self-assessment tool is suitable for all types of
education and training providers in the sector The state funded education and training
sector is a lsquobroad churchrsquo made up of Further Education Colleges 6th Form Colleges Private
training providers Adult and Community Learning providers Prison Education Service and
Unionlearn
2 Impact on Teaching and Learning
This criterion focuses on whether the tool measures the effective use of learning
technologies in teaching and learning and addresses outcomes for learners
3 Suitability for Self-assessment
This criterion considers whether the tool is robust for the purposes of self-assessment and
considers whether it is informed by relevant standards such as
New Professional Standards for teachers
Ofsted Common Inspection Framework
Digital literacy expectations for practitioners and students
Suitability for self-assessment also involves enabling all relevant parties to self-assess and
plan for improvement whatever their competence confidence level such as
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 6 | Coralesce Ltd wwwcoralescecom
Practitioners
Curriculum managers
ICT managers
Quality managers
Senior managers
Governors or equivalent
Employers
The tool needs to be able to be integrated successfully with the collegersquos existing self-
assessment and quality improvement processes Accessibility ease of use and length of the
tool also impacts on its suitability for self-assessment
4 Quality improvement
Effective self-assessment should lead to quality improvement This criterion considers
whether the tool effectively supports quality improvement by either supporting
development via easy to use links to resources ideas best practice exemplars training
materials and or eliciting quantitative data for benchmarking To ensure quality
improvement is appropriate and effective the tool needs to ensure that the data collected is
accurate and this may include strategies to ensure rigour
5 Sustainable
This criterion considers whether the tool is sustainable and capable of supporting
continuous quality improvement in the use of learning technologies for the sector in the
years to come
SECTOR ANALYSIS
The 12 tools were evaluated by sector as follows
School Sector Self-Assessment Frameworks
The Naace Self Review Framework (SRF) provides a framework for schools which is
designed to support them to review the use of technology in a structured way and to support
school improvement through reflective practice The framework links to the schools Self
Evaluation Framework (SEF) and therefore captures evidence for the annual self-evaluation
and for Ofsted inspections The tool focuses on ICT as well as Learning Technology but
also includes a set of statements in the section on developing pupilsrsquo digital literacy
The Washington State Technology Essential Conditions Rubric evaluates
institution progress against the Washington State Educational Standards the Essential
Academic Learning Requirements The rubric is relatively short and grades institution as
beginning progressing or proficient The rubric looks at five key aspects Vision planning
and policy technical literacy for learners integration standards for teachers technical
competencies of administrators communications and connections
DigiLit Leicester is a digital literacy framework developed as a result of lsquoBuilding Schools
for the Futurersquo ICT priorities It assesses competency against six strands of digital literacy
in the secondary school sector It puts learning and learners at the centre with a very clear
focus on outcomes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 7 | Coralesce Ltd wwwcoralescecom
FE and Skills Self-Assessment Frameworks
The Development Self Audit (DSA) was undertaken by Scottish Colleges as part of a
baseline study of e-activity in 2007 It is based on practical activities and adapted from
research by Dr Stephen Marshall who developed the eLearning Maturity Model for higher
education institutions in New Zealand The self-audit reviews five aspects Learning
Development Support Evaluation and Organisation As it is based on the eMM it also
maps the aspects against five dimensions of capability Policy Planning Delivery
Management Improvement These are then mapped to produce a management level e-
maturity grid
eQSF A Quality Standards Framework for ICT in Learning was developed using
European funding and piloted in North Ireland in 2009 10 It provides a comprehensive
evaluation and action planning tool consisting of thirteen questions covering three pillars
Values Pedagogy and Practice eStructure of the Learning Process Leadership and
Management The online tool facilitates self-assessment and results in action planning for
improvement
ACL e-Learning Positioning Statement was developed by the National Institute for
Adult and Continuing Education (NIACE) and the Centre for Excellence in Leadership
(CEL) in 2005 It was designed to support institutional strategic planning in e-learning and
institutional quality improvement processes It focused on five core strands Vision and
Strategic Planning Teaching and Learning Staff Development Infrastructure and
Managing ILT and e-Learning
Generator was a self-assessment tool developed through a partnership between LSN Jisc
and BECTA in 2008-2011 Whilst complex it enabled learning providers to identify the
impact that technology had in 9 key business processes against seven enablers of
technology The tool was evaluated against a set of outcome based questions and was
capable of reviewing two or more curriculum areas within an organisation
Review and Plan Technology in Action (RAPTA) was developed by Jisc in 2012-13
with support from Learning and Skills Improvement Service linked to the Ofsted Common
Inspection Framework It aimed to provide a structured framework which can be integrated
in to an organisationrsquos self-assessment process The tool was lengthy with questions
focused on teaching and learning and leadership and management A shorter quick review
version was also available
Higher Education Self-Assessment Frameworks
e-Learning Maturity Model is a mature evaluation framework developed by Dr Stephen Marshall and provides a quality improvement framework for HE institutions in New Zealand It is a complex system focused on online and blended learning The tool benchmarks e-learning capability using 35 inter-related processes It supports external assessment rather than self-assessment as it is externally validated
Industry Self-Assessment Frameworks
Toward Maturity is a mature self-assessment and benchmarking tool which is straightforward but lengthy to use Designed to promote reflection it uses a set of KPIs and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 8 | Coralesce Ltd wwwcoralescecom
identification of best practice to provide a sophisticated benchmarking tool Benchmarking reports are based on 9 KPIs and measure against three levels beginning progressing and proficient
Cross Sector Frameworks
The Open University Digital Literacy Framework identifies the competencies for
learners at HE levels 0123 to M (Level 0 equates to FE Level 3) across five aspects of
competence It provides a self-assessment tool which is presented in short sections The
framework could be used as a basis for individual self-assessment for both learners and
practitioners
The UNESCO Competency Framework for Teachers is an international framework
that supports teachers to develop students to become collaborative problem solving
creative learners It has three aspects Technology Literacy Knowledge Deepening
Knowledge Creation
ANALYSIS BY CRITERIA
The 12 tools were evaluated by criteria as follows
1 Teaching and Learning The school frameworks reviewed have a strong focus on outcomes for learners They use a similar model and focus on classroom based delivery but do not reference online or workplace learning In some cases the positioning of teaching and learning is overshadowed by a focus on leadership and management Apart from the DigiLit Leicester framework the school based tools omit to review the use of technology to support assessment monitoring and feedback The most recent FE and Learning and Skills frameworks eQSF and RAPTA sought to have teaching as a central theme RAPTA has a twin focus on inspiring and challenging learners and encouraging independent learning The eQSF framework however describes processes rather than outcomes The UNESCO framework applies to all sectors focuses on the development of teachers and has desired outcomes for learners as a central feature The Washington State Technology Essential Conditions Rubric has a similar focus Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners Whilst the HE framework eMM focuses on online learning it has a narrow focus on teaching and learning
2 Suitability for Self-assessment The scope of the varying frameworks is similar generally covering core questions on pedagogy outcomes for learners strategic planning and professional development for practitioners The extent to which the frameworks engage with the full range of stakeholders is varied some encouraging completion by one manager within the organisation as in Towards Maturity which can reduce the rigour of the findings In contrast others encourage involvement of learners teachers and managers in the process eg RAPTA and eQSF thus providing a more robust approach to self-assessment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 9 | Coralesce Ltd wwwcoralescecom
3 Quality Improvement A number of organisational frameworks focus on quality improvement such as the Naace and QSF frameworks which provide an on-line action planning tool Other frameworks including Naace provide sources of high quality support materials as does the Open University Digital Literacy Framework
4 Suitability for the whole Sector The tools that link to the Common Inspection Framework could be adapted for use across the sector however they would need adapting to encompass the range of settings represented across the sector particularly workplace learning Frameworks which focus on individual teacher competency such as UNESCO ICT Framework and the DigiLit Leicester would require limited adaptation for use in the FE and Skills Sector
5 Sustainability The tools that have demonstrated the greatest sustainability are the Towards Maturity tool and the Naace tool which have either attracted industry sponsorship to support its annual use over a 10 year period (Towards Maturity) or incorporate small charges for registration and use (Naace)
THEMATIC ANALYSIS
A thematic analysis highlights the question areas that are considered within each tool
covering strategic leadership infrastructure teaching and learning learner experience
staff development financial management and quality improvement All the 12 tools use
structured questions some have dichotomous questions (YES NO) whilst others are based
on levels of measurement (Likert Response Scaling) (Strongly Agree to Strongly Disagree)
or Cumulative Scaling (1-4 or Descriptive Names that indicate a Journey)
In reviewing the frameworks the following conclusions have been drawn which will be
tested at the next stage of consultation within the Learning Technologies Self-Assessment
Project to enable the development of a robust and valued self-assessment tool that builds on
previous work Our conclusions from this desk research are that a self-assessment tool
should
Be explicitly linked to how learning technologies support teaching and learning
Facilitate access by a range of stakeholders by being concise and designed to support
differing approaches to self-assessment
Use structured questions on a meaningful cumulative scale
Model guide and support good self-assessment practice
Provide effective support for quality improvement through for example providing
automatically generated action plans
Be sustainable for the sector once grant funding has ceased
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 10 | Coralesce Ltd wwwcoralescecom
Comparison of the themes used in Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 11 | Coralesce Ltd wwwcoralescecom
5 RESEARCH FINDINGS
A range of frameworks were identified as part of the research and these broadly fell into three categories
Organisational self-assessment
Individual self-assessment
Quality assurance frameworks for e-learning
The major focus of the Education and Training Foundationrsquos Learning Technology Self-Assessment Project is to develop an organisational tool with the potential to integrate individual practitioner self-assessment and therefore the summary below focuses on the first two types of framework Links to quality assurance framework for e-learning are provided in Appendix 1
METHODOLOGY
The methodology used in this Baseline Report was desk-based research to identify a range of self-assessment tools and quality frameworks covering schools further education and learning and skills higher education and industry Whilst the research provided a valuable and comprehensive overview of organisational and individual self-assessment tools and quality frameworks for e-learning it has not always been possible to access full question banks for all tools This is due to the fact that once an organisation responsible for a tool is closed not all archives are kept In the case of BECTA their self-assessment tool Generator was web-based and much of the material including the full question bank was not fully archived at their demise In addition the full set of questions used by Towards Maturity has been difficult to access due to the commercial nature of the organisation Where access has been impeded the research team have sourced a sample of questions through secondary literature
51 Organisational Self-Assessment Frameworks
Naace Self Review Framework (SRF) Sector Schools
Country England
Year 2014
A self-review framework designed to support schools to review their use of technology in a structured way and to support school improvement through reflective practice Developed against a well-researched and evidence based set of criteria Self-Review Framework (SRF) is a mature system originally developed by BECTA and now managed and updated annually by Naace
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 12 | Coralesce Ltd wwwcoralescecom
Naace is a membership organisation and provides an online tool which is accessible only to
member organisations (costs pound100 to join Naace and a further pound50 to access the online
tool) Members are well supported with training in the use of SRF and a useful website A
PDF version of the criteria is free to use
The SRF focuses on supporting pupils to become digitally literate and able to express
themselves and develop ideas through the use of ICT The tool is broad in its approach and
has less of an overall focus on teaching learning and assessment compared to the RAPTA
tool As this is a school based system it has a classroom based focus and does not reflect
the use of technology to support assessment monitoring and target setting
The framework links to the schools Self Evaluation Framework (SEF) and therefore
captures evidence for the annual self-evaluation and for Ofsted inspections The tool
focuses on ICT as well as Learning Technology but has a set of statements in the section on
developing digital literacy which could be adapted The framework consists of
Leadership and Management
Teaching and Learning
Professional Development
Resources
Each section is expanded into a number statements (54 in total) each of which is graded 1
to 4 against a clear set of grade descriptor (figure 1) Whilst the tool is long it is easy to
understand and designed to be used by different staff roles
The online tool provides clear guidance on each aspect and an opportunity to record
evidence and identify actions to improve
The online tool also provides a clear graphical representation of organisational capability
(figure 2) and although not currently benchmarked could be adapted to provide benchmark
data This process would lead to production of a detailed and specific plan for
improvement
The standards section links to an ICT Mark which recognises e-maturity The Mark is
externally assessed and verifies the judgements made in the SRF Following on from the
ICT Mark schools have an opportunity to apply for a Third Millennium Learning Award a
peer-referenced award which demonstrates that schools are providing education fit for the
21st century
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 13 | Coralesce Ltd wwwcoralescecom
Figure 1 Example Question from online SRF Tool
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 14 | Coralesce Ltd wwwcoralescecom
Figure 2 Graphical summary of outputs from online tool
Level Aiming For Enhancing Learning through ICT - level 3
1 Leadership and Management 1a-
1
1a-
2
1b-
1
1b-
2
1b-
3
1b-
4
1b-
5
1b-
6
1c-1 1c-2 1c-3 1c-4
3 4 X 4 3 4 3 3 X X X X
2 Use of ICT in the curriculum 2a-
1
2a-
2
2a-3 2a-4 2a-5 2a-6 2a-7 2b-
1
2b-
2
2b-
3
2b-
4
X X X X X X X X X X X
3 Teaching and Learning 3a-
1
3a-
2
3a-3 3a-4 3a-5 3b-
1
3b-
2
3b-
3
3b-
4
3b-
5
3b-
6
3b-
7
3b-
8
3b-
9
X X X X X X X X X X X X X X
4 Assessment of digital
capability
4a-
1
4a-
2
4a-3
X X X
5 Professional Development 5a-
1
5a-
2
5a-3 5a-4
X X X X
6 Resources 6a-
1
6a-
2
6a-3 6a-4 6a-5 6a-6 6b-
1
6b-
2
6b-
3
6b-
4
X X X X X X X X X X
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 15 | Coralesce Ltd wwwcoralescecom
Washington State Educational Technology Standards and Technology Essential
Conditions Rubric
Sector School
Country USA
Year 2008
A very well constructed model based on research on the benefits and dangers of technology
The standards put the learner at the heart of the system focusing on their expectations
Developed for schools as with the Naace system in England it is classroom focused The
Washington State materials consist of standards for learners and for teachers and are
supplemented with a Technology Essential Rubric for schools to evaluate progress at an
institutional level
The standards emanated from state legislature (1997) which established common learning
goals for all students in Washington State which provided a foundation for the
development of content called Essential Academic Learning Requirements (EALR)
Educational Technology was added to the EALR in 2007 and articulates high expectations
for what students should be able to do across all subject areas The standards for
educational technology were categorised as
Technology Integration
Digital Citizenship (technical literacy moving to technical fluency)
The standards are detailed albeit a little out of date with each EALR broken down into
components each with a grade level expectation for each year of education up to K12 (see
figure 3) Each grade level expectation has a descriptor of evidence of learning As the
standard requires use across subject areas it also provides examples across different subject
areas and levels including basic and 21st century learning environment The standards
provide detailed and useful materials to support teachers and curriculum areas to evaluate
and improve practice They could also be adapted for learner self-evaluation
The self-assessment tool the Technology Essential Conditions Rubric for Schools is
relatively short (8 pages) but would support effective self-assessment and has the potential
for benchmarking It uses a simple grading system beginning progressing proficient and
looks at five key areas
Vision planning and policy
Technical Literacy for learners ndash related to EALRs
Integration standards for teachers ndashrelated to EALRs
Technical competencies of administrators
Communication and Connections
Integration standards for teachers are described in three tiers Teacher Focus on
Productivity Instructional Presentation and Student Productivity Powerful Student-
Centred 21st century Learning Environment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 16 | Coralesce Ltd wwwcoralescecom
EALR 2 mdash DIGITAL CITIZENSHIP
Students demonstrate a clear understanding
of technology systems and operations and
practice safe legal and ethical behavior
Component 21 Practice Safety
Demonstrate safe legal and ethical behavior
in the use of information and technology
GLE Grade 6
212 Practice ethical and respectful behavior
Evidence
of
Learning
Comply with district Acceptable Use Policy (AUP)
Practice ethical and appropriate use of all media and comply with copyright law
Participate and engage with the global community within expected norms of behavior and positive interaction
Recognize report and
counteract cyberbullying
Examples
Recognize different types of illegal and unethical technology use
Identify and cite Creative Commons images in online presentations
Cite all sources properly
Observable indicators are provided for each tier together with helpful examples these could
be useful for practitioner level self-assessment Teachers are supported with extensive
materials available from the Washington State Office of the Superintendent of Public
Instruction wwwk12waus curriculuminstruct
Figure 3 An example of EALR for digital citizenship is shown below
Required
An Essential Academic Learning Requirement (EALR) is a broad statement of the learning that applies to Grades Kndash12 The Component is a statement that further defines and provides more specific information about the EALR There is at least one component for each EALR The Grade Level Expectation (GLE) is a statement containing the essential content or process to be learned and the cognitive demand required to learn it A revision of Bloomrsquos Taxonomy was used to categorize the cognitive demand required of the student Each GLE includes evidence of learning statements which are considered essential to the GLE
The GLE Numbering System identifies the EALR the component and the GLE in that order In the example at right the number 2 indicates the EALR the number 21 indicates the component and the number212 indicates the GLE Note Grade levels are not referenced in the numbering system The Evidence of Learning is a bulleted list of ways students can demonstrate learning considered essential to the GLE Educators are encouraged to identify additional ways in which the student can show proficiency as the educational technology GLEs are integrated across the curriculum Suggested The Examples provide specific illustrations of the learning However these examples are not exhaustive and educators are encouraged to find multiple ways by which learners can demonstrate what they know
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 17 | Coralesce Ltd wwwcoralescecom
Development Self Audit (DSA)
Sector Further Education
Country Scotland
Year 2007
Undertaken by Scottish Colleges as part of a baseline study of e-activity in 2007 the DSA
assessed e-maturity on a broad basis The development was funded by the Scottish Funding
Council and was informed by the practical activities and research of Dr Stephen Marshall
who developed an ideal model of e-maturity (eLearning Maturity Model)
The questions in the DSA cover five aspects
Learning (processes that impact on pedagogical aspects of e-learning)
Development (processes that surround the creation and maintenance of resources)
Support (processes surrounding oversight and management)
Evaluation (processes surrounding the evaluation and quality control)
Organisation (processes associated with planning and management)
As the DSA is based on the eMM developed in New Zealand it also maps the aspects against
the five dimensions of capability
Policy
Planning
Delivery
Management
Improvement
Using external assessment this process leads to the development of a grid which provides a
management level view of the e-maturity of a college an example is shown in figure 6
Figure 6 Management Level E-Maturity Grid
In addition to the completion of the grid the Scottish Funding Council also gathered
feedback on specific issues through the use of a questionnaire completed by institutions and
individuals which looked at the following themes
DSA
Po
licy
Pla
nn
ing
Delivery
Man
ag
em
en
t
Imp
rovem
en
t
Learning
L1 e-Learning is designed according to pedagogically sound principles N P L F P
L2 Learning skills are developed explicitly amp incrementally during delivery P F P P P
L3 Peer amp tutor interactions are provided explicitly during e-learning delivery P P N P P
L4 Courses take account of student workload and e-assessment demands P P N L P
L5 e-learning actively engage students P P P P N
L6 College defined (optional) na na na na na
Development
D1 Course development is directed and supported P P L F P
D2 Resources are maintained and re-used P F P P P
D3 Courses meet the needs of all students P P N P P
D4 The e-learning infrastructure is reliable robust and sufficient P P N L P
D5 The e-learning infrastructure is standardised and integrated P P P P N
D6 College defined (optional) na na na na na
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 18 | Coralesce Ltd wwwcoralescecom
Infrastructure
Teaching and Learning
Organisation
Strategic Leadership
e-Learning coverage by department subject area
The questions covered the extent to which e-learning was used across levels type of
learning and to what extent it was used in Work based Learning An important aspect that
the questionnaire covered under the heading of strategic leadership was the financial
aspects including affordability of software implementation and cost benefit analysis
eQSF A Quality Standards Framework for ICT in Learning
Sector Learning and Skills
Country Northern Ireland
Year 2009 10
eQSF was developed using European funding and provides a comprehensive evaluation and
action planning package to assess practice in the use of existing and new technologies The
framework consists of three pillars each with a series of standards and examples of
evidence
Values Pedagogy and Practice (see figure 4)
Structure of the Learning Process
Leadership and Management
The pillars break down into 13 quality standards each with a separate on-line template to
complete Each standard is graded on a scale of attainment from Absent to Advanced
However the statements tend to describe processes rather than outcomes and are quite
broad particularly in the section on Teaching and Learning
The online resource facilitates the self-assessment process provides opportunities to
identify evidence and results in an action plan for improvement The implementation
process involves establishing leadership and ownership reviewing practice action
planning implementing action plan evaluating progress and feedback
Positive feedback is available from the three pilot case studies that were developed
including North West College in NI which indicate the tool has been used at teacher
department and organisational level
http wwweqsfeu
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 19 | Coralesce Ltd wwwcoralescecom
Figure 4 Values Pedagogy amp Practice Standards
Standard Addressed
Yes No
Actions
Completed Current Attainment Level Signature
11
The organisation supports inclusiveness and accessibility through the use of ICT and sees ICT as promoting equality and diversity in the learning process
Absent Acquiring Achieved Advanced
out of
12
The organisation facilitates the adaptation of learning materials to meet a range of learner requirements and sees ICT as a mechanism for delivering such flexibility for the practitioner and learner
Absent Acquiring Achieved Advanced
out of
13
The organisation uses ICT to extend learning beyond the classroom and supports progression to further learning and career opportunities
Absent Acquiring Achieved Advanced
out of
14
The organisation is committed to the learnersrsquo outcomes being positively impacted by ICT
Absent Acquiring Achieved Advanced
out of
Comment
Adult and Community Learning e-Learning Positioning Statement
Sector Adult and Community Learning
Country England
Date 2005-2007
The ACL eLPs were designed by CEL (Centre for Excellence in Leadership) and NIACE
(National Institute for Adult amp Continuing Education) as an aid to both institutional
strategic planning in e-learning and to support institutional quality improvement processes
As well as its use within NIACEs e-learning management and leadership courses the tool
was also embedded in individual consultancies and within the NIACE Digital Learning
teams capital investment funding work The ACL eLPs tool was superseded by BECTAs
Technology Improvement Leadership Tool - Generator NIACE formed part of the
development team for Generator and continued to contribute to its development and
piloting with community-based adult learning providers The use of the Generator tool was
integrated into NIACEs e-learning capital funding projects (CAMEL) The e-Learning
positioning statement comprised 31 elements grouped under five themes Organisations
graded themselves against a five point scale ranging from not yet started to embedded The
ACL eLearning Positioning Statement was designed as an aid to both institutional strategic
planning in e-learning and to support institutional quality improvement processes
The framework reviewed Vision and Strategic Planning Teaching and Learning Staff
Development Infrastructure and Managing ILT and e-Learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 4 | Coralesce Ltd wwwcoralescecom
bring people together to grow ideas to improve education development and training
undertake research on practice to inform developments
provide innovative solutions to transform learning journeys
advance the promotion of equal opportunities
enable people to use technology to share learning
provide a high-quality shared curriculum management service
The Learning Technologies Self-Assessment Project builds on research work already
undertaken by Coralesce Ltd for the Education and Training Foundation from January ndash
March 2014 Coralesce Ltd undertook a strategic consultation on technology in teaching and
learning to inform the Foundation on the sector priorities on learning technologies
3 POLICY CONTEXT This development of a new self-assessment tool for the sector has taken place against a
backdrop of an emerging national debate about the future use of technology in learning with
the work of FELTAG This was set up in January 2013 by Matthew Hancock Minister of
State for Skills and Enterprise in BIS as a sector group to make practical recommendations
aimed at ensuring the effective use of digital technology in learning teaching and
assessment in Further Education and Skills On 28th February 2014 the FELTAG Group
published its recommendations across six work streams
Horizon-scanning
Investment and Capital Infrastructure
Regulation and Funding
Workforce capacity
Employers
Learners
In its workforce stream the FELTAG report highlighted the need to develop the whole
workforce calling for lsquosignificant investment in the knowledge skills and understanding of
the learning technologyrsquos potential among policy-makers governors principals senior and
middle management teachers and support staffrsquo One of the key workforce
recommendations was the need to develop a revised self-assessment tool to enable the
workforce to evaluate progress as well as a call for benchmarks to be established
The Government Response (BIS 2014) to FELTAG has endorsed the recommendation that
the entire workforce needs to be brought up to speed to fully understand the potential of
learning technology The recently published Governmentrsquos FE Workforce Strategy BIS
(2014) sets one of its key priorities as the effective use of technology in teaching and
learning ldquoThe objective is that FE providers universities and industry should
collaboratively put in place up-to-date and relevant professional development and initial
training for managers teachers and trainers that is aimed specifically at improving their
knowledge of and confidence in using learning technologyrdquo The development of an
effective tool to support individuals and institutions to measure their performance on the
use of technology in learning has therefore never been so crucial
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 5 | Coralesce Ltd wwwcoralescecom
4 EXECUTIVE SUMMARY This research has been undertaken on behalf of the Education and Training Foundation to
inform its Learning Technologies Self-Assessment project The project aims to produce a
self-assessment tool for the education and training sector which builds on the most recent
work undertaken by the precursor organisation LSIS (Learning and Skills Improvement
Service) This work resulted in the production of a self-assessment tool entitled the Review
and Planning for Technology in Action (RAPTA) tool
The revised tool also needs to build on best practice in other sectors and this baseline report
seeks to explore the range of practice and approaches adopted in other frameworks This
analysis is not an exhaustive list of all the tools available but rather a practical overview to
inform the next steps in developing an appropriate and contemporary Learning
Technologies Self-Assessment tool for the sector
The frameworks analysed fall into three broad categories Organisational self-assessment
Individual self-assessment and Quality assurance frameworks for e-learning This report
provides an analysis of the organisational and individual self-assessment and competency
frameworks from industry schools and further and higher education These are then
evaluated against the following criteria by sector and by criteria
Suitability for the Sector
Impact on Teaching and Learning
Suitability for Self-assessment
Quality improvement and
Sustainability
1 Suitability for whole sector
This criterion focuses on whether the self-assessment tool is suitable for all types of
education and training providers in the sector The state funded education and training
sector is a lsquobroad churchrsquo made up of Further Education Colleges 6th Form Colleges Private
training providers Adult and Community Learning providers Prison Education Service and
Unionlearn
2 Impact on Teaching and Learning
This criterion focuses on whether the tool measures the effective use of learning
technologies in teaching and learning and addresses outcomes for learners
3 Suitability for Self-assessment
This criterion considers whether the tool is robust for the purposes of self-assessment and
considers whether it is informed by relevant standards such as
New Professional Standards for teachers
Ofsted Common Inspection Framework
Digital literacy expectations for practitioners and students
Suitability for self-assessment also involves enabling all relevant parties to self-assess and
plan for improvement whatever their competence confidence level such as
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 6 | Coralesce Ltd wwwcoralescecom
Practitioners
Curriculum managers
ICT managers
Quality managers
Senior managers
Governors or equivalent
Employers
The tool needs to be able to be integrated successfully with the collegersquos existing self-
assessment and quality improvement processes Accessibility ease of use and length of the
tool also impacts on its suitability for self-assessment
4 Quality improvement
Effective self-assessment should lead to quality improvement This criterion considers
whether the tool effectively supports quality improvement by either supporting
development via easy to use links to resources ideas best practice exemplars training
materials and or eliciting quantitative data for benchmarking To ensure quality
improvement is appropriate and effective the tool needs to ensure that the data collected is
accurate and this may include strategies to ensure rigour
5 Sustainable
This criterion considers whether the tool is sustainable and capable of supporting
continuous quality improvement in the use of learning technologies for the sector in the
years to come
SECTOR ANALYSIS
The 12 tools were evaluated by sector as follows
School Sector Self-Assessment Frameworks
The Naace Self Review Framework (SRF) provides a framework for schools which is
designed to support them to review the use of technology in a structured way and to support
school improvement through reflective practice The framework links to the schools Self
Evaluation Framework (SEF) and therefore captures evidence for the annual self-evaluation
and for Ofsted inspections The tool focuses on ICT as well as Learning Technology but
also includes a set of statements in the section on developing pupilsrsquo digital literacy
The Washington State Technology Essential Conditions Rubric evaluates
institution progress against the Washington State Educational Standards the Essential
Academic Learning Requirements The rubric is relatively short and grades institution as
beginning progressing or proficient The rubric looks at five key aspects Vision planning
and policy technical literacy for learners integration standards for teachers technical
competencies of administrators communications and connections
DigiLit Leicester is a digital literacy framework developed as a result of lsquoBuilding Schools
for the Futurersquo ICT priorities It assesses competency against six strands of digital literacy
in the secondary school sector It puts learning and learners at the centre with a very clear
focus on outcomes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 7 | Coralesce Ltd wwwcoralescecom
FE and Skills Self-Assessment Frameworks
The Development Self Audit (DSA) was undertaken by Scottish Colleges as part of a
baseline study of e-activity in 2007 It is based on practical activities and adapted from
research by Dr Stephen Marshall who developed the eLearning Maturity Model for higher
education institutions in New Zealand The self-audit reviews five aspects Learning
Development Support Evaluation and Organisation As it is based on the eMM it also
maps the aspects against five dimensions of capability Policy Planning Delivery
Management Improvement These are then mapped to produce a management level e-
maturity grid
eQSF A Quality Standards Framework for ICT in Learning was developed using
European funding and piloted in North Ireland in 2009 10 It provides a comprehensive
evaluation and action planning tool consisting of thirteen questions covering three pillars
Values Pedagogy and Practice eStructure of the Learning Process Leadership and
Management The online tool facilitates self-assessment and results in action planning for
improvement
ACL e-Learning Positioning Statement was developed by the National Institute for
Adult and Continuing Education (NIACE) and the Centre for Excellence in Leadership
(CEL) in 2005 It was designed to support institutional strategic planning in e-learning and
institutional quality improvement processes It focused on five core strands Vision and
Strategic Planning Teaching and Learning Staff Development Infrastructure and
Managing ILT and e-Learning
Generator was a self-assessment tool developed through a partnership between LSN Jisc
and BECTA in 2008-2011 Whilst complex it enabled learning providers to identify the
impact that technology had in 9 key business processes against seven enablers of
technology The tool was evaluated against a set of outcome based questions and was
capable of reviewing two or more curriculum areas within an organisation
Review and Plan Technology in Action (RAPTA) was developed by Jisc in 2012-13
with support from Learning and Skills Improvement Service linked to the Ofsted Common
Inspection Framework It aimed to provide a structured framework which can be integrated
in to an organisationrsquos self-assessment process The tool was lengthy with questions
focused on teaching and learning and leadership and management A shorter quick review
version was also available
Higher Education Self-Assessment Frameworks
e-Learning Maturity Model is a mature evaluation framework developed by Dr Stephen Marshall and provides a quality improvement framework for HE institutions in New Zealand It is a complex system focused on online and blended learning The tool benchmarks e-learning capability using 35 inter-related processes It supports external assessment rather than self-assessment as it is externally validated
Industry Self-Assessment Frameworks
Toward Maturity is a mature self-assessment and benchmarking tool which is straightforward but lengthy to use Designed to promote reflection it uses a set of KPIs and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 8 | Coralesce Ltd wwwcoralescecom
identification of best practice to provide a sophisticated benchmarking tool Benchmarking reports are based on 9 KPIs and measure against three levels beginning progressing and proficient
Cross Sector Frameworks
The Open University Digital Literacy Framework identifies the competencies for
learners at HE levels 0123 to M (Level 0 equates to FE Level 3) across five aspects of
competence It provides a self-assessment tool which is presented in short sections The
framework could be used as a basis for individual self-assessment for both learners and
practitioners
The UNESCO Competency Framework for Teachers is an international framework
that supports teachers to develop students to become collaborative problem solving
creative learners It has three aspects Technology Literacy Knowledge Deepening
Knowledge Creation
ANALYSIS BY CRITERIA
The 12 tools were evaluated by criteria as follows
1 Teaching and Learning The school frameworks reviewed have a strong focus on outcomes for learners They use a similar model and focus on classroom based delivery but do not reference online or workplace learning In some cases the positioning of teaching and learning is overshadowed by a focus on leadership and management Apart from the DigiLit Leicester framework the school based tools omit to review the use of technology to support assessment monitoring and feedback The most recent FE and Learning and Skills frameworks eQSF and RAPTA sought to have teaching as a central theme RAPTA has a twin focus on inspiring and challenging learners and encouraging independent learning The eQSF framework however describes processes rather than outcomes The UNESCO framework applies to all sectors focuses on the development of teachers and has desired outcomes for learners as a central feature The Washington State Technology Essential Conditions Rubric has a similar focus Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners Whilst the HE framework eMM focuses on online learning it has a narrow focus on teaching and learning
2 Suitability for Self-assessment The scope of the varying frameworks is similar generally covering core questions on pedagogy outcomes for learners strategic planning and professional development for practitioners The extent to which the frameworks engage with the full range of stakeholders is varied some encouraging completion by one manager within the organisation as in Towards Maturity which can reduce the rigour of the findings In contrast others encourage involvement of learners teachers and managers in the process eg RAPTA and eQSF thus providing a more robust approach to self-assessment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 9 | Coralesce Ltd wwwcoralescecom
3 Quality Improvement A number of organisational frameworks focus on quality improvement such as the Naace and QSF frameworks which provide an on-line action planning tool Other frameworks including Naace provide sources of high quality support materials as does the Open University Digital Literacy Framework
4 Suitability for the whole Sector The tools that link to the Common Inspection Framework could be adapted for use across the sector however they would need adapting to encompass the range of settings represented across the sector particularly workplace learning Frameworks which focus on individual teacher competency such as UNESCO ICT Framework and the DigiLit Leicester would require limited adaptation for use in the FE and Skills Sector
5 Sustainability The tools that have demonstrated the greatest sustainability are the Towards Maturity tool and the Naace tool which have either attracted industry sponsorship to support its annual use over a 10 year period (Towards Maturity) or incorporate small charges for registration and use (Naace)
THEMATIC ANALYSIS
A thematic analysis highlights the question areas that are considered within each tool
covering strategic leadership infrastructure teaching and learning learner experience
staff development financial management and quality improvement All the 12 tools use
structured questions some have dichotomous questions (YES NO) whilst others are based
on levels of measurement (Likert Response Scaling) (Strongly Agree to Strongly Disagree)
or Cumulative Scaling (1-4 or Descriptive Names that indicate a Journey)
In reviewing the frameworks the following conclusions have been drawn which will be
tested at the next stage of consultation within the Learning Technologies Self-Assessment
Project to enable the development of a robust and valued self-assessment tool that builds on
previous work Our conclusions from this desk research are that a self-assessment tool
should
Be explicitly linked to how learning technologies support teaching and learning
Facilitate access by a range of stakeholders by being concise and designed to support
differing approaches to self-assessment
Use structured questions on a meaningful cumulative scale
Model guide and support good self-assessment practice
Provide effective support for quality improvement through for example providing
automatically generated action plans
Be sustainable for the sector once grant funding has ceased
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 10 | Coralesce Ltd wwwcoralescecom
Comparison of the themes used in Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 11 | Coralesce Ltd wwwcoralescecom
5 RESEARCH FINDINGS
A range of frameworks were identified as part of the research and these broadly fell into three categories
Organisational self-assessment
Individual self-assessment
Quality assurance frameworks for e-learning
The major focus of the Education and Training Foundationrsquos Learning Technology Self-Assessment Project is to develop an organisational tool with the potential to integrate individual practitioner self-assessment and therefore the summary below focuses on the first two types of framework Links to quality assurance framework for e-learning are provided in Appendix 1
METHODOLOGY
The methodology used in this Baseline Report was desk-based research to identify a range of self-assessment tools and quality frameworks covering schools further education and learning and skills higher education and industry Whilst the research provided a valuable and comprehensive overview of organisational and individual self-assessment tools and quality frameworks for e-learning it has not always been possible to access full question banks for all tools This is due to the fact that once an organisation responsible for a tool is closed not all archives are kept In the case of BECTA their self-assessment tool Generator was web-based and much of the material including the full question bank was not fully archived at their demise In addition the full set of questions used by Towards Maturity has been difficult to access due to the commercial nature of the organisation Where access has been impeded the research team have sourced a sample of questions through secondary literature
51 Organisational Self-Assessment Frameworks
Naace Self Review Framework (SRF) Sector Schools
Country England
Year 2014
A self-review framework designed to support schools to review their use of technology in a structured way and to support school improvement through reflective practice Developed against a well-researched and evidence based set of criteria Self-Review Framework (SRF) is a mature system originally developed by BECTA and now managed and updated annually by Naace
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 12 | Coralesce Ltd wwwcoralescecom
Naace is a membership organisation and provides an online tool which is accessible only to
member organisations (costs pound100 to join Naace and a further pound50 to access the online
tool) Members are well supported with training in the use of SRF and a useful website A
PDF version of the criteria is free to use
The SRF focuses on supporting pupils to become digitally literate and able to express
themselves and develop ideas through the use of ICT The tool is broad in its approach and
has less of an overall focus on teaching learning and assessment compared to the RAPTA
tool As this is a school based system it has a classroom based focus and does not reflect
the use of technology to support assessment monitoring and target setting
The framework links to the schools Self Evaluation Framework (SEF) and therefore
captures evidence for the annual self-evaluation and for Ofsted inspections The tool
focuses on ICT as well as Learning Technology but has a set of statements in the section on
developing digital literacy which could be adapted The framework consists of
Leadership and Management
Teaching and Learning
Professional Development
Resources
Each section is expanded into a number statements (54 in total) each of which is graded 1
to 4 against a clear set of grade descriptor (figure 1) Whilst the tool is long it is easy to
understand and designed to be used by different staff roles
The online tool provides clear guidance on each aspect and an opportunity to record
evidence and identify actions to improve
The online tool also provides a clear graphical representation of organisational capability
(figure 2) and although not currently benchmarked could be adapted to provide benchmark
data This process would lead to production of a detailed and specific plan for
improvement
The standards section links to an ICT Mark which recognises e-maturity The Mark is
externally assessed and verifies the judgements made in the SRF Following on from the
ICT Mark schools have an opportunity to apply for a Third Millennium Learning Award a
peer-referenced award which demonstrates that schools are providing education fit for the
21st century
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 13 | Coralesce Ltd wwwcoralescecom
Figure 1 Example Question from online SRF Tool
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 14 | Coralesce Ltd wwwcoralescecom
Figure 2 Graphical summary of outputs from online tool
Level Aiming For Enhancing Learning through ICT - level 3
1 Leadership and Management 1a-
1
1a-
2
1b-
1
1b-
2
1b-
3
1b-
4
1b-
5
1b-
6
1c-1 1c-2 1c-3 1c-4
3 4 X 4 3 4 3 3 X X X X
2 Use of ICT in the curriculum 2a-
1
2a-
2
2a-3 2a-4 2a-5 2a-6 2a-7 2b-
1
2b-
2
2b-
3
2b-
4
X X X X X X X X X X X
3 Teaching and Learning 3a-
1
3a-
2
3a-3 3a-4 3a-5 3b-
1
3b-
2
3b-
3
3b-
4
3b-
5
3b-
6
3b-
7
3b-
8
3b-
9
X X X X X X X X X X X X X X
4 Assessment of digital
capability
4a-
1
4a-
2
4a-3
X X X
5 Professional Development 5a-
1
5a-
2
5a-3 5a-4
X X X X
6 Resources 6a-
1
6a-
2
6a-3 6a-4 6a-5 6a-6 6b-
1
6b-
2
6b-
3
6b-
4
X X X X X X X X X X
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 15 | Coralesce Ltd wwwcoralescecom
Washington State Educational Technology Standards and Technology Essential
Conditions Rubric
Sector School
Country USA
Year 2008
A very well constructed model based on research on the benefits and dangers of technology
The standards put the learner at the heart of the system focusing on their expectations
Developed for schools as with the Naace system in England it is classroom focused The
Washington State materials consist of standards for learners and for teachers and are
supplemented with a Technology Essential Rubric for schools to evaluate progress at an
institutional level
The standards emanated from state legislature (1997) which established common learning
goals for all students in Washington State which provided a foundation for the
development of content called Essential Academic Learning Requirements (EALR)
Educational Technology was added to the EALR in 2007 and articulates high expectations
for what students should be able to do across all subject areas The standards for
educational technology were categorised as
Technology Integration
Digital Citizenship (technical literacy moving to technical fluency)
The standards are detailed albeit a little out of date with each EALR broken down into
components each with a grade level expectation for each year of education up to K12 (see
figure 3) Each grade level expectation has a descriptor of evidence of learning As the
standard requires use across subject areas it also provides examples across different subject
areas and levels including basic and 21st century learning environment The standards
provide detailed and useful materials to support teachers and curriculum areas to evaluate
and improve practice They could also be adapted for learner self-evaluation
The self-assessment tool the Technology Essential Conditions Rubric for Schools is
relatively short (8 pages) but would support effective self-assessment and has the potential
for benchmarking It uses a simple grading system beginning progressing proficient and
looks at five key areas
Vision planning and policy
Technical Literacy for learners ndash related to EALRs
Integration standards for teachers ndashrelated to EALRs
Technical competencies of administrators
Communication and Connections
Integration standards for teachers are described in three tiers Teacher Focus on
Productivity Instructional Presentation and Student Productivity Powerful Student-
Centred 21st century Learning Environment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 16 | Coralesce Ltd wwwcoralescecom
EALR 2 mdash DIGITAL CITIZENSHIP
Students demonstrate a clear understanding
of technology systems and operations and
practice safe legal and ethical behavior
Component 21 Practice Safety
Demonstrate safe legal and ethical behavior
in the use of information and technology
GLE Grade 6
212 Practice ethical and respectful behavior
Evidence
of
Learning
Comply with district Acceptable Use Policy (AUP)
Practice ethical and appropriate use of all media and comply with copyright law
Participate and engage with the global community within expected norms of behavior and positive interaction
Recognize report and
counteract cyberbullying
Examples
Recognize different types of illegal and unethical technology use
Identify and cite Creative Commons images in online presentations
Cite all sources properly
Observable indicators are provided for each tier together with helpful examples these could
be useful for practitioner level self-assessment Teachers are supported with extensive
materials available from the Washington State Office of the Superintendent of Public
Instruction wwwk12waus curriculuminstruct
Figure 3 An example of EALR for digital citizenship is shown below
Required
An Essential Academic Learning Requirement (EALR) is a broad statement of the learning that applies to Grades Kndash12 The Component is a statement that further defines and provides more specific information about the EALR There is at least one component for each EALR The Grade Level Expectation (GLE) is a statement containing the essential content or process to be learned and the cognitive demand required to learn it A revision of Bloomrsquos Taxonomy was used to categorize the cognitive demand required of the student Each GLE includes evidence of learning statements which are considered essential to the GLE
The GLE Numbering System identifies the EALR the component and the GLE in that order In the example at right the number 2 indicates the EALR the number 21 indicates the component and the number212 indicates the GLE Note Grade levels are not referenced in the numbering system The Evidence of Learning is a bulleted list of ways students can demonstrate learning considered essential to the GLE Educators are encouraged to identify additional ways in which the student can show proficiency as the educational technology GLEs are integrated across the curriculum Suggested The Examples provide specific illustrations of the learning However these examples are not exhaustive and educators are encouraged to find multiple ways by which learners can demonstrate what they know
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 17 | Coralesce Ltd wwwcoralescecom
Development Self Audit (DSA)
Sector Further Education
Country Scotland
Year 2007
Undertaken by Scottish Colleges as part of a baseline study of e-activity in 2007 the DSA
assessed e-maturity on a broad basis The development was funded by the Scottish Funding
Council and was informed by the practical activities and research of Dr Stephen Marshall
who developed an ideal model of e-maturity (eLearning Maturity Model)
The questions in the DSA cover five aspects
Learning (processes that impact on pedagogical aspects of e-learning)
Development (processes that surround the creation and maintenance of resources)
Support (processes surrounding oversight and management)
Evaluation (processes surrounding the evaluation and quality control)
Organisation (processes associated with planning and management)
As the DSA is based on the eMM developed in New Zealand it also maps the aspects against
the five dimensions of capability
Policy
Planning
Delivery
Management
Improvement
Using external assessment this process leads to the development of a grid which provides a
management level view of the e-maturity of a college an example is shown in figure 6
Figure 6 Management Level E-Maturity Grid
In addition to the completion of the grid the Scottish Funding Council also gathered
feedback on specific issues through the use of a questionnaire completed by institutions and
individuals which looked at the following themes
DSA
Po
licy
Pla
nn
ing
Delivery
Man
ag
em
en
t
Imp
rovem
en
t
Learning
L1 e-Learning is designed according to pedagogically sound principles N P L F P
L2 Learning skills are developed explicitly amp incrementally during delivery P F P P P
L3 Peer amp tutor interactions are provided explicitly during e-learning delivery P P N P P
L4 Courses take account of student workload and e-assessment demands P P N L P
L5 e-learning actively engage students P P P P N
L6 College defined (optional) na na na na na
Development
D1 Course development is directed and supported P P L F P
D2 Resources are maintained and re-used P F P P P
D3 Courses meet the needs of all students P P N P P
D4 The e-learning infrastructure is reliable robust and sufficient P P N L P
D5 The e-learning infrastructure is standardised and integrated P P P P N
D6 College defined (optional) na na na na na
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 18 | Coralesce Ltd wwwcoralescecom
Infrastructure
Teaching and Learning
Organisation
Strategic Leadership
e-Learning coverage by department subject area
The questions covered the extent to which e-learning was used across levels type of
learning and to what extent it was used in Work based Learning An important aspect that
the questionnaire covered under the heading of strategic leadership was the financial
aspects including affordability of software implementation and cost benefit analysis
eQSF A Quality Standards Framework for ICT in Learning
Sector Learning and Skills
Country Northern Ireland
Year 2009 10
eQSF was developed using European funding and provides a comprehensive evaluation and
action planning package to assess practice in the use of existing and new technologies The
framework consists of three pillars each with a series of standards and examples of
evidence
Values Pedagogy and Practice (see figure 4)
Structure of the Learning Process
Leadership and Management
The pillars break down into 13 quality standards each with a separate on-line template to
complete Each standard is graded on a scale of attainment from Absent to Advanced
However the statements tend to describe processes rather than outcomes and are quite
broad particularly in the section on Teaching and Learning
The online resource facilitates the self-assessment process provides opportunities to
identify evidence and results in an action plan for improvement The implementation
process involves establishing leadership and ownership reviewing practice action
planning implementing action plan evaluating progress and feedback
Positive feedback is available from the three pilot case studies that were developed
including North West College in NI which indicate the tool has been used at teacher
department and organisational level
http wwweqsfeu
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 19 | Coralesce Ltd wwwcoralescecom
Figure 4 Values Pedagogy amp Practice Standards
Standard Addressed
Yes No
Actions
Completed Current Attainment Level Signature
11
The organisation supports inclusiveness and accessibility through the use of ICT and sees ICT as promoting equality and diversity in the learning process
Absent Acquiring Achieved Advanced
out of
12
The organisation facilitates the adaptation of learning materials to meet a range of learner requirements and sees ICT as a mechanism for delivering such flexibility for the practitioner and learner
Absent Acquiring Achieved Advanced
out of
13
The organisation uses ICT to extend learning beyond the classroom and supports progression to further learning and career opportunities
Absent Acquiring Achieved Advanced
out of
14
The organisation is committed to the learnersrsquo outcomes being positively impacted by ICT
Absent Acquiring Achieved Advanced
out of
Comment
Adult and Community Learning e-Learning Positioning Statement
Sector Adult and Community Learning
Country England
Date 2005-2007
The ACL eLPs were designed by CEL (Centre for Excellence in Leadership) and NIACE
(National Institute for Adult amp Continuing Education) as an aid to both institutional
strategic planning in e-learning and to support institutional quality improvement processes
As well as its use within NIACEs e-learning management and leadership courses the tool
was also embedded in individual consultancies and within the NIACE Digital Learning
teams capital investment funding work The ACL eLPs tool was superseded by BECTAs
Technology Improvement Leadership Tool - Generator NIACE formed part of the
development team for Generator and continued to contribute to its development and
piloting with community-based adult learning providers The use of the Generator tool was
integrated into NIACEs e-learning capital funding projects (CAMEL) The e-Learning
positioning statement comprised 31 elements grouped under five themes Organisations
graded themselves against a five point scale ranging from not yet started to embedded The
ACL eLearning Positioning Statement was designed as an aid to both institutional strategic
planning in e-learning and to support institutional quality improvement processes
The framework reviewed Vision and Strategic Planning Teaching and Learning Staff
Development Infrastructure and Managing ILT and e-Learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 5 | Coralesce Ltd wwwcoralescecom
4 EXECUTIVE SUMMARY This research has been undertaken on behalf of the Education and Training Foundation to
inform its Learning Technologies Self-Assessment project The project aims to produce a
self-assessment tool for the education and training sector which builds on the most recent
work undertaken by the precursor organisation LSIS (Learning and Skills Improvement
Service) This work resulted in the production of a self-assessment tool entitled the Review
and Planning for Technology in Action (RAPTA) tool
The revised tool also needs to build on best practice in other sectors and this baseline report
seeks to explore the range of practice and approaches adopted in other frameworks This
analysis is not an exhaustive list of all the tools available but rather a practical overview to
inform the next steps in developing an appropriate and contemporary Learning
Technologies Self-Assessment tool for the sector
The frameworks analysed fall into three broad categories Organisational self-assessment
Individual self-assessment and Quality assurance frameworks for e-learning This report
provides an analysis of the organisational and individual self-assessment and competency
frameworks from industry schools and further and higher education These are then
evaluated against the following criteria by sector and by criteria
Suitability for the Sector
Impact on Teaching and Learning
Suitability for Self-assessment
Quality improvement and
Sustainability
1 Suitability for whole sector
This criterion focuses on whether the self-assessment tool is suitable for all types of
education and training providers in the sector The state funded education and training
sector is a lsquobroad churchrsquo made up of Further Education Colleges 6th Form Colleges Private
training providers Adult and Community Learning providers Prison Education Service and
Unionlearn
2 Impact on Teaching and Learning
This criterion focuses on whether the tool measures the effective use of learning
technologies in teaching and learning and addresses outcomes for learners
3 Suitability for Self-assessment
This criterion considers whether the tool is robust for the purposes of self-assessment and
considers whether it is informed by relevant standards such as
New Professional Standards for teachers
Ofsted Common Inspection Framework
Digital literacy expectations for practitioners and students
Suitability for self-assessment also involves enabling all relevant parties to self-assess and
plan for improvement whatever their competence confidence level such as
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 6 | Coralesce Ltd wwwcoralescecom
Practitioners
Curriculum managers
ICT managers
Quality managers
Senior managers
Governors or equivalent
Employers
The tool needs to be able to be integrated successfully with the collegersquos existing self-
assessment and quality improvement processes Accessibility ease of use and length of the
tool also impacts on its suitability for self-assessment
4 Quality improvement
Effective self-assessment should lead to quality improvement This criterion considers
whether the tool effectively supports quality improvement by either supporting
development via easy to use links to resources ideas best practice exemplars training
materials and or eliciting quantitative data for benchmarking To ensure quality
improvement is appropriate and effective the tool needs to ensure that the data collected is
accurate and this may include strategies to ensure rigour
5 Sustainable
This criterion considers whether the tool is sustainable and capable of supporting
continuous quality improvement in the use of learning technologies for the sector in the
years to come
SECTOR ANALYSIS
The 12 tools were evaluated by sector as follows
School Sector Self-Assessment Frameworks
The Naace Self Review Framework (SRF) provides a framework for schools which is
designed to support them to review the use of technology in a structured way and to support
school improvement through reflective practice The framework links to the schools Self
Evaluation Framework (SEF) and therefore captures evidence for the annual self-evaluation
and for Ofsted inspections The tool focuses on ICT as well as Learning Technology but
also includes a set of statements in the section on developing pupilsrsquo digital literacy
The Washington State Technology Essential Conditions Rubric evaluates
institution progress against the Washington State Educational Standards the Essential
Academic Learning Requirements The rubric is relatively short and grades institution as
beginning progressing or proficient The rubric looks at five key aspects Vision planning
and policy technical literacy for learners integration standards for teachers technical
competencies of administrators communications and connections
DigiLit Leicester is a digital literacy framework developed as a result of lsquoBuilding Schools
for the Futurersquo ICT priorities It assesses competency against six strands of digital literacy
in the secondary school sector It puts learning and learners at the centre with a very clear
focus on outcomes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 7 | Coralesce Ltd wwwcoralescecom
FE and Skills Self-Assessment Frameworks
The Development Self Audit (DSA) was undertaken by Scottish Colleges as part of a
baseline study of e-activity in 2007 It is based on practical activities and adapted from
research by Dr Stephen Marshall who developed the eLearning Maturity Model for higher
education institutions in New Zealand The self-audit reviews five aspects Learning
Development Support Evaluation and Organisation As it is based on the eMM it also
maps the aspects against five dimensions of capability Policy Planning Delivery
Management Improvement These are then mapped to produce a management level e-
maturity grid
eQSF A Quality Standards Framework for ICT in Learning was developed using
European funding and piloted in North Ireland in 2009 10 It provides a comprehensive
evaluation and action planning tool consisting of thirteen questions covering three pillars
Values Pedagogy and Practice eStructure of the Learning Process Leadership and
Management The online tool facilitates self-assessment and results in action planning for
improvement
ACL e-Learning Positioning Statement was developed by the National Institute for
Adult and Continuing Education (NIACE) and the Centre for Excellence in Leadership
(CEL) in 2005 It was designed to support institutional strategic planning in e-learning and
institutional quality improvement processes It focused on five core strands Vision and
Strategic Planning Teaching and Learning Staff Development Infrastructure and
Managing ILT and e-Learning
Generator was a self-assessment tool developed through a partnership between LSN Jisc
and BECTA in 2008-2011 Whilst complex it enabled learning providers to identify the
impact that technology had in 9 key business processes against seven enablers of
technology The tool was evaluated against a set of outcome based questions and was
capable of reviewing two or more curriculum areas within an organisation
Review and Plan Technology in Action (RAPTA) was developed by Jisc in 2012-13
with support from Learning and Skills Improvement Service linked to the Ofsted Common
Inspection Framework It aimed to provide a structured framework which can be integrated
in to an organisationrsquos self-assessment process The tool was lengthy with questions
focused on teaching and learning and leadership and management A shorter quick review
version was also available
Higher Education Self-Assessment Frameworks
e-Learning Maturity Model is a mature evaluation framework developed by Dr Stephen Marshall and provides a quality improvement framework for HE institutions in New Zealand It is a complex system focused on online and blended learning The tool benchmarks e-learning capability using 35 inter-related processes It supports external assessment rather than self-assessment as it is externally validated
Industry Self-Assessment Frameworks
Toward Maturity is a mature self-assessment and benchmarking tool which is straightforward but lengthy to use Designed to promote reflection it uses a set of KPIs and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 8 | Coralesce Ltd wwwcoralescecom
identification of best practice to provide a sophisticated benchmarking tool Benchmarking reports are based on 9 KPIs and measure against three levels beginning progressing and proficient
Cross Sector Frameworks
The Open University Digital Literacy Framework identifies the competencies for
learners at HE levels 0123 to M (Level 0 equates to FE Level 3) across five aspects of
competence It provides a self-assessment tool which is presented in short sections The
framework could be used as a basis for individual self-assessment for both learners and
practitioners
The UNESCO Competency Framework for Teachers is an international framework
that supports teachers to develop students to become collaborative problem solving
creative learners It has three aspects Technology Literacy Knowledge Deepening
Knowledge Creation
ANALYSIS BY CRITERIA
The 12 tools were evaluated by criteria as follows
1 Teaching and Learning The school frameworks reviewed have a strong focus on outcomes for learners They use a similar model and focus on classroom based delivery but do not reference online or workplace learning In some cases the positioning of teaching and learning is overshadowed by a focus on leadership and management Apart from the DigiLit Leicester framework the school based tools omit to review the use of technology to support assessment monitoring and feedback The most recent FE and Learning and Skills frameworks eQSF and RAPTA sought to have teaching as a central theme RAPTA has a twin focus on inspiring and challenging learners and encouraging independent learning The eQSF framework however describes processes rather than outcomes The UNESCO framework applies to all sectors focuses on the development of teachers and has desired outcomes for learners as a central feature The Washington State Technology Essential Conditions Rubric has a similar focus Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners Whilst the HE framework eMM focuses on online learning it has a narrow focus on teaching and learning
2 Suitability for Self-assessment The scope of the varying frameworks is similar generally covering core questions on pedagogy outcomes for learners strategic planning and professional development for practitioners The extent to which the frameworks engage with the full range of stakeholders is varied some encouraging completion by one manager within the organisation as in Towards Maturity which can reduce the rigour of the findings In contrast others encourage involvement of learners teachers and managers in the process eg RAPTA and eQSF thus providing a more robust approach to self-assessment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 9 | Coralesce Ltd wwwcoralescecom
3 Quality Improvement A number of organisational frameworks focus on quality improvement such as the Naace and QSF frameworks which provide an on-line action planning tool Other frameworks including Naace provide sources of high quality support materials as does the Open University Digital Literacy Framework
4 Suitability for the whole Sector The tools that link to the Common Inspection Framework could be adapted for use across the sector however they would need adapting to encompass the range of settings represented across the sector particularly workplace learning Frameworks which focus on individual teacher competency such as UNESCO ICT Framework and the DigiLit Leicester would require limited adaptation for use in the FE and Skills Sector
5 Sustainability The tools that have demonstrated the greatest sustainability are the Towards Maturity tool and the Naace tool which have either attracted industry sponsorship to support its annual use over a 10 year period (Towards Maturity) or incorporate small charges for registration and use (Naace)
THEMATIC ANALYSIS
A thematic analysis highlights the question areas that are considered within each tool
covering strategic leadership infrastructure teaching and learning learner experience
staff development financial management and quality improvement All the 12 tools use
structured questions some have dichotomous questions (YES NO) whilst others are based
on levels of measurement (Likert Response Scaling) (Strongly Agree to Strongly Disagree)
or Cumulative Scaling (1-4 or Descriptive Names that indicate a Journey)
In reviewing the frameworks the following conclusions have been drawn which will be
tested at the next stage of consultation within the Learning Technologies Self-Assessment
Project to enable the development of a robust and valued self-assessment tool that builds on
previous work Our conclusions from this desk research are that a self-assessment tool
should
Be explicitly linked to how learning technologies support teaching and learning
Facilitate access by a range of stakeholders by being concise and designed to support
differing approaches to self-assessment
Use structured questions on a meaningful cumulative scale
Model guide and support good self-assessment practice
Provide effective support for quality improvement through for example providing
automatically generated action plans
Be sustainable for the sector once grant funding has ceased
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 10 | Coralesce Ltd wwwcoralescecom
Comparison of the themes used in Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 11 | Coralesce Ltd wwwcoralescecom
5 RESEARCH FINDINGS
A range of frameworks were identified as part of the research and these broadly fell into three categories
Organisational self-assessment
Individual self-assessment
Quality assurance frameworks for e-learning
The major focus of the Education and Training Foundationrsquos Learning Technology Self-Assessment Project is to develop an organisational tool with the potential to integrate individual practitioner self-assessment and therefore the summary below focuses on the first two types of framework Links to quality assurance framework for e-learning are provided in Appendix 1
METHODOLOGY
The methodology used in this Baseline Report was desk-based research to identify a range of self-assessment tools and quality frameworks covering schools further education and learning and skills higher education and industry Whilst the research provided a valuable and comprehensive overview of organisational and individual self-assessment tools and quality frameworks for e-learning it has not always been possible to access full question banks for all tools This is due to the fact that once an organisation responsible for a tool is closed not all archives are kept In the case of BECTA their self-assessment tool Generator was web-based and much of the material including the full question bank was not fully archived at their demise In addition the full set of questions used by Towards Maturity has been difficult to access due to the commercial nature of the organisation Where access has been impeded the research team have sourced a sample of questions through secondary literature
51 Organisational Self-Assessment Frameworks
Naace Self Review Framework (SRF) Sector Schools
Country England
Year 2014
A self-review framework designed to support schools to review their use of technology in a structured way and to support school improvement through reflective practice Developed against a well-researched and evidence based set of criteria Self-Review Framework (SRF) is a mature system originally developed by BECTA and now managed and updated annually by Naace
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 12 | Coralesce Ltd wwwcoralescecom
Naace is a membership organisation and provides an online tool which is accessible only to
member organisations (costs pound100 to join Naace and a further pound50 to access the online
tool) Members are well supported with training in the use of SRF and a useful website A
PDF version of the criteria is free to use
The SRF focuses on supporting pupils to become digitally literate and able to express
themselves and develop ideas through the use of ICT The tool is broad in its approach and
has less of an overall focus on teaching learning and assessment compared to the RAPTA
tool As this is a school based system it has a classroom based focus and does not reflect
the use of technology to support assessment monitoring and target setting
The framework links to the schools Self Evaluation Framework (SEF) and therefore
captures evidence for the annual self-evaluation and for Ofsted inspections The tool
focuses on ICT as well as Learning Technology but has a set of statements in the section on
developing digital literacy which could be adapted The framework consists of
Leadership and Management
Teaching and Learning
Professional Development
Resources
Each section is expanded into a number statements (54 in total) each of which is graded 1
to 4 against a clear set of grade descriptor (figure 1) Whilst the tool is long it is easy to
understand and designed to be used by different staff roles
The online tool provides clear guidance on each aspect and an opportunity to record
evidence and identify actions to improve
The online tool also provides a clear graphical representation of organisational capability
(figure 2) and although not currently benchmarked could be adapted to provide benchmark
data This process would lead to production of a detailed and specific plan for
improvement
The standards section links to an ICT Mark which recognises e-maturity The Mark is
externally assessed and verifies the judgements made in the SRF Following on from the
ICT Mark schools have an opportunity to apply for a Third Millennium Learning Award a
peer-referenced award which demonstrates that schools are providing education fit for the
21st century
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 13 | Coralesce Ltd wwwcoralescecom
Figure 1 Example Question from online SRF Tool
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 14 | Coralesce Ltd wwwcoralescecom
Figure 2 Graphical summary of outputs from online tool
Level Aiming For Enhancing Learning through ICT - level 3
1 Leadership and Management 1a-
1
1a-
2
1b-
1
1b-
2
1b-
3
1b-
4
1b-
5
1b-
6
1c-1 1c-2 1c-3 1c-4
3 4 X 4 3 4 3 3 X X X X
2 Use of ICT in the curriculum 2a-
1
2a-
2
2a-3 2a-4 2a-5 2a-6 2a-7 2b-
1
2b-
2
2b-
3
2b-
4
X X X X X X X X X X X
3 Teaching and Learning 3a-
1
3a-
2
3a-3 3a-4 3a-5 3b-
1
3b-
2
3b-
3
3b-
4
3b-
5
3b-
6
3b-
7
3b-
8
3b-
9
X X X X X X X X X X X X X X
4 Assessment of digital
capability
4a-
1
4a-
2
4a-3
X X X
5 Professional Development 5a-
1
5a-
2
5a-3 5a-4
X X X X
6 Resources 6a-
1
6a-
2
6a-3 6a-4 6a-5 6a-6 6b-
1
6b-
2
6b-
3
6b-
4
X X X X X X X X X X
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 15 | Coralesce Ltd wwwcoralescecom
Washington State Educational Technology Standards and Technology Essential
Conditions Rubric
Sector School
Country USA
Year 2008
A very well constructed model based on research on the benefits and dangers of technology
The standards put the learner at the heart of the system focusing on their expectations
Developed for schools as with the Naace system in England it is classroom focused The
Washington State materials consist of standards for learners and for teachers and are
supplemented with a Technology Essential Rubric for schools to evaluate progress at an
institutional level
The standards emanated from state legislature (1997) which established common learning
goals for all students in Washington State which provided a foundation for the
development of content called Essential Academic Learning Requirements (EALR)
Educational Technology was added to the EALR in 2007 and articulates high expectations
for what students should be able to do across all subject areas The standards for
educational technology were categorised as
Technology Integration
Digital Citizenship (technical literacy moving to technical fluency)
The standards are detailed albeit a little out of date with each EALR broken down into
components each with a grade level expectation for each year of education up to K12 (see
figure 3) Each grade level expectation has a descriptor of evidence of learning As the
standard requires use across subject areas it also provides examples across different subject
areas and levels including basic and 21st century learning environment The standards
provide detailed and useful materials to support teachers and curriculum areas to evaluate
and improve practice They could also be adapted for learner self-evaluation
The self-assessment tool the Technology Essential Conditions Rubric for Schools is
relatively short (8 pages) but would support effective self-assessment and has the potential
for benchmarking It uses a simple grading system beginning progressing proficient and
looks at five key areas
Vision planning and policy
Technical Literacy for learners ndash related to EALRs
Integration standards for teachers ndashrelated to EALRs
Technical competencies of administrators
Communication and Connections
Integration standards for teachers are described in three tiers Teacher Focus on
Productivity Instructional Presentation and Student Productivity Powerful Student-
Centred 21st century Learning Environment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 16 | Coralesce Ltd wwwcoralescecom
EALR 2 mdash DIGITAL CITIZENSHIP
Students demonstrate a clear understanding
of technology systems and operations and
practice safe legal and ethical behavior
Component 21 Practice Safety
Demonstrate safe legal and ethical behavior
in the use of information and technology
GLE Grade 6
212 Practice ethical and respectful behavior
Evidence
of
Learning
Comply with district Acceptable Use Policy (AUP)
Practice ethical and appropriate use of all media and comply with copyright law
Participate and engage with the global community within expected norms of behavior and positive interaction
Recognize report and
counteract cyberbullying
Examples
Recognize different types of illegal and unethical technology use
Identify and cite Creative Commons images in online presentations
Cite all sources properly
Observable indicators are provided for each tier together with helpful examples these could
be useful for practitioner level self-assessment Teachers are supported with extensive
materials available from the Washington State Office of the Superintendent of Public
Instruction wwwk12waus curriculuminstruct
Figure 3 An example of EALR for digital citizenship is shown below
Required
An Essential Academic Learning Requirement (EALR) is a broad statement of the learning that applies to Grades Kndash12 The Component is a statement that further defines and provides more specific information about the EALR There is at least one component for each EALR The Grade Level Expectation (GLE) is a statement containing the essential content or process to be learned and the cognitive demand required to learn it A revision of Bloomrsquos Taxonomy was used to categorize the cognitive demand required of the student Each GLE includes evidence of learning statements which are considered essential to the GLE
The GLE Numbering System identifies the EALR the component and the GLE in that order In the example at right the number 2 indicates the EALR the number 21 indicates the component and the number212 indicates the GLE Note Grade levels are not referenced in the numbering system The Evidence of Learning is a bulleted list of ways students can demonstrate learning considered essential to the GLE Educators are encouraged to identify additional ways in which the student can show proficiency as the educational technology GLEs are integrated across the curriculum Suggested The Examples provide specific illustrations of the learning However these examples are not exhaustive and educators are encouraged to find multiple ways by which learners can demonstrate what they know
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 17 | Coralesce Ltd wwwcoralescecom
Development Self Audit (DSA)
Sector Further Education
Country Scotland
Year 2007
Undertaken by Scottish Colleges as part of a baseline study of e-activity in 2007 the DSA
assessed e-maturity on a broad basis The development was funded by the Scottish Funding
Council and was informed by the practical activities and research of Dr Stephen Marshall
who developed an ideal model of e-maturity (eLearning Maturity Model)
The questions in the DSA cover five aspects
Learning (processes that impact on pedagogical aspects of e-learning)
Development (processes that surround the creation and maintenance of resources)
Support (processes surrounding oversight and management)
Evaluation (processes surrounding the evaluation and quality control)
Organisation (processes associated with planning and management)
As the DSA is based on the eMM developed in New Zealand it also maps the aspects against
the five dimensions of capability
Policy
Planning
Delivery
Management
Improvement
Using external assessment this process leads to the development of a grid which provides a
management level view of the e-maturity of a college an example is shown in figure 6
Figure 6 Management Level E-Maturity Grid
In addition to the completion of the grid the Scottish Funding Council also gathered
feedback on specific issues through the use of a questionnaire completed by institutions and
individuals which looked at the following themes
DSA
Po
licy
Pla
nn
ing
Delivery
Man
ag
em
en
t
Imp
rovem
en
t
Learning
L1 e-Learning is designed according to pedagogically sound principles N P L F P
L2 Learning skills are developed explicitly amp incrementally during delivery P F P P P
L3 Peer amp tutor interactions are provided explicitly during e-learning delivery P P N P P
L4 Courses take account of student workload and e-assessment demands P P N L P
L5 e-learning actively engage students P P P P N
L6 College defined (optional) na na na na na
Development
D1 Course development is directed and supported P P L F P
D2 Resources are maintained and re-used P F P P P
D3 Courses meet the needs of all students P P N P P
D4 The e-learning infrastructure is reliable robust and sufficient P P N L P
D5 The e-learning infrastructure is standardised and integrated P P P P N
D6 College defined (optional) na na na na na
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 18 | Coralesce Ltd wwwcoralescecom
Infrastructure
Teaching and Learning
Organisation
Strategic Leadership
e-Learning coverage by department subject area
The questions covered the extent to which e-learning was used across levels type of
learning and to what extent it was used in Work based Learning An important aspect that
the questionnaire covered under the heading of strategic leadership was the financial
aspects including affordability of software implementation and cost benefit analysis
eQSF A Quality Standards Framework for ICT in Learning
Sector Learning and Skills
Country Northern Ireland
Year 2009 10
eQSF was developed using European funding and provides a comprehensive evaluation and
action planning package to assess practice in the use of existing and new technologies The
framework consists of three pillars each with a series of standards and examples of
evidence
Values Pedagogy and Practice (see figure 4)
Structure of the Learning Process
Leadership and Management
The pillars break down into 13 quality standards each with a separate on-line template to
complete Each standard is graded on a scale of attainment from Absent to Advanced
However the statements tend to describe processes rather than outcomes and are quite
broad particularly in the section on Teaching and Learning
The online resource facilitates the self-assessment process provides opportunities to
identify evidence and results in an action plan for improvement The implementation
process involves establishing leadership and ownership reviewing practice action
planning implementing action plan evaluating progress and feedback
Positive feedback is available from the three pilot case studies that were developed
including North West College in NI which indicate the tool has been used at teacher
department and organisational level
http wwweqsfeu
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 19 | Coralesce Ltd wwwcoralescecom
Figure 4 Values Pedagogy amp Practice Standards
Standard Addressed
Yes No
Actions
Completed Current Attainment Level Signature
11
The organisation supports inclusiveness and accessibility through the use of ICT and sees ICT as promoting equality and diversity in the learning process
Absent Acquiring Achieved Advanced
out of
12
The organisation facilitates the adaptation of learning materials to meet a range of learner requirements and sees ICT as a mechanism for delivering such flexibility for the practitioner and learner
Absent Acquiring Achieved Advanced
out of
13
The organisation uses ICT to extend learning beyond the classroom and supports progression to further learning and career opportunities
Absent Acquiring Achieved Advanced
out of
14
The organisation is committed to the learnersrsquo outcomes being positively impacted by ICT
Absent Acquiring Achieved Advanced
out of
Comment
Adult and Community Learning e-Learning Positioning Statement
Sector Adult and Community Learning
Country England
Date 2005-2007
The ACL eLPs were designed by CEL (Centre for Excellence in Leadership) and NIACE
(National Institute for Adult amp Continuing Education) as an aid to both institutional
strategic planning in e-learning and to support institutional quality improvement processes
As well as its use within NIACEs e-learning management and leadership courses the tool
was also embedded in individual consultancies and within the NIACE Digital Learning
teams capital investment funding work The ACL eLPs tool was superseded by BECTAs
Technology Improvement Leadership Tool - Generator NIACE formed part of the
development team for Generator and continued to contribute to its development and
piloting with community-based adult learning providers The use of the Generator tool was
integrated into NIACEs e-learning capital funding projects (CAMEL) The e-Learning
positioning statement comprised 31 elements grouped under five themes Organisations
graded themselves against a five point scale ranging from not yet started to embedded The
ACL eLearning Positioning Statement was designed as an aid to both institutional strategic
planning in e-learning and to support institutional quality improvement processes
The framework reviewed Vision and Strategic Planning Teaching and Learning Staff
Development Infrastructure and Managing ILT and e-Learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 6 | Coralesce Ltd wwwcoralescecom
Practitioners
Curriculum managers
ICT managers
Quality managers
Senior managers
Governors or equivalent
Employers
The tool needs to be able to be integrated successfully with the collegersquos existing self-
assessment and quality improvement processes Accessibility ease of use and length of the
tool also impacts on its suitability for self-assessment
4 Quality improvement
Effective self-assessment should lead to quality improvement This criterion considers
whether the tool effectively supports quality improvement by either supporting
development via easy to use links to resources ideas best practice exemplars training
materials and or eliciting quantitative data for benchmarking To ensure quality
improvement is appropriate and effective the tool needs to ensure that the data collected is
accurate and this may include strategies to ensure rigour
5 Sustainable
This criterion considers whether the tool is sustainable and capable of supporting
continuous quality improvement in the use of learning technologies for the sector in the
years to come
SECTOR ANALYSIS
The 12 tools were evaluated by sector as follows
School Sector Self-Assessment Frameworks
The Naace Self Review Framework (SRF) provides a framework for schools which is
designed to support them to review the use of technology in a structured way and to support
school improvement through reflective practice The framework links to the schools Self
Evaluation Framework (SEF) and therefore captures evidence for the annual self-evaluation
and for Ofsted inspections The tool focuses on ICT as well as Learning Technology but
also includes a set of statements in the section on developing pupilsrsquo digital literacy
The Washington State Technology Essential Conditions Rubric evaluates
institution progress against the Washington State Educational Standards the Essential
Academic Learning Requirements The rubric is relatively short and grades institution as
beginning progressing or proficient The rubric looks at five key aspects Vision planning
and policy technical literacy for learners integration standards for teachers technical
competencies of administrators communications and connections
DigiLit Leicester is a digital literacy framework developed as a result of lsquoBuilding Schools
for the Futurersquo ICT priorities It assesses competency against six strands of digital literacy
in the secondary school sector It puts learning and learners at the centre with a very clear
focus on outcomes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 7 | Coralesce Ltd wwwcoralescecom
FE and Skills Self-Assessment Frameworks
The Development Self Audit (DSA) was undertaken by Scottish Colleges as part of a
baseline study of e-activity in 2007 It is based on practical activities and adapted from
research by Dr Stephen Marshall who developed the eLearning Maturity Model for higher
education institutions in New Zealand The self-audit reviews five aspects Learning
Development Support Evaluation and Organisation As it is based on the eMM it also
maps the aspects against five dimensions of capability Policy Planning Delivery
Management Improvement These are then mapped to produce a management level e-
maturity grid
eQSF A Quality Standards Framework for ICT in Learning was developed using
European funding and piloted in North Ireland in 2009 10 It provides a comprehensive
evaluation and action planning tool consisting of thirteen questions covering three pillars
Values Pedagogy and Practice eStructure of the Learning Process Leadership and
Management The online tool facilitates self-assessment and results in action planning for
improvement
ACL e-Learning Positioning Statement was developed by the National Institute for
Adult and Continuing Education (NIACE) and the Centre for Excellence in Leadership
(CEL) in 2005 It was designed to support institutional strategic planning in e-learning and
institutional quality improvement processes It focused on five core strands Vision and
Strategic Planning Teaching and Learning Staff Development Infrastructure and
Managing ILT and e-Learning
Generator was a self-assessment tool developed through a partnership between LSN Jisc
and BECTA in 2008-2011 Whilst complex it enabled learning providers to identify the
impact that technology had in 9 key business processes against seven enablers of
technology The tool was evaluated against a set of outcome based questions and was
capable of reviewing two or more curriculum areas within an organisation
Review and Plan Technology in Action (RAPTA) was developed by Jisc in 2012-13
with support from Learning and Skills Improvement Service linked to the Ofsted Common
Inspection Framework It aimed to provide a structured framework which can be integrated
in to an organisationrsquos self-assessment process The tool was lengthy with questions
focused on teaching and learning and leadership and management A shorter quick review
version was also available
Higher Education Self-Assessment Frameworks
e-Learning Maturity Model is a mature evaluation framework developed by Dr Stephen Marshall and provides a quality improvement framework for HE institutions in New Zealand It is a complex system focused on online and blended learning The tool benchmarks e-learning capability using 35 inter-related processes It supports external assessment rather than self-assessment as it is externally validated
Industry Self-Assessment Frameworks
Toward Maturity is a mature self-assessment and benchmarking tool which is straightforward but lengthy to use Designed to promote reflection it uses a set of KPIs and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 8 | Coralesce Ltd wwwcoralescecom
identification of best practice to provide a sophisticated benchmarking tool Benchmarking reports are based on 9 KPIs and measure against three levels beginning progressing and proficient
Cross Sector Frameworks
The Open University Digital Literacy Framework identifies the competencies for
learners at HE levels 0123 to M (Level 0 equates to FE Level 3) across five aspects of
competence It provides a self-assessment tool which is presented in short sections The
framework could be used as a basis for individual self-assessment for both learners and
practitioners
The UNESCO Competency Framework for Teachers is an international framework
that supports teachers to develop students to become collaborative problem solving
creative learners It has three aspects Technology Literacy Knowledge Deepening
Knowledge Creation
ANALYSIS BY CRITERIA
The 12 tools were evaluated by criteria as follows
1 Teaching and Learning The school frameworks reviewed have a strong focus on outcomes for learners They use a similar model and focus on classroom based delivery but do not reference online or workplace learning In some cases the positioning of teaching and learning is overshadowed by a focus on leadership and management Apart from the DigiLit Leicester framework the school based tools omit to review the use of technology to support assessment monitoring and feedback The most recent FE and Learning and Skills frameworks eQSF and RAPTA sought to have teaching as a central theme RAPTA has a twin focus on inspiring and challenging learners and encouraging independent learning The eQSF framework however describes processes rather than outcomes The UNESCO framework applies to all sectors focuses on the development of teachers and has desired outcomes for learners as a central feature The Washington State Technology Essential Conditions Rubric has a similar focus Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners Whilst the HE framework eMM focuses on online learning it has a narrow focus on teaching and learning
2 Suitability for Self-assessment The scope of the varying frameworks is similar generally covering core questions on pedagogy outcomes for learners strategic planning and professional development for practitioners The extent to which the frameworks engage with the full range of stakeholders is varied some encouraging completion by one manager within the organisation as in Towards Maturity which can reduce the rigour of the findings In contrast others encourage involvement of learners teachers and managers in the process eg RAPTA and eQSF thus providing a more robust approach to self-assessment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 9 | Coralesce Ltd wwwcoralescecom
3 Quality Improvement A number of organisational frameworks focus on quality improvement such as the Naace and QSF frameworks which provide an on-line action planning tool Other frameworks including Naace provide sources of high quality support materials as does the Open University Digital Literacy Framework
4 Suitability for the whole Sector The tools that link to the Common Inspection Framework could be adapted for use across the sector however they would need adapting to encompass the range of settings represented across the sector particularly workplace learning Frameworks which focus on individual teacher competency such as UNESCO ICT Framework and the DigiLit Leicester would require limited adaptation for use in the FE and Skills Sector
5 Sustainability The tools that have demonstrated the greatest sustainability are the Towards Maturity tool and the Naace tool which have either attracted industry sponsorship to support its annual use over a 10 year period (Towards Maturity) or incorporate small charges for registration and use (Naace)
THEMATIC ANALYSIS
A thematic analysis highlights the question areas that are considered within each tool
covering strategic leadership infrastructure teaching and learning learner experience
staff development financial management and quality improvement All the 12 tools use
structured questions some have dichotomous questions (YES NO) whilst others are based
on levels of measurement (Likert Response Scaling) (Strongly Agree to Strongly Disagree)
or Cumulative Scaling (1-4 or Descriptive Names that indicate a Journey)
In reviewing the frameworks the following conclusions have been drawn which will be
tested at the next stage of consultation within the Learning Technologies Self-Assessment
Project to enable the development of a robust and valued self-assessment tool that builds on
previous work Our conclusions from this desk research are that a self-assessment tool
should
Be explicitly linked to how learning technologies support teaching and learning
Facilitate access by a range of stakeholders by being concise and designed to support
differing approaches to self-assessment
Use structured questions on a meaningful cumulative scale
Model guide and support good self-assessment practice
Provide effective support for quality improvement through for example providing
automatically generated action plans
Be sustainable for the sector once grant funding has ceased
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 10 | Coralesce Ltd wwwcoralescecom
Comparison of the themes used in Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 11 | Coralesce Ltd wwwcoralescecom
5 RESEARCH FINDINGS
A range of frameworks were identified as part of the research and these broadly fell into three categories
Organisational self-assessment
Individual self-assessment
Quality assurance frameworks for e-learning
The major focus of the Education and Training Foundationrsquos Learning Technology Self-Assessment Project is to develop an organisational tool with the potential to integrate individual practitioner self-assessment and therefore the summary below focuses on the first two types of framework Links to quality assurance framework for e-learning are provided in Appendix 1
METHODOLOGY
The methodology used in this Baseline Report was desk-based research to identify a range of self-assessment tools and quality frameworks covering schools further education and learning and skills higher education and industry Whilst the research provided a valuable and comprehensive overview of organisational and individual self-assessment tools and quality frameworks for e-learning it has not always been possible to access full question banks for all tools This is due to the fact that once an organisation responsible for a tool is closed not all archives are kept In the case of BECTA their self-assessment tool Generator was web-based and much of the material including the full question bank was not fully archived at their demise In addition the full set of questions used by Towards Maturity has been difficult to access due to the commercial nature of the organisation Where access has been impeded the research team have sourced a sample of questions through secondary literature
51 Organisational Self-Assessment Frameworks
Naace Self Review Framework (SRF) Sector Schools
Country England
Year 2014
A self-review framework designed to support schools to review their use of technology in a structured way and to support school improvement through reflective practice Developed against a well-researched and evidence based set of criteria Self-Review Framework (SRF) is a mature system originally developed by BECTA and now managed and updated annually by Naace
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 12 | Coralesce Ltd wwwcoralescecom
Naace is a membership organisation and provides an online tool which is accessible only to
member organisations (costs pound100 to join Naace and a further pound50 to access the online
tool) Members are well supported with training in the use of SRF and a useful website A
PDF version of the criteria is free to use
The SRF focuses on supporting pupils to become digitally literate and able to express
themselves and develop ideas through the use of ICT The tool is broad in its approach and
has less of an overall focus on teaching learning and assessment compared to the RAPTA
tool As this is a school based system it has a classroom based focus and does not reflect
the use of technology to support assessment monitoring and target setting
The framework links to the schools Self Evaluation Framework (SEF) and therefore
captures evidence for the annual self-evaluation and for Ofsted inspections The tool
focuses on ICT as well as Learning Technology but has a set of statements in the section on
developing digital literacy which could be adapted The framework consists of
Leadership and Management
Teaching and Learning
Professional Development
Resources
Each section is expanded into a number statements (54 in total) each of which is graded 1
to 4 against a clear set of grade descriptor (figure 1) Whilst the tool is long it is easy to
understand and designed to be used by different staff roles
The online tool provides clear guidance on each aspect and an opportunity to record
evidence and identify actions to improve
The online tool also provides a clear graphical representation of organisational capability
(figure 2) and although not currently benchmarked could be adapted to provide benchmark
data This process would lead to production of a detailed and specific plan for
improvement
The standards section links to an ICT Mark which recognises e-maturity The Mark is
externally assessed and verifies the judgements made in the SRF Following on from the
ICT Mark schools have an opportunity to apply for a Third Millennium Learning Award a
peer-referenced award which demonstrates that schools are providing education fit for the
21st century
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 13 | Coralesce Ltd wwwcoralescecom
Figure 1 Example Question from online SRF Tool
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 14 | Coralesce Ltd wwwcoralescecom
Figure 2 Graphical summary of outputs from online tool
Level Aiming For Enhancing Learning through ICT - level 3
1 Leadership and Management 1a-
1
1a-
2
1b-
1
1b-
2
1b-
3
1b-
4
1b-
5
1b-
6
1c-1 1c-2 1c-3 1c-4
3 4 X 4 3 4 3 3 X X X X
2 Use of ICT in the curriculum 2a-
1
2a-
2
2a-3 2a-4 2a-5 2a-6 2a-7 2b-
1
2b-
2
2b-
3
2b-
4
X X X X X X X X X X X
3 Teaching and Learning 3a-
1
3a-
2
3a-3 3a-4 3a-5 3b-
1
3b-
2
3b-
3
3b-
4
3b-
5
3b-
6
3b-
7
3b-
8
3b-
9
X X X X X X X X X X X X X X
4 Assessment of digital
capability
4a-
1
4a-
2
4a-3
X X X
5 Professional Development 5a-
1
5a-
2
5a-3 5a-4
X X X X
6 Resources 6a-
1
6a-
2
6a-3 6a-4 6a-5 6a-6 6b-
1
6b-
2
6b-
3
6b-
4
X X X X X X X X X X
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 15 | Coralesce Ltd wwwcoralescecom
Washington State Educational Technology Standards and Technology Essential
Conditions Rubric
Sector School
Country USA
Year 2008
A very well constructed model based on research on the benefits and dangers of technology
The standards put the learner at the heart of the system focusing on their expectations
Developed for schools as with the Naace system in England it is classroom focused The
Washington State materials consist of standards for learners and for teachers and are
supplemented with a Technology Essential Rubric for schools to evaluate progress at an
institutional level
The standards emanated from state legislature (1997) which established common learning
goals for all students in Washington State which provided a foundation for the
development of content called Essential Academic Learning Requirements (EALR)
Educational Technology was added to the EALR in 2007 and articulates high expectations
for what students should be able to do across all subject areas The standards for
educational technology were categorised as
Technology Integration
Digital Citizenship (technical literacy moving to technical fluency)
The standards are detailed albeit a little out of date with each EALR broken down into
components each with a grade level expectation for each year of education up to K12 (see
figure 3) Each grade level expectation has a descriptor of evidence of learning As the
standard requires use across subject areas it also provides examples across different subject
areas and levels including basic and 21st century learning environment The standards
provide detailed and useful materials to support teachers and curriculum areas to evaluate
and improve practice They could also be adapted for learner self-evaluation
The self-assessment tool the Technology Essential Conditions Rubric for Schools is
relatively short (8 pages) but would support effective self-assessment and has the potential
for benchmarking It uses a simple grading system beginning progressing proficient and
looks at five key areas
Vision planning and policy
Technical Literacy for learners ndash related to EALRs
Integration standards for teachers ndashrelated to EALRs
Technical competencies of administrators
Communication and Connections
Integration standards for teachers are described in three tiers Teacher Focus on
Productivity Instructional Presentation and Student Productivity Powerful Student-
Centred 21st century Learning Environment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 16 | Coralesce Ltd wwwcoralescecom
EALR 2 mdash DIGITAL CITIZENSHIP
Students demonstrate a clear understanding
of technology systems and operations and
practice safe legal and ethical behavior
Component 21 Practice Safety
Demonstrate safe legal and ethical behavior
in the use of information and technology
GLE Grade 6
212 Practice ethical and respectful behavior
Evidence
of
Learning
Comply with district Acceptable Use Policy (AUP)
Practice ethical and appropriate use of all media and comply with copyright law
Participate and engage with the global community within expected norms of behavior and positive interaction
Recognize report and
counteract cyberbullying
Examples
Recognize different types of illegal and unethical technology use
Identify and cite Creative Commons images in online presentations
Cite all sources properly
Observable indicators are provided for each tier together with helpful examples these could
be useful for practitioner level self-assessment Teachers are supported with extensive
materials available from the Washington State Office of the Superintendent of Public
Instruction wwwk12waus curriculuminstruct
Figure 3 An example of EALR for digital citizenship is shown below
Required
An Essential Academic Learning Requirement (EALR) is a broad statement of the learning that applies to Grades Kndash12 The Component is a statement that further defines and provides more specific information about the EALR There is at least one component for each EALR The Grade Level Expectation (GLE) is a statement containing the essential content or process to be learned and the cognitive demand required to learn it A revision of Bloomrsquos Taxonomy was used to categorize the cognitive demand required of the student Each GLE includes evidence of learning statements which are considered essential to the GLE
The GLE Numbering System identifies the EALR the component and the GLE in that order In the example at right the number 2 indicates the EALR the number 21 indicates the component and the number212 indicates the GLE Note Grade levels are not referenced in the numbering system The Evidence of Learning is a bulleted list of ways students can demonstrate learning considered essential to the GLE Educators are encouraged to identify additional ways in which the student can show proficiency as the educational technology GLEs are integrated across the curriculum Suggested The Examples provide specific illustrations of the learning However these examples are not exhaustive and educators are encouraged to find multiple ways by which learners can demonstrate what they know
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 17 | Coralesce Ltd wwwcoralescecom
Development Self Audit (DSA)
Sector Further Education
Country Scotland
Year 2007
Undertaken by Scottish Colleges as part of a baseline study of e-activity in 2007 the DSA
assessed e-maturity on a broad basis The development was funded by the Scottish Funding
Council and was informed by the practical activities and research of Dr Stephen Marshall
who developed an ideal model of e-maturity (eLearning Maturity Model)
The questions in the DSA cover five aspects
Learning (processes that impact on pedagogical aspects of e-learning)
Development (processes that surround the creation and maintenance of resources)
Support (processes surrounding oversight and management)
Evaluation (processes surrounding the evaluation and quality control)
Organisation (processes associated with planning and management)
As the DSA is based on the eMM developed in New Zealand it also maps the aspects against
the five dimensions of capability
Policy
Planning
Delivery
Management
Improvement
Using external assessment this process leads to the development of a grid which provides a
management level view of the e-maturity of a college an example is shown in figure 6
Figure 6 Management Level E-Maturity Grid
In addition to the completion of the grid the Scottish Funding Council also gathered
feedback on specific issues through the use of a questionnaire completed by institutions and
individuals which looked at the following themes
DSA
Po
licy
Pla
nn
ing
Delivery
Man
ag
em
en
t
Imp
rovem
en
t
Learning
L1 e-Learning is designed according to pedagogically sound principles N P L F P
L2 Learning skills are developed explicitly amp incrementally during delivery P F P P P
L3 Peer amp tutor interactions are provided explicitly during e-learning delivery P P N P P
L4 Courses take account of student workload and e-assessment demands P P N L P
L5 e-learning actively engage students P P P P N
L6 College defined (optional) na na na na na
Development
D1 Course development is directed and supported P P L F P
D2 Resources are maintained and re-used P F P P P
D3 Courses meet the needs of all students P P N P P
D4 The e-learning infrastructure is reliable robust and sufficient P P N L P
D5 The e-learning infrastructure is standardised and integrated P P P P N
D6 College defined (optional) na na na na na
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 18 | Coralesce Ltd wwwcoralescecom
Infrastructure
Teaching and Learning
Organisation
Strategic Leadership
e-Learning coverage by department subject area
The questions covered the extent to which e-learning was used across levels type of
learning and to what extent it was used in Work based Learning An important aspect that
the questionnaire covered under the heading of strategic leadership was the financial
aspects including affordability of software implementation and cost benefit analysis
eQSF A Quality Standards Framework for ICT in Learning
Sector Learning and Skills
Country Northern Ireland
Year 2009 10
eQSF was developed using European funding and provides a comprehensive evaluation and
action planning package to assess practice in the use of existing and new technologies The
framework consists of three pillars each with a series of standards and examples of
evidence
Values Pedagogy and Practice (see figure 4)
Structure of the Learning Process
Leadership and Management
The pillars break down into 13 quality standards each with a separate on-line template to
complete Each standard is graded on a scale of attainment from Absent to Advanced
However the statements tend to describe processes rather than outcomes and are quite
broad particularly in the section on Teaching and Learning
The online resource facilitates the self-assessment process provides opportunities to
identify evidence and results in an action plan for improvement The implementation
process involves establishing leadership and ownership reviewing practice action
planning implementing action plan evaluating progress and feedback
Positive feedback is available from the three pilot case studies that were developed
including North West College in NI which indicate the tool has been used at teacher
department and organisational level
http wwweqsfeu
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 19 | Coralesce Ltd wwwcoralescecom
Figure 4 Values Pedagogy amp Practice Standards
Standard Addressed
Yes No
Actions
Completed Current Attainment Level Signature
11
The organisation supports inclusiveness and accessibility through the use of ICT and sees ICT as promoting equality and diversity in the learning process
Absent Acquiring Achieved Advanced
out of
12
The organisation facilitates the adaptation of learning materials to meet a range of learner requirements and sees ICT as a mechanism for delivering such flexibility for the practitioner and learner
Absent Acquiring Achieved Advanced
out of
13
The organisation uses ICT to extend learning beyond the classroom and supports progression to further learning and career opportunities
Absent Acquiring Achieved Advanced
out of
14
The organisation is committed to the learnersrsquo outcomes being positively impacted by ICT
Absent Acquiring Achieved Advanced
out of
Comment
Adult and Community Learning e-Learning Positioning Statement
Sector Adult and Community Learning
Country England
Date 2005-2007
The ACL eLPs were designed by CEL (Centre for Excellence in Leadership) and NIACE
(National Institute for Adult amp Continuing Education) as an aid to both institutional
strategic planning in e-learning and to support institutional quality improvement processes
As well as its use within NIACEs e-learning management and leadership courses the tool
was also embedded in individual consultancies and within the NIACE Digital Learning
teams capital investment funding work The ACL eLPs tool was superseded by BECTAs
Technology Improvement Leadership Tool - Generator NIACE formed part of the
development team for Generator and continued to contribute to its development and
piloting with community-based adult learning providers The use of the Generator tool was
integrated into NIACEs e-learning capital funding projects (CAMEL) The e-Learning
positioning statement comprised 31 elements grouped under five themes Organisations
graded themselves against a five point scale ranging from not yet started to embedded The
ACL eLearning Positioning Statement was designed as an aid to both institutional strategic
planning in e-learning and to support institutional quality improvement processes
The framework reviewed Vision and Strategic Planning Teaching and Learning Staff
Development Infrastructure and Managing ILT and e-Learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 7 | Coralesce Ltd wwwcoralescecom
FE and Skills Self-Assessment Frameworks
The Development Self Audit (DSA) was undertaken by Scottish Colleges as part of a
baseline study of e-activity in 2007 It is based on practical activities and adapted from
research by Dr Stephen Marshall who developed the eLearning Maturity Model for higher
education institutions in New Zealand The self-audit reviews five aspects Learning
Development Support Evaluation and Organisation As it is based on the eMM it also
maps the aspects against five dimensions of capability Policy Planning Delivery
Management Improvement These are then mapped to produce a management level e-
maturity grid
eQSF A Quality Standards Framework for ICT in Learning was developed using
European funding and piloted in North Ireland in 2009 10 It provides a comprehensive
evaluation and action planning tool consisting of thirteen questions covering three pillars
Values Pedagogy and Practice eStructure of the Learning Process Leadership and
Management The online tool facilitates self-assessment and results in action planning for
improvement
ACL e-Learning Positioning Statement was developed by the National Institute for
Adult and Continuing Education (NIACE) and the Centre for Excellence in Leadership
(CEL) in 2005 It was designed to support institutional strategic planning in e-learning and
institutional quality improvement processes It focused on five core strands Vision and
Strategic Planning Teaching and Learning Staff Development Infrastructure and
Managing ILT and e-Learning
Generator was a self-assessment tool developed through a partnership between LSN Jisc
and BECTA in 2008-2011 Whilst complex it enabled learning providers to identify the
impact that technology had in 9 key business processes against seven enablers of
technology The tool was evaluated against a set of outcome based questions and was
capable of reviewing two or more curriculum areas within an organisation
Review and Plan Technology in Action (RAPTA) was developed by Jisc in 2012-13
with support from Learning and Skills Improvement Service linked to the Ofsted Common
Inspection Framework It aimed to provide a structured framework which can be integrated
in to an organisationrsquos self-assessment process The tool was lengthy with questions
focused on teaching and learning and leadership and management A shorter quick review
version was also available
Higher Education Self-Assessment Frameworks
e-Learning Maturity Model is a mature evaluation framework developed by Dr Stephen Marshall and provides a quality improvement framework for HE institutions in New Zealand It is a complex system focused on online and blended learning The tool benchmarks e-learning capability using 35 inter-related processes It supports external assessment rather than self-assessment as it is externally validated
Industry Self-Assessment Frameworks
Toward Maturity is a mature self-assessment and benchmarking tool which is straightforward but lengthy to use Designed to promote reflection it uses a set of KPIs and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 8 | Coralesce Ltd wwwcoralescecom
identification of best practice to provide a sophisticated benchmarking tool Benchmarking reports are based on 9 KPIs and measure against three levels beginning progressing and proficient
Cross Sector Frameworks
The Open University Digital Literacy Framework identifies the competencies for
learners at HE levels 0123 to M (Level 0 equates to FE Level 3) across five aspects of
competence It provides a self-assessment tool which is presented in short sections The
framework could be used as a basis for individual self-assessment for both learners and
practitioners
The UNESCO Competency Framework for Teachers is an international framework
that supports teachers to develop students to become collaborative problem solving
creative learners It has three aspects Technology Literacy Knowledge Deepening
Knowledge Creation
ANALYSIS BY CRITERIA
The 12 tools were evaluated by criteria as follows
1 Teaching and Learning The school frameworks reviewed have a strong focus on outcomes for learners They use a similar model and focus on classroom based delivery but do not reference online or workplace learning In some cases the positioning of teaching and learning is overshadowed by a focus on leadership and management Apart from the DigiLit Leicester framework the school based tools omit to review the use of technology to support assessment monitoring and feedback The most recent FE and Learning and Skills frameworks eQSF and RAPTA sought to have teaching as a central theme RAPTA has a twin focus on inspiring and challenging learners and encouraging independent learning The eQSF framework however describes processes rather than outcomes The UNESCO framework applies to all sectors focuses on the development of teachers and has desired outcomes for learners as a central feature The Washington State Technology Essential Conditions Rubric has a similar focus Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners Whilst the HE framework eMM focuses on online learning it has a narrow focus on teaching and learning
2 Suitability for Self-assessment The scope of the varying frameworks is similar generally covering core questions on pedagogy outcomes for learners strategic planning and professional development for practitioners The extent to which the frameworks engage with the full range of stakeholders is varied some encouraging completion by one manager within the organisation as in Towards Maturity which can reduce the rigour of the findings In contrast others encourage involvement of learners teachers and managers in the process eg RAPTA and eQSF thus providing a more robust approach to self-assessment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 9 | Coralesce Ltd wwwcoralescecom
3 Quality Improvement A number of organisational frameworks focus on quality improvement such as the Naace and QSF frameworks which provide an on-line action planning tool Other frameworks including Naace provide sources of high quality support materials as does the Open University Digital Literacy Framework
4 Suitability for the whole Sector The tools that link to the Common Inspection Framework could be adapted for use across the sector however they would need adapting to encompass the range of settings represented across the sector particularly workplace learning Frameworks which focus on individual teacher competency such as UNESCO ICT Framework and the DigiLit Leicester would require limited adaptation for use in the FE and Skills Sector
5 Sustainability The tools that have demonstrated the greatest sustainability are the Towards Maturity tool and the Naace tool which have either attracted industry sponsorship to support its annual use over a 10 year period (Towards Maturity) or incorporate small charges for registration and use (Naace)
THEMATIC ANALYSIS
A thematic analysis highlights the question areas that are considered within each tool
covering strategic leadership infrastructure teaching and learning learner experience
staff development financial management and quality improvement All the 12 tools use
structured questions some have dichotomous questions (YES NO) whilst others are based
on levels of measurement (Likert Response Scaling) (Strongly Agree to Strongly Disagree)
or Cumulative Scaling (1-4 or Descriptive Names that indicate a Journey)
In reviewing the frameworks the following conclusions have been drawn which will be
tested at the next stage of consultation within the Learning Technologies Self-Assessment
Project to enable the development of a robust and valued self-assessment tool that builds on
previous work Our conclusions from this desk research are that a self-assessment tool
should
Be explicitly linked to how learning technologies support teaching and learning
Facilitate access by a range of stakeholders by being concise and designed to support
differing approaches to self-assessment
Use structured questions on a meaningful cumulative scale
Model guide and support good self-assessment practice
Provide effective support for quality improvement through for example providing
automatically generated action plans
Be sustainable for the sector once grant funding has ceased
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 10 | Coralesce Ltd wwwcoralescecom
Comparison of the themes used in Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 11 | Coralesce Ltd wwwcoralescecom
5 RESEARCH FINDINGS
A range of frameworks were identified as part of the research and these broadly fell into three categories
Organisational self-assessment
Individual self-assessment
Quality assurance frameworks for e-learning
The major focus of the Education and Training Foundationrsquos Learning Technology Self-Assessment Project is to develop an organisational tool with the potential to integrate individual practitioner self-assessment and therefore the summary below focuses on the first two types of framework Links to quality assurance framework for e-learning are provided in Appendix 1
METHODOLOGY
The methodology used in this Baseline Report was desk-based research to identify a range of self-assessment tools and quality frameworks covering schools further education and learning and skills higher education and industry Whilst the research provided a valuable and comprehensive overview of organisational and individual self-assessment tools and quality frameworks for e-learning it has not always been possible to access full question banks for all tools This is due to the fact that once an organisation responsible for a tool is closed not all archives are kept In the case of BECTA their self-assessment tool Generator was web-based and much of the material including the full question bank was not fully archived at their demise In addition the full set of questions used by Towards Maturity has been difficult to access due to the commercial nature of the organisation Where access has been impeded the research team have sourced a sample of questions through secondary literature
51 Organisational Self-Assessment Frameworks
Naace Self Review Framework (SRF) Sector Schools
Country England
Year 2014
A self-review framework designed to support schools to review their use of technology in a structured way and to support school improvement through reflective practice Developed against a well-researched and evidence based set of criteria Self-Review Framework (SRF) is a mature system originally developed by BECTA and now managed and updated annually by Naace
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 12 | Coralesce Ltd wwwcoralescecom
Naace is a membership organisation and provides an online tool which is accessible only to
member organisations (costs pound100 to join Naace and a further pound50 to access the online
tool) Members are well supported with training in the use of SRF and a useful website A
PDF version of the criteria is free to use
The SRF focuses on supporting pupils to become digitally literate and able to express
themselves and develop ideas through the use of ICT The tool is broad in its approach and
has less of an overall focus on teaching learning and assessment compared to the RAPTA
tool As this is a school based system it has a classroom based focus and does not reflect
the use of technology to support assessment monitoring and target setting
The framework links to the schools Self Evaluation Framework (SEF) and therefore
captures evidence for the annual self-evaluation and for Ofsted inspections The tool
focuses on ICT as well as Learning Technology but has a set of statements in the section on
developing digital literacy which could be adapted The framework consists of
Leadership and Management
Teaching and Learning
Professional Development
Resources
Each section is expanded into a number statements (54 in total) each of which is graded 1
to 4 against a clear set of grade descriptor (figure 1) Whilst the tool is long it is easy to
understand and designed to be used by different staff roles
The online tool provides clear guidance on each aspect and an opportunity to record
evidence and identify actions to improve
The online tool also provides a clear graphical representation of organisational capability
(figure 2) and although not currently benchmarked could be adapted to provide benchmark
data This process would lead to production of a detailed and specific plan for
improvement
The standards section links to an ICT Mark which recognises e-maturity The Mark is
externally assessed and verifies the judgements made in the SRF Following on from the
ICT Mark schools have an opportunity to apply for a Third Millennium Learning Award a
peer-referenced award which demonstrates that schools are providing education fit for the
21st century
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 13 | Coralesce Ltd wwwcoralescecom
Figure 1 Example Question from online SRF Tool
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 14 | Coralesce Ltd wwwcoralescecom
Figure 2 Graphical summary of outputs from online tool
Level Aiming For Enhancing Learning through ICT - level 3
1 Leadership and Management 1a-
1
1a-
2
1b-
1
1b-
2
1b-
3
1b-
4
1b-
5
1b-
6
1c-1 1c-2 1c-3 1c-4
3 4 X 4 3 4 3 3 X X X X
2 Use of ICT in the curriculum 2a-
1
2a-
2
2a-3 2a-4 2a-5 2a-6 2a-7 2b-
1
2b-
2
2b-
3
2b-
4
X X X X X X X X X X X
3 Teaching and Learning 3a-
1
3a-
2
3a-3 3a-4 3a-5 3b-
1
3b-
2
3b-
3
3b-
4
3b-
5
3b-
6
3b-
7
3b-
8
3b-
9
X X X X X X X X X X X X X X
4 Assessment of digital
capability
4a-
1
4a-
2
4a-3
X X X
5 Professional Development 5a-
1
5a-
2
5a-3 5a-4
X X X X
6 Resources 6a-
1
6a-
2
6a-3 6a-4 6a-5 6a-6 6b-
1
6b-
2
6b-
3
6b-
4
X X X X X X X X X X
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 15 | Coralesce Ltd wwwcoralescecom
Washington State Educational Technology Standards and Technology Essential
Conditions Rubric
Sector School
Country USA
Year 2008
A very well constructed model based on research on the benefits and dangers of technology
The standards put the learner at the heart of the system focusing on their expectations
Developed for schools as with the Naace system in England it is classroom focused The
Washington State materials consist of standards for learners and for teachers and are
supplemented with a Technology Essential Rubric for schools to evaluate progress at an
institutional level
The standards emanated from state legislature (1997) which established common learning
goals for all students in Washington State which provided a foundation for the
development of content called Essential Academic Learning Requirements (EALR)
Educational Technology was added to the EALR in 2007 and articulates high expectations
for what students should be able to do across all subject areas The standards for
educational technology were categorised as
Technology Integration
Digital Citizenship (technical literacy moving to technical fluency)
The standards are detailed albeit a little out of date with each EALR broken down into
components each with a grade level expectation for each year of education up to K12 (see
figure 3) Each grade level expectation has a descriptor of evidence of learning As the
standard requires use across subject areas it also provides examples across different subject
areas and levels including basic and 21st century learning environment The standards
provide detailed and useful materials to support teachers and curriculum areas to evaluate
and improve practice They could also be adapted for learner self-evaluation
The self-assessment tool the Technology Essential Conditions Rubric for Schools is
relatively short (8 pages) but would support effective self-assessment and has the potential
for benchmarking It uses a simple grading system beginning progressing proficient and
looks at five key areas
Vision planning and policy
Technical Literacy for learners ndash related to EALRs
Integration standards for teachers ndashrelated to EALRs
Technical competencies of administrators
Communication and Connections
Integration standards for teachers are described in three tiers Teacher Focus on
Productivity Instructional Presentation and Student Productivity Powerful Student-
Centred 21st century Learning Environment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 16 | Coralesce Ltd wwwcoralescecom
EALR 2 mdash DIGITAL CITIZENSHIP
Students demonstrate a clear understanding
of technology systems and operations and
practice safe legal and ethical behavior
Component 21 Practice Safety
Demonstrate safe legal and ethical behavior
in the use of information and technology
GLE Grade 6
212 Practice ethical and respectful behavior
Evidence
of
Learning
Comply with district Acceptable Use Policy (AUP)
Practice ethical and appropriate use of all media and comply with copyright law
Participate and engage with the global community within expected norms of behavior and positive interaction
Recognize report and
counteract cyberbullying
Examples
Recognize different types of illegal and unethical technology use
Identify and cite Creative Commons images in online presentations
Cite all sources properly
Observable indicators are provided for each tier together with helpful examples these could
be useful for practitioner level self-assessment Teachers are supported with extensive
materials available from the Washington State Office of the Superintendent of Public
Instruction wwwk12waus curriculuminstruct
Figure 3 An example of EALR for digital citizenship is shown below
Required
An Essential Academic Learning Requirement (EALR) is a broad statement of the learning that applies to Grades Kndash12 The Component is a statement that further defines and provides more specific information about the EALR There is at least one component for each EALR The Grade Level Expectation (GLE) is a statement containing the essential content or process to be learned and the cognitive demand required to learn it A revision of Bloomrsquos Taxonomy was used to categorize the cognitive demand required of the student Each GLE includes evidence of learning statements which are considered essential to the GLE
The GLE Numbering System identifies the EALR the component and the GLE in that order In the example at right the number 2 indicates the EALR the number 21 indicates the component and the number212 indicates the GLE Note Grade levels are not referenced in the numbering system The Evidence of Learning is a bulleted list of ways students can demonstrate learning considered essential to the GLE Educators are encouraged to identify additional ways in which the student can show proficiency as the educational technology GLEs are integrated across the curriculum Suggested The Examples provide specific illustrations of the learning However these examples are not exhaustive and educators are encouraged to find multiple ways by which learners can demonstrate what they know
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 17 | Coralesce Ltd wwwcoralescecom
Development Self Audit (DSA)
Sector Further Education
Country Scotland
Year 2007
Undertaken by Scottish Colleges as part of a baseline study of e-activity in 2007 the DSA
assessed e-maturity on a broad basis The development was funded by the Scottish Funding
Council and was informed by the practical activities and research of Dr Stephen Marshall
who developed an ideal model of e-maturity (eLearning Maturity Model)
The questions in the DSA cover five aspects
Learning (processes that impact on pedagogical aspects of e-learning)
Development (processes that surround the creation and maintenance of resources)
Support (processes surrounding oversight and management)
Evaluation (processes surrounding the evaluation and quality control)
Organisation (processes associated with planning and management)
As the DSA is based on the eMM developed in New Zealand it also maps the aspects against
the five dimensions of capability
Policy
Planning
Delivery
Management
Improvement
Using external assessment this process leads to the development of a grid which provides a
management level view of the e-maturity of a college an example is shown in figure 6
Figure 6 Management Level E-Maturity Grid
In addition to the completion of the grid the Scottish Funding Council also gathered
feedback on specific issues through the use of a questionnaire completed by institutions and
individuals which looked at the following themes
DSA
Po
licy
Pla
nn
ing
Delivery
Man
ag
em
en
t
Imp
rovem
en
t
Learning
L1 e-Learning is designed according to pedagogically sound principles N P L F P
L2 Learning skills are developed explicitly amp incrementally during delivery P F P P P
L3 Peer amp tutor interactions are provided explicitly during e-learning delivery P P N P P
L4 Courses take account of student workload and e-assessment demands P P N L P
L5 e-learning actively engage students P P P P N
L6 College defined (optional) na na na na na
Development
D1 Course development is directed and supported P P L F P
D2 Resources are maintained and re-used P F P P P
D3 Courses meet the needs of all students P P N P P
D4 The e-learning infrastructure is reliable robust and sufficient P P N L P
D5 The e-learning infrastructure is standardised and integrated P P P P N
D6 College defined (optional) na na na na na
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 18 | Coralesce Ltd wwwcoralescecom
Infrastructure
Teaching and Learning
Organisation
Strategic Leadership
e-Learning coverage by department subject area
The questions covered the extent to which e-learning was used across levels type of
learning and to what extent it was used in Work based Learning An important aspect that
the questionnaire covered under the heading of strategic leadership was the financial
aspects including affordability of software implementation and cost benefit analysis
eQSF A Quality Standards Framework for ICT in Learning
Sector Learning and Skills
Country Northern Ireland
Year 2009 10
eQSF was developed using European funding and provides a comprehensive evaluation and
action planning package to assess practice in the use of existing and new technologies The
framework consists of three pillars each with a series of standards and examples of
evidence
Values Pedagogy and Practice (see figure 4)
Structure of the Learning Process
Leadership and Management
The pillars break down into 13 quality standards each with a separate on-line template to
complete Each standard is graded on a scale of attainment from Absent to Advanced
However the statements tend to describe processes rather than outcomes and are quite
broad particularly in the section on Teaching and Learning
The online resource facilitates the self-assessment process provides opportunities to
identify evidence and results in an action plan for improvement The implementation
process involves establishing leadership and ownership reviewing practice action
planning implementing action plan evaluating progress and feedback
Positive feedback is available from the three pilot case studies that were developed
including North West College in NI which indicate the tool has been used at teacher
department and organisational level
http wwweqsfeu
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 19 | Coralesce Ltd wwwcoralescecom
Figure 4 Values Pedagogy amp Practice Standards
Standard Addressed
Yes No
Actions
Completed Current Attainment Level Signature
11
The organisation supports inclusiveness and accessibility through the use of ICT and sees ICT as promoting equality and diversity in the learning process
Absent Acquiring Achieved Advanced
out of
12
The organisation facilitates the adaptation of learning materials to meet a range of learner requirements and sees ICT as a mechanism for delivering such flexibility for the practitioner and learner
Absent Acquiring Achieved Advanced
out of
13
The organisation uses ICT to extend learning beyond the classroom and supports progression to further learning and career opportunities
Absent Acquiring Achieved Advanced
out of
14
The organisation is committed to the learnersrsquo outcomes being positively impacted by ICT
Absent Acquiring Achieved Advanced
out of
Comment
Adult and Community Learning e-Learning Positioning Statement
Sector Adult and Community Learning
Country England
Date 2005-2007
The ACL eLPs were designed by CEL (Centre for Excellence in Leadership) and NIACE
(National Institute for Adult amp Continuing Education) as an aid to both institutional
strategic planning in e-learning and to support institutional quality improvement processes
As well as its use within NIACEs e-learning management and leadership courses the tool
was also embedded in individual consultancies and within the NIACE Digital Learning
teams capital investment funding work The ACL eLPs tool was superseded by BECTAs
Technology Improvement Leadership Tool - Generator NIACE formed part of the
development team for Generator and continued to contribute to its development and
piloting with community-based adult learning providers The use of the Generator tool was
integrated into NIACEs e-learning capital funding projects (CAMEL) The e-Learning
positioning statement comprised 31 elements grouped under five themes Organisations
graded themselves against a five point scale ranging from not yet started to embedded The
ACL eLearning Positioning Statement was designed as an aid to both institutional strategic
planning in e-learning and to support institutional quality improvement processes
The framework reviewed Vision and Strategic Planning Teaching and Learning Staff
Development Infrastructure and Managing ILT and e-Learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 8 | Coralesce Ltd wwwcoralescecom
identification of best practice to provide a sophisticated benchmarking tool Benchmarking reports are based on 9 KPIs and measure against three levels beginning progressing and proficient
Cross Sector Frameworks
The Open University Digital Literacy Framework identifies the competencies for
learners at HE levels 0123 to M (Level 0 equates to FE Level 3) across five aspects of
competence It provides a self-assessment tool which is presented in short sections The
framework could be used as a basis for individual self-assessment for both learners and
practitioners
The UNESCO Competency Framework for Teachers is an international framework
that supports teachers to develop students to become collaborative problem solving
creative learners It has three aspects Technology Literacy Knowledge Deepening
Knowledge Creation
ANALYSIS BY CRITERIA
The 12 tools were evaluated by criteria as follows
1 Teaching and Learning The school frameworks reviewed have a strong focus on outcomes for learners They use a similar model and focus on classroom based delivery but do not reference online or workplace learning In some cases the positioning of teaching and learning is overshadowed by a focus on leadership and management Apart from the DigiLit Leicester framework the school based tools omit to review the use of technology to support assessment monitoring and feedback The most recent FE and Learning and Skills frameworks eQSF and RAPTA sought to have teaching as a central theme RAPTA has a twin focus on inspiring and challenging learners and encouraging independent learning The eQSF framework however describes processes rather than outcomes The UNESCO framework applies to all sectors focuses on the development of teachers and has desired outcomes for learners as a central feature The Washington State Technology Essential Conditions Rubric has a similar focus Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners Whilst the HE framework eMM focuses on online learning it has a narrow focus on teaching and learning
2 Suitability for Self-assessment The scope of the varying frameworks is similar generally covering core questions on pedagogy outcomes for learners strategic planning and professional development for practitioners The extent to which the frameworks engage with the full range of stakeholders is varied some encouraging completion by one manager within the organisation as in Towards Maturity which can reduce the rigour of the findings In contrast others encourage involvement of learners teachers and managers in the process eg RAPTA and eQSF thus providing a more robust approach to self-assessment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 9 | Coralesce Ltd wwwcoralescecom
3 Quality Improvement A number of organisational frameworks focus on quality improvement such as the Naace and QSF frameworks which provide an on-line action planning tool Other frameworks including Naace provide sources of high quality support materials as does the Open University Digital Literacy Framework
4 Suitability for the whole Sector The tools that link to the Common Inspection Framework could be adapted for use across the sector however they would need adapting to encompass the range of settings represented across the sector particularly workplace learning Frameworks which focus on individual teacher competency such as UNESCO ICT Framework and the DigiLit Leicester would require limited adaptation for use in the FE and Skills Sector
5 Sustainability The tools that have demonstrated the greatest sustainability are the Towards Maturity tool and the Naace tool which have either attracted industry sponsorship to support its annual use over a 10 year period (Towards Maturity) or incorporate small charges for registration and use (Naace)
THEMATIC ANALYSIS
A thematic analysis highlights the question areas that are considered within each tool
covering strategic leadership infrastructure teaching and learning learner experience
staff development financial management and quality improvement All the 12 tools use
structured questions some have dichotomous questions (YES NO) whilst others are based
on levels of measurement (Likert Response Scaling) (Strongly Agree to Strongly Disagree)
or Cumulative Scaling (1-4 or Descriptive Names that indicate a Journey)
In reviewing the frameworks the following conclusions have been drawn which will be
tested at the next stage of consultation within the Learning Technologies Self-Assessment
Project to enable the development of a robust and valued self-assessment tool that builds on
previous work Our conclusions from this desk research are that a self-assessment tool
should
Be explicitly linked to how learning technologies support teaching and learning
Facilitate access by a range of stakeholders by being concise and designed to support
differing approaches to self-assessment
Use structured questions on a meaningful cumulative scale
Model guide and support good self-assessment practice
Provide effective support for quality improvement through for example providing
automatically generated action plans
Be sustainable for the sector once grant funding has ceased
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 10 | Coralesce Ltd wwwcoralescecom
Comparison of the themes used in Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 11 | Coralesce Ltd wwwcoralescecom
5 RESEARCH FINDINGS
A range of frameworks were identified as part of the research and these broadly fell into three categories
Organisational self-assessment
Individual self-assessment
Quality assurance frameworks for e-learning
The major focus of the Education and Training Foundationrsquos Learning Technology Self-Assessment Project is to develop an organisational tool with the potential to integrate individual practitioner self-assessment and therefore the summary below focuses on the first two types of framework Links to quality assurance framework for e-learning are provided in Appendix 1
METHODOLOGY
The methodology used in this Baseline Report was desk-based research to identify a range of self-assessment tools and quality frameworks covering schools further education and learning and skills higher education and industry Whilst the research provided a valuable and comprehensive overview of organisational and individual self-assessment tools and quality frameworks for e-learning it has not always been possible to access full question banks for all tools This is due to the fact that once an organisation responsible for a tool is closed not all archives are kept In the case of BECTA their self-assessment tool Generator was web-based and much of the material including the full question bank was not fully archived at their demise In addition the full set of questions used by Towards Maturity has been difficult to access due to the commercial nature of the organisation Where access has been impeded the research team have sourced a sample of questions through secondary literature
51 Organisational Self-Assessment Frameworks
Naace Self Review Framework (SRF) Sector Schools
Country England
Year 2014
A self-review framework designed to support schools to review their use of technology in a structured way and to support school improvement through reflective practice Developed against a well-researched and evidence based set of criteria Self-Review Framework (SRF) is a mature system originally developed by BECTA and now managed and updated annually by Naace
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 12 | Coralesce Ltd wwwcoralescecom
Naace is a membership organisation and provides an online tool which is accessible only to
member organisations (costs pound100 to join Naace and a further pound50 to access the online
tool) Members are well supported with training in the use of SRF and a useful website A
PDF version of the criteria is free to use
The SRF focuses on supporting pupils to become digitally literate and able to express
themselves and develop ideas through the use of ICT The tool is broad in its approach and
has less of an overall focus on teaching learning and assessment compared to the RAPTA
tool As this is a school based system it has a classroom based focus and does not reflect
the use of technology to support assessment monitoring and target setting
The framework links to the schools Self Evaluation Framework (SEF) and therefore
captures evidence for the annual self-evaluation and for Ofsted inspections The tool
focuses on ICT as well as Learning Technology but has a set of statements in the section on
developing digital literacy which could be adapted The framework consists of
Leadership and Management
Teaching and Learning
Professional Development
Resources
Each section is expanded into a number statements (54 in total) each of which is graded 1
to 4 against a clear set of grade descriptor (figure 1) Whilst the tool is long it is easy to
understand and designed to be used by different staff roles
The online tool provides clear guidance on each aspect and an opportunity to record
evidence and identify actions to improve
The online tool also provides a clear graphical representation of organisational capability
(figure 2) and although not currently benchmarked could be adapted to provide benchmark
data This process would lead to production of a detailed and specific plan for
improvement
The standards section links to an ICT Mark which recognises e-maturity The Mark is
externally assessed and verifies the judgements made in the SRF Following on from the
ICT Mark schools have an opportunity to apply for a Third Millennium Learning Award a
peer-referenced award which demonstrates that schools are providing education fit for the
21st century
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 13 | Coralesce Ltd wwwcoralescecom
Figure 1 Example Question from online SRF Tool
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 14 | Coralesce Ltd wwwcoralescecom
Figure 2 Graphical summary of outputs from online tool
Level Aiming For Enhancing Learning through ICT - level 3
1 Leadership and Management 1a-
1
1a-
2
1b-
1
1b-
2
1b-
3
1b-
4
1b-
5
1b-
6
1c-1 1c-2 1c-3 1c-4
3 4 X 4 3 4 3 3 X X X X
2 Use of ICT in the curriculum 2a-
1
2a-
2
2a-3 2a-4 2a-5 2a-6 2a-7 2b-
1
2b-
2
2b-
3
2b-
4
X X X X X X X X X X X
3 Teaching and Learning 3a-
1
3a-
2
3a-3 3a-4 3a-5 3b-
1
3b-
2
3b-
3
3b-
4
3b-
5
3b-
6
3b-
7
3b-
8
3b-
9
X X X X X X X X X X X X X X
4 Assessment of digital
capability
4a-
1
4a-
2
4a-3
X X X
5 Professional Development 5a-
1
5a-
2
5a-3 5a-4
X X X X
6 Resources 6a-
1
6a-
2
6a-3 6a-4 6a-5 6a-6 6b-
1
6b-
2
6b-
3
6b-
4
X X X X X X X X X X
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 15 | Coralesce Ltd wwwcoralescecom
Washington State Educational Technology Standards and Technology Essential
Conditions Rubric
Sector School
Country USA
Year 2008
A very well constructed model based on research on the benefits and dangers of technology
The standards put the learner at the heart of the system focusing on their expectations
Developed for schools as with the Naace system in England it is classroom focused The
Washington State materials consist of standards for learners and for teachers and are
supplemented with a Technology Essential Rubric for schools to evaluate progress at an
institutional level
The standards emanated from state legislature (1997) which established common learning
goals for all students in Washington State which provided a foundation for the
development of content called Essential Academic Learning Requirements (EALR)
Educational Technology was added to the EALR in 2007 and articulates high expectations
for what students should be able to do across all subject areas The standards for
educational technology were categorised as
Technology Integration
Digital Citizenship (technical literacy moving to technical fluency)
The standards are detailed albeit a little out of date with each EALR broken down into
components each with a grade level expectation for each year of education up to K12 (see
figure 3) Each grade level expectation has a descriptor of evidence of learning As the
standard requires use across subject areas it also provides examples across different subject
areas and levels including basic and 21st century learning environment The standards
provide detailed and useful materials to support teachers and curriculum areas to evaluate
and improve practice They could also be adapted for learner self-evaluation
The self-assessment tool the Technology Essential Conditions Rubric for Schools is
relatively short (8 pages) but would support effective self-assessment and has the potential
for benchmarking It uses a simple grading system beginning progressing proficient and
looks at five key areas
Vision planning and policy
Technical Literacy for learners ndash related to EALRs
Integration standards for teachers ndashrelated to EALRs
Technical competencies of administrators
Communication and Connections
Integration standards for teachers are described in three tiers Teacher Focus on
Productivity Instructional Presentation and Student Productivity Powerful Student-
Centred 21st century Learning Environment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 16 | Coralesce Ltd wwwcoralescecom
EALR 2 mdash DIGITAL CITIZENSHIP
Students demonstrate a clear understanding
of technology systems and operations and
practice safe legal and ethical behavior
Component 21 Practice Safety
Demonstrate safe legal and ethical behavior
in the use of information and technology
GLE Grade 6
212 Practice ethical and respectful behavior
Evidence
of
Learning
Comply with district Acceptable Use Policy (AUP)
Practice ethical and appropriate use of all media and comply with copyright law
Participate and engage with the global community within expected norms of behavior and positive interaction
Recognize report and
counteract cyberbullying
Examples
Recognize different types of illegal and unethical technology use
Identify and cite Creative Commons images in online presentations
Cite all sources properly
Observable indicators are provided for each tier together with helpful examples these could
be useful for practitioner level self-assessment Teachers are supported with extensive
materials available from the Washington State Office of the Superintendent of Public
Instruction wwwk12waus curriculuminstruct
Figure 3 An example of EALR for digital citizenship is shown below
Required
An Essential Academic Learning Requirement (EALR) is a broad statement of the learning that applies to Grades Kndash12 The Component is a statement that further defines and provides more specific information about the EALR There is at least one component for each EALR The Grade Level Expectation (GLE) is a statement containing the essential content or process to be learned and the cognitive demand required to learn it A revision of Bloomrsquos Taxonomy was used to categorize the cognitive demand required of the student Each GLE includes evidence of learning statements which are considered essential to the GLE
The GLE Numbering System identifies the EALR the component and the GLE in that order In the example at right the number 2 indicates the EALR the number 21 indicates the component and the number212 indicates the GLE Note Grade levels are not referenced in the numbering system The Evidence of Learning is a bulleted list of ways students can demonstrate learning considered essential to the GLE Educators are encouraged to identify additional ways in which the student can show proficiency as the educational technology GLEs are integrated across the curriculum Suggested The Examples provide specific illustrations of the learning However these examples are not exhaustive and educators are encouraged to find multiple ways by which learners can demonstrate what they know
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 17 | Coralesce Ltd wwwcoralescecom
Development Self Audit (DSA)
Sector Further Education
Country Scotland
Year 2007
Undertaken by Scottish Colleges as part of a baseline study of e-activity in 2007 the DSA
assessed e-maturity on a broad basis The development was funded by the Scottish Funding
Council and was informed by the practical activities and research of Dr Stephen Marshall
who developed an ideal model of e-maturity (eLearning Maturity Model)
The questions in the DSA cover five aspects
Learning (processes that impact on pedagogical aspects of e-learning)
Development (processes that surround the creation and maintenance of resources)
Support (processes surrounding oversight and management)
Evaluation (processes surrounding the evaluation and quality control)
Organisation (processes associated with planning and management)
As the DSA is based on the eMM developed in New Zealand it also maps the aspects against
the five dimensions of capability
Policy
Planning
Delivery
Management
Improvement
Using external assessment this process leads to the development of a grid which provides a
management level view of the e-maturity of a college an example is shown in figure 6
Figure 6 Management Level E-Maturity Grid
In addition to the completion of the grid the Scottish Funding Council also gathered
feedback on specific issues through the use of a questionnaire completed by institutions and
individuals which looked at the following themes
DSA
Po
licy
Pla
nn
ing
Delivery
Man
ag
em
en
t
Imp
rovem
en
t
Learning
L1 e-Learning is designed according to pedagogically sound principles N P L F P
L2 Learning skills are developed explicitly amp incrementally during delivery P F P P P
L3 Peer amp tutor interactions are provided explicitly during e-learning delivery P P N P P
L4 Courses take account of student workload and e-assessment demands P P N L P
L5 e-learning actively engage students P P P P N
L6 College defined (optional) na na na na na
Development
D1 Course development is directed and supported P P L F P
D2 Resources are maintained and re-used P F P P P
D3 Courses meet the needs of all students P P N P P
D4 The e-learning infrastructure is reliable robust and sufficient P P N L P
D5 The e-learning infrastructure is standardised and integrated P P P P N
D6 College defined (optional) na na na na na
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 18 | Coralesce Ltd wwwcoralescecom
Infrastructure
Teaching and Learning
Organisation
Strategic Leadership
e-Learning coverage by department subject area
The questions covered the extent to which e-learning was used across levels type of
learning and to what extent it was used in Work based Learning An important aspect that
the questionnaire covered under the heading of strategic leadership was the financial
aspects including affordability of software implementation and cost benefit analysis
eQSF A Quality Standards Framework for ICT in Learning
Sector Learning and Skills
Country Northern Ireland
Year 2009 10
eQSF was developed using European funding and provides a comprehensive evaluation and
action planning package to assess practice in the use of existing and new technologies The
framework consists of three pillars each with a series of standards and examples of
evidence
Values Pedagogy and Practice (see figure 4)
Structure of the Learning Process
Leadership and Management
The pillars break down into 13 quality standards each with a separate on-line template to
complete Each standard is graded on a scale of attainment from Absent to Advanced
However the statements tend to describe processes rather than outcomes and are quite
broad particularly in the section on Teaching and Learning
The online resource facilitates the self-assessment process provides opportunities to
identify evidence and results in an action plan for improvement The implementation
process involves establishing leadership and ownership reviewing practice action
planning implementing action plan evaluating progress and feedback
Positive feedback is available from the three pilot case studies that were developed
including North West College in NI which indicate the tool has been used at teacher
department and organisational level
http wwweqsfeu
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 19 | Coralesce Ltd wwwcoralescecom
Figure 4 Values Pedagogy amp Practice Standards
Standard Addressed
Yes No
Actions
Completed Current Attainment Level Signature
11
The organisation supports inclusiveness and accessibility through the use of ICT and sees ICT as promoting equality and diversity in the learning process
Absent Acquiring Achieved Advanced
out of
12
The organisation facilitates the adaptation of learning materials to meet a range of learner requirements and sees ICT as a mechanism for delivering such flexibility for the practitioner and learner
Absent Acquiring Achieved Advanced
out of
13
The organisation uses ICT to extend learning beyond the classroom and supports progression to further learning and career opportunities
Absent Acquiring Achieved Advanced
out of
14
The organisation is committed to the learnersrsquo outcomes being positively impacted by ICT
Absent Acquiring Achieved Advanced
out of
Comment
Adult and Community Learning e-Learning Positioning Statement
Sector Adult and Community Learning
Country England
Date 2005-2007
The ACL eLPs were designed by CEL (Centre for Excellence in Leadership) and NIACE
(National Institute for Adult amp Continuing Education) as an aid to both institutional
strategic planning in e-learning and to support institutional quality improvement processes
As well as its use within NIACEs e-learning management and leadership courses the tool
was also embedded in individual consultancies and within the NIACE Digital Learning
teams capital investment funding work The ACL eLPs tool was superseded by BECTAs
Technology Improvement Leadership Tool - Generator NIACE formed part of the
development team for Generator and continued to contribute to its development and
piloting with community-based adult learning providers The use of the Generator tool was
integrated into NIACEs e-learning capital funding projects (CAMEL) The e-Learning
positioning statement comprised 31 elements grouped under five themes Organisations
graded themselves against a five point scale ranging from not yet started to embedded The
ACL eLearning Positioning Statement was designed as an aid to both institutional strategic
planning in e-learning and to support institutional quality improvement processes
The framework reviewed Vision and Strategic Planning Teaching and Learning Staff
Development Infrastructure and Managing ILT and e-Learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 9 | Coralesce Ltd wwwcoralescecom
3 Quality Improvement A number of organisational frameworks focus on quality improvement such as the Naace and QSF frameworks which provide an on-line action planning tool Other frameworks including Naace provide sources of high quality support materials as does the Open University Digital Literacy Framework
4 Suitability for the whole Sector The tools that link to the Common Inspection Framework could be adapted for use across the sector however they would need adapting to encompass the range of settings represented across the sector particularly workplace learning Frameworks which focus on individual teacher competency such as UNESCO ICT Framework and the DigiLit Leicester would require limited adaptation for use in the FE and Skills Sector
5 Sustainability The tools that have demonstrated the greatest sustainability are the Towards Maturity tool and the Naace tool which have either attracted industry sponsorship to support its annual use over a 10 year period (Towards Maturity) or incorporate small charges for registration and use (Naace)
THEMATIC ANALYSIS
A thematic analysis highlights the question areas that are considered within each tool
covering strategic leadership infrastructure teaching and learning learner experience
staff development financial management and quality improvement All the 12 tools use
structured questions some have dichotomous questions (YES NO) whilst others are based
on levels of measurement (Likert Response Scaling) (Strongly Agree to Strongly Disagree)
or Cumulative Scaling (1-4 or Descriptive Names that indicate a Journey)
In reviewing the frameworks the following conclusions have been drawn which will be
tested at the next stage of consultation within the Learning Technologies Self-Assessment
Project to enable the development of a robust and valued self-assessment tool that builds on
previous work Our conclusions from this desk research are that a self-assessment tool
should
Be explicitly linked to how learning technologies support teaching and learning
Facilitate access by a range of stakeholders by being concise and designed to support
differing approaches to self-assessment
Use structured questions on a meaningful cumulative scale
Model guide and support good self-assessment practice
Provide effective support for quality improvement through for example providing
automatically generated action plans
Be sustainable for the sector once grant funding has ceased
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 10 | Coralesce Ltd wwwcoralescecom
Comparison of the themes used in Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 11 | Coralesce Ltd wwwcoralescecom
5 RESEARCH FINDINGS
A range of frameworks were identified as part of the research and these broadly fell into three categories
Organisational self-assessment
Individual self-assessment
Quality assurance frameworks for e-learning
The major focus of the Education and Training Foundationrsquos Learning Technology Self-Assessment Project is to develop an organisational tool with the potential to integrate individual practitioner self-assessment and therefore the summary below focuses on the first two types of framework Links to quality assurance framework for e-learning are provided in Appendix 1
METHODOLOGY
The methodology used in this Baseline Report was desk-based research to identify a range of self-assessment tools and quality frameworks covering schools further education and learning and skills higher education and industry Whilst the research provided a valuable and comprehensive overview of organisational and individual self-assessment tools and quality frameworks for e-learning it has not always been possible to access full question banks for all tools This is due to the fact that once an organisation responsible for a tool is closed not all archives are kept In the case of BECTA their self-assessment tool Generator was web-based and much of the material including the full question bank was not fully archived at their demise In addition the full set of questions used by Towards Maturity has been difficult to access due to the commercial nature of the organisation Where access has been impeded the research team have sourced a sample of questions through secondary literature
51 Organisational Self-Assessment Frameworks
Naace Self Review Framework (SRF) Sector Schools
Country England
Year 2014
A self-review framework designed to support schools to review their use of technology in a structured way and to support school improvement through reflective practice Developed against a well-researched and evidence based set of criteria Self-Review Framework (SRF) is a mature system originally developed by BECTA and now managed and updated annually by Naace
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 12 | Coralesce Ltd wwwcoralescecom
Naace is a membership organisation and provides an online tool which is accessible only to
member organisations (costs pound100 to join Naace and a further pound50 to access the online
tool) Members are well supported with training in the use of SRF and a useful website A
PDF version of the criteria is free to use
The SRF focuses on supporting pupils to become digitally literate and able to express
themselves and develop ideas through the use of ICT The tool is broad in its approach and
has less of an overall focus on teaching learning and assessment compared to the RAPTA
tool As this is a school based system it has a classroom based focus and does not reflect
the use of technology to support assessment monitoring and target setting
The framework links to the schools Self Evaluation Framework (SEF) and therefore
captures evidence for the annual self-evaluation and for Ofsted inspections The tool
focuses on ICT as well as Learning Technology but has a set of statements in the section on
developing digital literacy which could be adapted The framework consists of
Leadership and Management
Teaching and Learning
Professional Development
Resources
Each section is expanded into a number statements (54 in total) each of which is graded 1
to 4 against a clear set of grade descriptor (figure 1) Whilst the tool is long it is easy to
understand and designed to be used by different staff roles
The online tool provides clear guidance on each aspect and an opportunity to record
evidence and identify actions to improve
The online tool also provides a clear graphical representation of organisational capability
(figure 2) and although not currently benchmarked could be adapted to provide benchmark
data This process would lead to production of a detailed and specific plan for
improvement
The standards section links to an ICT Mark which recognises e-maturity The Mark is
externally assessed and verifies the judgements made in the SRF Following on from the
ICT Mark schools have an opportunity to apply for a Third Millennium Learning Award a
peer-referenced award which demonstrates that schools are providing education fit for the
21st century
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 13 | Coralesce Ltd wwwcoralescecom
Figure 1 Example Question from online SRF Tool
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 14 | Coralesce Ltd wwwcoralescecom
Figure 2 Graphical summary of outputs from online tool
Level Aiming For Enhancing Learning through ICT - level 3
1 Leadership and Management 1a-
1
1a-
2
1b-
1
1b-
2
1b-
3
1b-
4
1b-
5
1b-
6
1c-1 1c-2 1c-3 1c-4
3 4 X 4 3 4 3 3 X X X X
2 Use of ICT in the curriculum 2a-
1
2a-
2
2a-3 2a-4 2a-5 2a-6 2a-7 2b-
1
2b-
2
2b-
3
2b-
4
X X X X X X X X X X X
3 Teaching and Learning 3a-
1
3a-
2
3a-3 3a-4 3a-5 3b-
1
3b-
2
3b-
3
3b-
4
3b-
5
3b-
6
3b-
7
3b-
8
3b-
9
X X X X X X X X X X X X X X
4 Assessment of digital
capability
4a-
1
4a-
2
4a-3
X X X
5 Professional Development 5a-
1
5a-
2
5a-3 5a-4
X X X X
6 Resources 6a-
1
6a-
2
6a-3 6a-4 6a-5 6a-6 6b-
1
6b-
2
6b-
3
6b-
4
X X X X X X X X X X
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 15 | Coralesce Ltd wwwcoralescecom
Washington State Educational Technology Standards and Technology Essential
Conditions Rubric
Sector School
Country USA
Year 2008
A very well constructed model based on research on the benefits and dangers of technology
The standards put the learner at the heart of the system focusing on their expectations
Developed for schools as with the Naace system in England it is classroom focused The
Washington State materials consist of standards for learners and for teachers and are
supplemented with a Technology Essential Rubric for schools to evaluate progress at an
institutional level
The standards emanated from state legislature (1997) which established common learning
goals for all students in Washington State which provided a foundation for the
development of content called Essential Academic Learning Requirements (EALR)
Educational Technology was added to the EALR in 2007 and articulates high expectations
for what students should be able to do across all subject areas The standards for
educational technology were categorised as
Technology Integration
Digital Citizenship (technical literacy moving to technical fluency)
The standards are detailed albeit a little out of date with each EALR broken down into
components each with a grade level expectation for each year of education up to K12 (see
figure 3) Each grade level expectation has a descriptor of evidence of learning As the
standard requires use across subject areas it also provides examples across different subject
areas and levels including basic and 21st century learning environment The standards
provide detailed and useful materials to support teachers and curriculum areas to evaluate
and improve practice They could also be adapted for learner self-evaluation
The self-assessment tool the Technology Essential Conditions Rubric for Schools is
relatively short (8 pages) but would support effective self-assessment and has the potential
for benchmarking It uses a simple grading system beginning progressing proficient and
looks at five key areas
Vision planning and policy
Technical Literacy for learners ndash related to EALRs
Integration standards for teachers ndashrelated to EALRs
Technical competencies of administrators
Communication and Connections
Integration standards for teachers are described in three tiers Teacher Focus on
Productivity Instructional Presentation and Student Productivity Powerful Student-
Centred 21st century Learning Environment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 16 | Coralesce Ltd wwwcoralescecom
EALR 2 mdash DIGITAL CITIZENSHIP
Students demonstrate a clear understanding
of technology systems and operations and
practice safe legal and ethical behavior
Component 21 Practice Safety
Demonstrate safe legal and ethical behavior
in the use of information and technology
GLE Grade 6
212 Practice ethical and respectful behavior
Evidence
of
Learning
Comply with district Acceptable Use Policy (AUP)
Practice ethical and appropriate use of all media and comply with copyright law
Participate and engage with the global community within expected norms of behavior and positive interaction
Recognize report and
counteract cyberbullying
Examples
Recognize different types of illegal and unethical technology use
Identify and cite Creative Commons images in online presentations
Cite all sources properly
Observable indicators are provided for each tier together with helpful examples these could
be useful for practitioner level self-assessment Teachers are supported with extensive
materials available from the Washington State Office of the Superintendent of Public
Instruction wwwk12waus curriculuminstruct
Figure 3 An example of EALR for digital citizenship is shown below
Required
An Essential Academic Learning Requirement (EALR) is a broad statement of the learning that applies to Grades Kndash12 The Component is a statement that further defines and provides more specific information about the EALR There is at least one component for each EALR The Grade Level Expectation (GLE) is a statement containing the essential content or process to be learned and the cognitive demand required to learn it A revision of Bloomrsquos Taxonomy was used to categorize the cognitive demand required of the student Each GLE includes evidence of learning statements which are considered essential to the GLE
The GLE Numbering System identifies the EALR the component and the GLE in that order In the example at right the number 2 indicates the EALR the number 21 indicates the component and the number212 indicates the GLE Note Grade levels are not referenced in the numbering system The Evidence of Learning is a bulleted list of ways students can demonstrate learning considered essential to the GLE Educators are encouraged to identify additional ways in which the student can show proficiency as the educational technology GLEs are integrated across the curriculum Suggested The Examples provide specific illustrations of the learning However these examples are not exhaustive and educators are encouraged to find multiple ways by which learners can demonstrate what they know
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 17 | Coralesce Ltd wwwcoralescecom
Development Self Audit (DSA)
Sector Further Education
Country Scotland
Year 2007
Undertaken by Scottish Colleges as part of a baseline study of e-activity in 2007 the DSA
assessed e-maturity on a broad basis The development was funded by the Scottish Funding
Council and was informed by the practical activities and research of Dr Stephen Marshall
who developed an ideal model of e-maturity (eLearning Maturity Model)
The questions in the DSA cover five aspects
Learning (processes that impact on pedagogical aspects of e-learning)
Development (processes that surround the creation and maintenance of resources)
Support (processes surrounding oversight and management)
Evaluation (processes surrounding the evaluation and quality control)
Organisation (processes associated with planning and management)
As the DSA is based on the eMM developed in New Zealand it also maps the aspects against
the five dimensions of capability
Policy
Planning
Delivery
Management
Improvement
Using external assessment this process leads to the development of a grid which provides a
management level view of the e-maturity of a college an example is shown in figure 6
Figure 6 Management Level E-Maturity Grid
In addition to the completion of the grid the Scottish Funding Council also gathered
feedback on specific issues through the use of a questionnaire completed by institutions and
individuals which looked at the following themes
DSA
Po
licy
Pla
nn
ing
Delivery
Man
ag
em
en
t
Imp
rovem
en
t
Learning
L1 e-Learning is designed according to pedagogically sound principles N P L F P
L2 Learning skills are developed explicitly amp incrementally during delivery P F P P P
L3 Peer amp tutor interactions are provided explicitly during e-learning delivery P P N P P
L4 Courses take account of student workload and e-assessment demands P P N L P
L5 e-learning actively engage students P P P P N
L6 College defined (optional) na na na na na
Development
D1 Course development is directed and supported P P L F P
D2 Resources are maintained and re-used P F P P P
D3 Courses meet the needs of all students P P N P P
D4 The e-learning infrastructure is reliable robust and sufficient P P N L P
D5 The e-learning infrastructure is standardised and integrated P P P P N
D6 College defined (optional) na na na na na
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 18 | Coralesce Ltd wwwcoralescecom
Infrastructure
Teaching and Learning
Organisation
Strategic Leadership
e-Learning coverage by department subject area
The questions covered the extent to which e-learning was used across levels type of
learning and to what extent it was used in Work based Learning An important aspect that
the questionnaire covered under the heading of strategic leadership was the financial
aspects including affordability of software implementation and cost benefit analysis
eQSF A Quality Standards Framework for ICT in Learning
Sector Learning and Skills
Country Northern Ireland
Year 2009 10
eQSF was developed using European funding and provides a comprehensive evaluation and
action planning package to assess practice in the use of existing and new technologies The
framework consists of three pillars each with a series of standards and examples of
evidence
Values Pedagogy and Practice (see figure 4)
Structure of the Learning Process
Leadership and Management
The pillars break down into 13 quality standards each with a separate on-line template to
complete Each standard is graded on a scale of attainment from Absent to Advanced
However the statements tend to describe processes rather than outcomes and are quite
broad particularly in the section on Teaching and Learning
The online resource facilitates the self-assessment process provides opportunities to
identify evidence and results in an action plan for improvement The implementation
process involves establishing leadership and ownership reviewing practice action
planning implementing action plan evaluating progress and feedback
Positive feedback is available from the three pilot case studies that were developed
including North West College in NI which indicate the tool has been used at teacher
department and organisational level
http wwweqsfeu
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 19 | Coralesce Ltd wwwcoralescecom
Figure 4 Values Pedagogy amp Practice Standards
Standard Addressed
Yes No
Actions
Completed Current Attainment Level Signature
11
The organisation supports inclusiveness and accessibility through the use of ICT and sees ICT as promoting equality and diversity in the learning process
Absent Acquiring Achieved Advanced
out of
12
The organisation facilitates the adaptation of learning materials to meet a range of learner requirements and sees ICT as a mechanism for delivering such flexibility for the practitioner and learner
Absent Acquiring Achieved Advanced
out of
13
The organisation uses ICT to extend learning beyond the classroom and supports progression to further learning and career opportunities
Absent Acquiring Achieved Advanced
out of
14
The organisation is committed to the learnersrsquo outcomes being positively impacted by ICT
Absent Acquiring Achieved Advanced
out of
Comment
Adult and Community Learning e-Learning Positioning Statement
Sector Adult and Community Learning
Country England
Date 2005-2007
The ACL eLPs were designed by CEL (Centre for Excellence in Leadership) and NIACE
(National Institute for Adult amp Continuing Education) as an aid to both institutional
strategic planning in e-learning and to support institutional quality improvement processes
As well as its use within NIACEs e-learning management and leadership courses the tool
was also embedded in individual consultancies and within the NIACE Digital Learning
teams capital investment funding work The ACL eLPs tool was superseded by BECTAs
Technology Improvement Leadership Tool - Generator NIACE formed part of the
development team for Generator and continued to contribute to its development and
piloting with community-based adult learning providers The use of the Generator tool was
integrated into NIACEs e-learning capital funding projects (CAMEL) The e-Learning
positioning statement comprised 31 elements grouped under five themes Organisations
graded themselves against a five point scale ranging from not yet started to embedded The
ACL eLearning Positioning Statement was designed as an aid to both institutional strategic
planning in e-learning and to support institutional quality improvement processes
The framework reviewed Vision and Strategic Planning Teaching and Learning Staff
Development Infrastructure and Managing ILT and e-Learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 10 | Coralesce Ltd wwwcoralescecom
Comparison of the themes used in Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 11 | Coralesce Ltd wwwcoralescecom
5 RESEARCH FINDINGS
A range of frameworks were identified as part of the research and these broadly fell into three categories
Organisational self-assessment
Individual self-assessment
Quality assurance frameworks for e-learning
The major focus of the Education and Training Foundationrsquos Learning Technology Self-Assessment Project is to develop an organisational tool with the potential to integrate individual practitioner self-assessment and therefore the summary below focuses on the first two types of framework Links to quality assurance framework for e-learning are provided in Appendix 1
METHODOLOGY
The methodology used in this Baseline Report was desk-based research to identify a range of self-assessment tools and quality frameworks covering schools further education and learning and skills higher education and industry Whilst the research provided a valuable and comprehensive overview of organisational and individual self-assessment tools and quality frameworks for e-learning it has not always been possible to access full question banks for all tools This is due to the fact that once an organisation responsible for a tool is closed not all archives are kept In the case of BECTA their self-assessment tool Generator was web-based and much of the material including the full question bank was not fully archived at their demise In addition the full set of questions used by Towards Maturity has been difficult to access due to the commercial nature of the organisation Where access has been impeded the research team have sourced a sample of questions through secondary literature
51 Organisational Self-Assessment Frameworks
Naace Self Review Framework (SRF) Sector Schools
Country England
Year 2014
A self-review framework designed to support schools to review their use of technology in a structured way and to support school improvement through reflective practice Developed against a well-researched and evidence based set of criteria Self-Review Framework (SRF) is a mature system originally developed by BECTA and now managed and updated annually by Naace
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 12 | Coralesce Ltd wwwcoralescecom
Naace is a membership organisation and provides an online tool which is accessible only to
member organisations (costs pound100 to join Naace and a further pound50 to access the online
tool) Members are well supported with training in the use of SRF and a useful website A
PDF version of the criteria is free to use
The SRF focuses on supporting pupils to become digitally literate and able to express
themselves and develop ideas through the use of ICT The tool is broad in its approach and
has less of an overall focus on teaching learning and assessment compared to the RAPTA
tool As this is a school based system it has a classroom based focus and does not reflect
the use of technology to support assessment monitoring and target setting
The framework links to the schools Self Evaluation Framework (SEF) and therefore
captures evidence for the annual self-evaluation and for Ofsted inspections The tool
focuses on ICT as well as Learning Technology but has a set of statements in the section on
developing digital literacy which could be adapted The framework consists of
Leadership and Management
Teaching and Learning
Professional Development
Resources
Each section is expanded into a number statements (54 in total) each of which is graded 1
to 4 against a clear set of grade descriptor (figure 1) Whilst the tool is long it is easy to
understand and designed to be used by different staff roles
The online tool provides clear guidance on each aspect and an opportunity to record
evidence and identify actions to improve
The online tool also provides a clear graphical representation of organisational capability
(figure 2) and although not currently benchmarked could be adapted to provide benchmark
data This process would lead to production of a detailed and specific plan for
improvement
The standards section links to an ICT Mark which recognises e-maturity The Mark is
externally assessed and verifies the judgements made in the SRF Following on from the
ICT Mark schools have an opportunity to apply for a Third Millennium Learning Award a
peer-referenced award which demonstrates that schools are providing education fit for the
21st century
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 13 | Coralesce Ltd wwwcoralescecom
Figure 1 Example Question from online SRF Tool
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 14 | Coralesce Ltd wwwcoralescecom
Figure 2 Graphical summary of outputs from online tool
Level Aiming For Enhancing Learning through ICT - level 3
1 Leadership and Management 1a-
1
1a-
2
1b-
1
1b-
2
1b-
3
1b-
4
1b-
5
1b-
6
1c-1 1c-2 1c-3 1c-4
3 4 X 4 3 4 3 3 X X X X
2 Use of ICT in the curriculum 2a-
1
2a-
2
2a-3 2a-4 2a-5 2a-6 2a-7 2b-
1
2b-
2
2b-
3
2b-
4
X X X X X X X X X X X
3 Teaching and Learning 3a-
1
3a-
2
3a-3 3a-4 3a-5 3b-
1
3b-
2
3b-
3
3b-
4
3b-
5
3b-
6
3b-
7
3b-
8
3b-
9
X X X X X X X X X X X X X X
4 Assessment of digital
capability
4a-
1
4a-
2
4a-3
X X X
5 Professional Development 5a-
1
5a-
2
5a-3 5a-4
X X X X
6 Resources 6a-
1
6a-
2
6a-3 6a-4 6a-5 6a-6 6b-
1
6b-
2
6b-
3
6b-
4
X X X X X X X X X X
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 15 | Coralesce Ltd wwwcoralescecom
Washington State Educational Technology Standards and Technology Essential
Conditions Rubric
Sector School
Country USA
Year 2008
A very well constructed model based on research on the benefits and dangers of technology
The standards put the learner at the heart of the system focusing on their expectations
Developed for schools as with the Naace system in England it is classroom focused The
Washington State materials consist of standards for learners and for teachers and are
supplemented with a Technology Essential Rubric for schools to evaluate progress at an
institutional level
The standards emanated from state legislature (1997) which established common learning
goals for all students in Washington State which provided a foundation for the
development of content called Essential Academic Learning Requirements (EALR)
Educational Technology was added to the EALR in 2007 and articulates high expectations
for what students should be able to do across all subject areas The standards for
educational technology were categorised as
Technology Integration
Digital Citizenship (technical literacy moving to technical fluency)
The standards are detailed albeit a little out of date with each EALR broken down into
components each with a grade level expectation for each year of education up to K12 (see
figure 3) Each grade level expectation has a descriptor of evidence of learning As the
standard requires use across subject areas it also provides examples across different subject
areas and levels including basic and 21st century learning environment The standards
provide detailed and useful materials to support teachers and curriculum areas to evaluate
and improve practice They could also be adapted for learner self-evaluation
The self-assessment tool the Technology Essential Conditions Rubric for Schools is
relatively short (8 pages) but would support effective self-assessment and has the potential
for benchmarking It uses a simple grading system beginning progressing proficient and
looks at five key areas
Vision planning and policy
Technical Literacy for learners ndash related to EALRs
Integration standards for teachers ndashrelated to EALRs
Technical competencies of administrators
Communication and Connections
Integration standards for teachers are described in three tiers Teacher Focus on
Productivity Instructional Presentation and Student Productivity Powerful Student-
Centred 21st century Learning Environment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 16 | Coralesce Ltd wwwcoralescecom
EALR 2 mdash DIGITAL CITIZENSHIP
Students demonstrate a clear understanding
of technology systems and operations and
practice safe legal and ethical behavior
Component 21 Practice Safety
Demonstrate safe legal and ethical behavior
in the use of information and technology
GLE Grade 6
212 Practice ethical and respectful behavior
Evidence
of
Learning
Comply with district Acceptable Use Policy (AUP)
Practice ethical and appropriate use of all media and comply with copyright law
Participate and engage with the global community within expected norms of behavior and positive interaction
Recognize report and
counteract cyberbullying
Examples
Recognize different types of illegal and unethical technology use
Identify and cite Creative Commons images in online presentations
Cite all sources properly
Observable indicators are provided for each tier together with helpful examples these could
be useful for practitioner level self-assessment Teachers are supported with extensive
materials available from the Washington State Office of the Superintendent of Public
Instruction wwwk12waus curriculuminstruct
Figure 3 An example of EALR for digital citizenship is shown below
Required
An Essential Academic Learning Requirement (EALR) is a broad statement of the learning that applies to Grades Kndash12 The Component is a statement that further defines and provides more specific information about the EALR There is at least one component for each EALR The Grade Level Expectation (GLE) is a statement containing the essential content or process to be learned and the cognitive demand required to learn it A revision of Bloomrsquos Taxonomy was used to categorize the cognitive demand required of the student Each GLE includes evidence of learning statements which are considered essential to the GLE
The GLE Numbering System identifies the EALR the component and the GLE in that order In the example at right the number 2 indicates the EALR the number 21 indicates the component and the number212 indicates the GLE Note Grade levels are not referenced in the numbering system The Evidence of Learning is a bulleted list of ways students can demonstrate learning considered essential to the GLE Educators are encouraged to identify additional ways in which the student can show proficiency as the educational technology GLEs are integrated across the curriculum Suggested The Examples provide specific illustrations of the learning However these examples are not exhaustive and educators are encouraged to find multiple ways by which learners can demonstrate what they know
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 17 | Coralesce Ltd wwwcoralescecom
Development Self Audit (DSA)
Sector Further Education
Country Scotland
Year 2007
Undertaken by Scottish Colleges as part of a baseline study of e-activity in 2007 the DSA
assessed e-maturity on a broad basis The development was funded by the Scottish Funding
Council and was informed by the practical activities and research of Dr Stephen Marshall
who developed an ideal model of e-maturity (eLearning Maturity Model)
The questions in the DSA cover five aspects
Learning (processes that impact on pedagogical aspects of e-learning)
Development (processes that surround the creation and maintenance of resources)
Support (processes surrounding oversight and management)
Evaluation (processes surrounding the evaluation and quality control)
Organisation (processes associated with planning and management)
As the DSA is based on the eMM developed in New Zealand it also maps the aspects against
the five dimensions of capability
Policy
Planning
Delivery
Management
Improvement
Using external assessment this process leads to the development of a grid which provides a
management level view of the e-maturity of a college an example is shown in figure 6
Figure 6 Management Level E-Maturity Grid
In addition to the completion of the grid the Scottish Funding Council also gathered
feedback on specific issues through the use of a questionnaire completed by institutions and
individuals which looked at the following themes
DSA
Po
licy
Pla
nn
ing
Delivery
Man
ag
em
en
t
Imp
rovem
en
t
Learning
L1 e-Learning is designed according to pedagogically sound principles N P L F P
L2 Learning skills are developed explicitly amp incrementally during delivery P F P P P
L3 Peer amp tutor interactions are provided explicitly during e-learning delivery P P N P P
L4 Courses take account of student workload and e-assessment demands P P N L P
L5 e-learning actively engage students P P P P N
L6 College defined (optional) na na na na na
Development
D1 Course development is directed and supported P P L F P
D2 Resources are maintained and re-used P F P P P
D3 Courses meet the needs of all students P P N P P
D4 The e-learning infrastructure is reliable robust and sufficient P P N L P
D5 The e-learning infrastructure is standardised and integrated P P P P N
D6 College defined (optional) na na na na na
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 18 | Coralesce Ltd wwwcoralescecom
Infrastructure
Teaching and Learning
Organisation
Strategic Leadership
e-Learning coverage by department subject area
The questions covered the extent to which e-learning was used across levels type of
learning and to what extent it was used in Work based Learning An important aspect that
the questionnaire covered under the heading of strategic leadership was the financial
aspects including affordability of software implementation and cost benefit analysis
eQSF A Quality Standards Framework for ICT in Learning
Sector Learning and Skills
Country Northern Ireland
Year 2009 10
eQSF was developed using European funding and provides a comprehensive evaluation and
action planning package to assess practice in the use of existing and new technologies The
framework consists of three pillars each with a series of standards and examples of
evidence
Values Pedagogy and Practice (see figure 4)
Structure of the Learning Process
Leadership and Management
The pillars break down into 13 quality standards each with a separate on-line template to
complete Each standard is graded on a scale of attainment from Absent to Advanced
However the statements tend to describe processes rather than outcomes and are quite
broad particularly in the section on Teaching and Learning
The online resource facilitates the self-assessment process provides opportunities to
identify evidence and results in an action plan for improvement The implementation
process involves establishing leadership and ownership reviewing practice action
planning implementing action plan evaluating progress and feedback
Positive feedback is available from the three pilot case studies that were developed
including North West College in NI which indicate the tool has been used at teacher
department and organisational level
http wwweqsfeu
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 19 | Coralesce Ltd wwwcoralescecom
Figure 4 Values Pedagogy amp Practice Standards
Standard Addressed
Yes No
Actions
Completed Current Attainment Level Signature
11
The organisation supports inclusiveness and accessibility through the use of ICT and sees ICT as promoting equality and diversity in the learning process
Absent Acquiring Achieved Advanced
out of
12
The organisation facilitates the adaptation of learning materials to meet a range of learner requirements and sees ICT as a mechanism for delivering such flexibility for the practitioner and learner
Absent Acquiring Achieved Advanced
out of
13
The organisation uses ICT to extend learning beyond the classroom and supports progression to further learning and career opportunities
Absent Acquiring Achieved Advanced
out of
14
The organisation is committed to the learnersrsquo outcomes being positively impacted by ICT
Absent Acquiring Achieved Advanced
out of
Comment
Adult and Community Learning e-Learning Positioning Statement
Sector Adult and Community Learning
Country England
Date 2005-2007
The ACL eLPs were designed by CEL (Centre for Excellence in Leadership) and NIACE
(National Institute for Adult amp Continuing Education) as an aid to both institutional
strategic planning in e-learning and to support institutional quality improvement processes
As well as its use within NIACEs e-learning management and leadership courses the tool
was also embedded in individual consultancies and within the NIACE Digital Learning
teams capital investment funding work The ACL eLPs tool was superseded by BECTAs
Technology Improvement Leadership Tool - Generator NIACE formed part of the
development team for Generator and continued to contribute to its development and
piloting with community-based adult learning providers The use of the Generator tool was
integrated into NIACEs e-learning capital funding projects (CAMEL) The e-Learning
positioning statement comprised 31 elements grouped under five themes Organisations
graded themselves against a five point scale ranging from not yet started to embedded The
ACL eLearning Positioning Statement was designed as an aid to both institutional strategic
planning in e-learning and to support institutional quality improvement processes
The framework reviewed Vision and Strategic Planning Teaching and Learning Staff
Development Infrastructure and Managing ILT and e-Learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 11 | Coralesce Ltd wwwcoralescecom
5 RESEARCH FINDINGS
A range of frameworks were identified as part of the research and these broadly fell into three categories
Organisational self-assessment
Individual self-assessment
Quality assurance frameworks for e-learning
The major focus of the Education and Training Foundationrsquos Learning Technology Self-Assessment Project is to develop an organisational tool with the potential to integrate individual practitioner self-assessment and therefore the summary below focuses on the first two types of framework Links to quality assurance framework for e-learning are provided in Appendix 1
METHODOLOGY
The methodology used in this Baseline Report was desk-based research to identify a range of self-assessment tools and quality frameworks covering schools further education and learning and skills higher education and industry Whilst the research provided a valuable and comprehensive overview of organisational and individual self-assessment tools and quality frameworks for e-learning it has not always been possible to access full question banks for all tools This is due to the fact that once an organisation responsible for a tool is closed not all archives are kept In the case of BECTA their self-assessment tool Generator was web-based and much of the material including the full question bank was not fully archived at their demise In addition the full set of questions used by Towards Maturity has been difficult to access due to the commercial nature of the organisation Where access has been impeded the research team have sourced a sample of questions through secondary literature
51 Organisational Self-Assessment Frameworks
Naace Self Review Framework (SRF) Sector Schools
Country England
Year 2014
A self-review framework designed to support schools to review their use of technology in a structured way and to support school improvement through reflective practice Developed against a well-researched and evidence based set of criteria Self-Review Framework (SRF) is a mature system originally developed by BECTA and now managed and updated annually by Naace
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 12 | Coralesce Ltd wwwcoralescecom
Naace is a membership organisation and provides an online tool which is accessible only to
member organisations (costs pound100 to join Naace and a further pound50 to access the online
tool) Members are well supported with training in the use of SRF and a useful website A
PDF version of the criteria is free to use
The SRF focuses on supporting pupils to become digitally literate and able to express
themselves and develop ideas through the use of ICT The tool is broad in its approach and
has less of an overall focus on teaching learning and assessment compared to the RAPTA
tool As this is a school based system it has a classroom based focus and does not reflect
the use of technology to support assessment monitoring and target setting
The framework links to the schools Self Evaluation Framework (SEF) and therefore
captures evidence for the annual self-evaluation and for Ofsted inspections The tool
focuses on ICT as well as Learning Technology but has a set of statements in the section on
developing digital literacy which could be adapted The framework consists of
Leadership and Management
Teaching and Learning
Professional Development
Resources
Each section is expanded into a number statements (54 in total) each of which is graded 1
to 4 against a clear set of grade descriptor (figure 1) Whilst the tool is long it is easy to
understand and designed to be used by different staff roles
The online tool provides clear guidance on each aspect and an opportunity to record
evidence and identify actions to improve
The online tool also provides a clear graphical representation of organisational capability
(figure 2) and although not currently benchmarked could be adapted to provide benchmark
data This process would lead to production of a detailed and specific plan for
improvement
The standards section links to an ICT Mark which recognises e-maturity The Mark is
externally assessed and verifies the judgements made in the SRF Following on from the
ICT Mark schools have an opportunity to apply for a Third Millennium Learning Award a
peer-referenced award which demonstrates that schools are providing education fit for the
21st century
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 13 | Coralesce Ltd wwwcoralescecom
Figure 1 Example Question from online SRF Tool
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 14 | Coralesce Ltd wwwcoralescecom
Figure 2 Graphical summary of outputs from online tool
Level Aiming For Enhancing Learning through ICT - level 3
1 Leadership and Management 1a-
1
1a-
2
1b-
1
1b-
2
1b-
3
1b-
4
1b-
5
1b-
6
1c-1 1c-2 1c-3 1c-4
3 4 X 4 3 4 3 3 X X X X
2 Use of ICT in the curriculum 2a-
1
2a-
2
2a-3 2a-4 2a-5 2a-6 2a-7 2b-
1
2b-
2
2b-
3
2b-
4
X X X X X X X X X X X
3 Teaching and Learning 3a-
1
3a-
2
3a-3 3a-4 3a-5 3b-
1
3b-
2
3b-
3
3b-
4
3b-
5
3b-
6
3b-
7
3b-
8
3b-
9
X X X X X X X X X X X X X X
4 Assessment of digital
capability
4a-
1
4a-
2
4a-3
X X X
5 Professional Development 5a-
1
5a-
2
5a-3 5a-4
X X X X
6 Resources 6a-
1
6a-
2
6a-3 6a-4 6a-5 6a-6 6b-
1
6b-
2
6b-
3
6b-
4
X X X X X X X X X X
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 15 | Coralesce Ltd wwwcoralescecom
Washington State Educational Technology Standards and Technology Essential
Conditions Rubric
Sector School
Country USA
Year 2008
A very well constructed model based on research on the benefits and dangers of technology
The standards put the learner at the heart of the system focusing on their expectations
Developed for schools as with the Naace system in England it is classroom focused The
Washington State materials consist of standards for learners and for teachers and are
supplemented with a Technology Essential Rubric for schools to evaluate progress at an
institutional level
The standards emanated from state legislature (1997) which established common learning
goals for all students in Washington State which provided a foundation for the
development of content called Essential Academic Learning Requirements (EALR)
Educational Technology was added to the EALR in 2007 and articulates high expectations
for what students should be able to do across all subject areas The standards for
educational technology were categorised as
Technology Integration
Digital Citizenship (technical literacy moving to technical fluency)
The standards are detailed albeit a little out of date with each EALR broken down into
components each with a grade level expectation for each year of education up to K12 (see
figure 3) Each grade level expectation has a descriptor of evidence of learning As the
standard requires use across subject areas it also provides examples across different subject
areas and levels including basic and 21st century learning environment The standards
provide detailed and useful materials to support teachers and curriculum areas to evaluate
and improve practice They could also be adapted for learner self-evaluation
The self-assessment tool the Technology Essential Conditions Rubric for Schools is
relatively short (8 pages) but would support effective self-assessment and has the potential
for benchmarking It uses a simple grading system beginning progressing proficient and
looks at five key areas
Vision planning and policy
Technical Literacy for learners ndash related to EALRs
Integration standards for teachers ndashrelated to EALRs
Technical competencies of administrators
Communication and Connections
Integration standards for teachers are described in three tiers Teacher Focus on
Productivity Instructional Presentation and Student Productivity Powerful Student-
Centred 21st century Learning Environment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 16 | Coralesce Ltd wwwcoralescecom
EALR 2 mdash DIGITAL CITIZENSHIP
Students demonstrate a clear understanding
of technology systems and operations and
practice safe legal and ethical behavior
Component 21 Practice Safety
Demonstrate safe legal and ethical behavior
in the use of information and technology
GLE Grade 6
212 Practice ethical and respectful behavior
Evidence
of
Learning
Comply with district Acceptable Use Policy (AUP)
Practice ethical and appropriate use of all media and comply with copyright law
Participate and engage with the global community within expected norms of behavior and positive interaction
Recognize report and
counteract cyberbullying
Examples
Recognize different types of illegal and unethical technology use
Identify and cite Creative Commons images in online presentations
Cite all sources properly
Observable indicators are provided for each tier together with helpful examples these could
be useful for practitioner level self-assessment Teachers are supported with extensive
materials available from the Washington State Office of the Superintendent of Public
Instruction wwwk12waus curriculuminstruct
Figure 3 An example of EALR for digital citizenship is shown below
Required
An Essential Academic Learning Requirement (EALR) is a broad statement of the learning that applies to Grades Kndash12 The Component is a statement that further defines and provides more specific information about the EALR There is at least one component for each EALR The Grade Level Expectation (GLE) is a statement containing the essential content or process to be learned and the cognitive demand required to learn it A revision of Bloomrsquos Taxonomy was used to categorize the cognitive demand required of the student Each GLE includes evidence of learning statements which are considered essential to the GLE
The GLE Numbering System identifies the EALR the component and the GLE in that order In the example at right the number 2 indicates the EALR the number 21 indicates the component and the number212 indicates the GLE Note Grade levels are not referenced in the numbering system The Evidence of Learning is a bulleted list of ways students can demonstrate learning considered essential to the GLE Educators are encouraged to identify additional ways in which the student can show proficiency as the educational technology GLEs are integrated across the curriculum Suggested The Examples provide specific illustrations of the learning However these examples are not exhaustive and educators are encouraged to find multiple ways by which learners can demonstrate what they know
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 17 | Coralesce Ltd wwwcoralescecom
Development Self Audit (DSA)
Sector Further Education
Country Scotland
Year 2007
Undertaken by Scottish Colleges as part of a baseline study of e-activity in 2007 the DSA
assessed e-maturity on a broad basis The development was funded by the Scottish Funding
Council and was informed by the practical activities and research of Dr Stephen Marshall
who developed an ideal model of e-maturity (eLearning Maturity Model)
The questions in the DSA cover five aspects
Learning (processes that impact on pedagogical aspects of e-learning)
Development (processes that surround the creation and maintenance of resources)
Support (processes surrounding oversight and management)
Evaluation (processes surrounding the evaluation and quality control)
Organisation (processes associated with planning and management)
As the DSA is based on the eMM developed in New Zealand it also maps the aspects against
the five dimensions of capability
Policy
Planning
Delivery
Management
Improvement
Using external assessment this process leads to the development of a grid which provides a
management level view of the e-maturity of a college an example is shown in figure 6
Figure 6 Management Level E-Maturity Grid
In addition to the completion of the grid the Scottish Funding Council also gathered
feedback on specific issues through the use of a questionnaire completed by institutions and
individuals which looked at the following themes
DSA
Po
licy
Pla
nn
ing
Delivery
Man
ag
em
en
t
Imp
rovem
en
t
Learning
L1 e-Learning is designed according to pedagogically sound principles N P L F P
L2 Learning skills are developed explicitly amp incrementally during delivery P F P P P
L3 Peer amp tutor interactions are provided explicitly during e-learning delivery P P N P P
L4 Courses take account of student workload and e-assessment demands P P N L P
L5 e-learning actively engage students P P P P N
L6 College defined (optional) na na na na na
Development
D1 Course development is directed and supported P P L F P
D2 Resources are maintained and re-used P F P P P
D3 Courses meet the needs of all students P P N P P
D4 The e-learning infrastructure is reliable robust and sufficient P P N L P
D5 The e-learning infrastructure is standardised and integrated P P P P N
D6 College defined (optional) na na na na na
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 18 | Coralesce Ltd wwwcoralescecom
Infrastructure
Teaching and Learning
Organisation
Strategic Leadership
e-Learning coverage by department subject area
The questions covered the extent to which e-learning was used across levels type of
learning and to what extent it was used in Work based Learning An important aspect that
the questionnaire covered under the heading of strategic leadership was the financial
aspects including affordability of software implementation and cost benefit analysis
eQSF A Quality Standards Framework for ICT in Learning
Sector Learning and Skills
Country Northern Ireland
Year 2009 10
eQSF was developed using European funding and provides a comprehensive evaluation and
action planning package to assess practice in the use of existing and new technologies The
framework consists of three pillars each with a series of standards and examples of
evidence
Values Pedagogy and Practice (see figure 4)
Structure of the Learning Process
Leadership and Management
The pillars break down into 13 quality standards each with a separate on-line template to
complete Each standard is graded on a scale of attainment from Absent to Advanced
However the statements tend to describe processes rather than outcomes and are quite
broad particularly in the section on Teaching and Learning
The online resource facilitates the self-assessment process provides opportunities to
identify evidence and results in an action plan for improvement The implementation
process involves establishing leadership and ownership reviewing practice action
planning implementing action plan evaluating progress and feedback
Positive feedback is available from the three pilot case studies that were developed
including North West College in NI which indicate the tool has been used at teacher
department and organisational level
http wwweqsfeu
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 19 | Coralesce Ltd wwwcoralescecom
Figure 4 Values Pedagogy amp Practice Standards
Standard Addressed
Yes No
Actions
Completed Current Attainment Level Signature
11
The organisation supports inclusiveness and accessibility through the use of ICT and sees ICT as promoting equality and diversity in the learning process
Absent Acquiring Achieved Advanced
out of
12
The organisation facilitates the adaptation of learning materials to meet a range of learner requirements and sees ICT as a mechanism for delivering such flexibility for the practitioner and learner
Absent Acquiring Achieved Advanced
out of
13
The organisation uses ICT to extend learning beyond the classroom and supports progression to further learning and career opportunities
Absent Acquiring Achieved Advanced
out of
14
The organisation is committed to the learnersrsquo outcomes being positively impacted by ICT
Absent Acquiring Achieved Advanced
out of
Comment
Adult and Community Learning e-Learning Positioning Statement
Sector Adult and Community Learning
Country England
Date 2005-2007
The ACL eLPs were designed by CEL (Centre for Excellence in Leadership) and NIACE
(National Institute for Adult amp Continuing Education) as an aid to both institutional
strategic planning in e-learning and to support institutional quality improvement processes
As well as its use within NIACEs e-learning management and leadership courses the tool
was also embedded in individual consultancies and within the NIACE Digital Learning
teams capital investment funding work The ACL eLPs tool was superseded by BECTAs
Technology Improvement Leadership Tool - Generator NIACE formed part of the
development team for Generator and continued to contribute to its development and
piloting with community-based adult learning providers The use of the Generator tool was
integrated into NIACEs e-learning capital funding projects (CAMEL) The e-Learning
positioning statement comprised 31 elements grouped under five themes Organisations
graded themselves against a five point scale ranging from not yet started to embedded The
ACL eLearning Positioning Statement was designed as an aid to both institutional strategic
planning in e-learning and to support institutional quality improvement processes
The framework reviewed Vision and Strategic Planning Teaching and Learning Staff
Development Infrastructure and Managing ILT and e-Learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 12 | Coralesce Ltd wwwcoralescecom
Naace is a membership organisation and provides an online tool which is accessible only to
member organisations (costs pound100 to join Naace and a further pound50 to access the online
tool) Members are well supported with training in the use of SRF and a useful website A
PDF version of the criteria is free to use
The SRF focuses on supporting pupils to become digitally literate and able to express
themselves and develop ideas through the use of ICT The tool is broad in its approach and
has less of an overall focus on teaching learning and assessment compared to the RAPTA
tool As this is a school based system it has a classroom based focus and does not reflect
the use of technology to support assessment monitoring and target setting
The framework links to the schools Self Evaluation Framework (SEF) and therefore
captures evidence for the annual self-evaluation and for Ofsted inspections The tool
focuses on ICT as well as Learning Technology but has a set of statements in the section on
developing digital literacy which could be adapted The framework consists of
Leadership and Management
Teaching and Learning
Professional Development
Resources
Each section is expanded into a number statements (54 in total) each of which is graded 1
to 4 against a clear set of grade descriptor (figure 1) Whilst the tool is long it is easy to
understand and designed to be used by different staff roles
The online tool provides clear guidance on each aspect and an opportunity to record
evidence and identify actions to improve
The online tool also provides a clear graphical representation of organisational capability
(figure 2) and although not currently benchmarked could be adapted to provide benchmark
data This process would lead to production of a detailed and specific plan for
improvement
The standards section links to an ICT Mark which recognises e-maturity The Mark is
externally assessed and verifies the judgements made in the SRF Following on from the
ICT Mark schools have an opportunity to apply for a Third Millennium Learning Award a
peer-referenced award which demonstrates that schools are providing education fit for the
21st century
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 13 | Coralesce Ltd wwwcoralescecom
Figure 1 Example Question from online SRF Tool
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 14 | Coralesce Ltd wwwcoralescecom
Figure 2 Graphical summary of outputs from online tool
Level Aiming For Enhancing Learning through ICT - level 3
1 Leadership and Management 1a-
1
1a-
2
1b-
1
1b-
2
1b-
3
1b-
4
1b-
5
1b-
6
1c-1 1c-2 1c-3 1c-4
3 4 X 4 3 4 3 3 X X X X
2 Use of ICT in the curriculum 2a-
1
2a-
2
2a-3 2a-4 2a-5 2a-6 2a-7 2b-
1
2b-
2
2b-
3
2b-
4
X X X X X X X X X X X
3 Teaching and Learning 3a-
1
3a-
2
3a-3 3a-4 3a-5 3b-
1
3b-
2
3b-
3
3b-
4
3b-
5
3b-
6
3b-
7
3b-
8
3b-
9
X X X X X X X X X X X X X X
4 Assessment of digital
capability
4a-
1
4a-
2
4a-3
X X X
5 Professional Development 5a-
1
5a-
2
5a-3 5a-4
X X X X
6 Resources 6a-
1
6a-
2
6a-3 6a-4 6a-5 6a-6 6b-
1
6b-
2
6b-
3
6b-
4
X X X X X X X X X X
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 15 | Coralesce Ltd wwwcoralescecom
Washington State Educational Technology Standards and Technology Essential
Conditions Rubric
Sector School
Country USA
Year 2008
A very well constructed model based on research on the benefits and dangers of technology
The standards put the learner at the heart of the system focusing on their expectations
Developed for schools as with the Naace system in England it is classroom focused The
Washington State materials consist of standards for learners and for teachers and are
supplemented with a Technology Essential Rubric for schools to evaluate progress at an
institutional level
The standards emanated from state legislature (1997) which established common learning
goals for all students in Washington State which provided a foundation for the
development of content called Essential Academic Learning Requirements (EALR)
Educational Technology was added to the EALR in 2007 and articulates high expectations
for what students should be able to do across all subject areas The standards for
educational technology were categorised as
Technology Integration
Digital Citizenship (technical literacy moving to technical fluency)
The standards are detailed albeit a little out of date with each EALR broken down into
components each with a grade level expectation for each year of education up to K12 (see
figure 3) Each grade level expectation has a descriptor of evidence of learning As the
standard requires use across subject areas it also provides examples across different subject
areas and levels including basic and 21st century learning environment The standards
provide detailed and useful materials to support teachers and curriculum areas to evaluate
and improve practice They could also be adapted for learner self-evaluation
The self-assessment tool the Technology Essential Conditions Rubric for Schools is
relatively short (8 pages) but would support effective self-assessment and has the potential
for benchmarking It uses a simple grading system beginning progressing proficient and
looks at five key areas
Vision planning and policy
Technical Literacy for learners ndash related to EALRs
Integration standards for teachers ndashrelated to EALRs
Technical competencies of administrators
Communication and Connections
Integration standards for teachers are described in three tiers Teacher Focus on
Productivity Instructional Presentation and Student Productivity Powerful Student-
Centred 21st century Learning Environment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 16 | Coralesce Ltd wwwcoralescecom
EALR 2 mdash DIGITAL CITIZENSHIP
Students demonstrate a clear understanding
of technology systems and operations and
practice safe legal and ethical behavior
Component 21 Practice Safety
Demonstrate safe legal and ethical behavior
in the use of information and technology
GLE Grade 6
212 Practice ethical and respectful behavior
Evidence
of
Learning
Comply with district Acceptable Use Policy (AUP)
Practice ethical and appropriate use of all media and comply with copyright law
Participate and engage with the global community within expected norms of behavior and positive interaction
Recognize report and
counteract cyberbullying
Examples
Recognize different types of illegal and unethical technology use
Identify and cite Creative Commons images in online presentations
Cite all sources properly
Observable indicators are provided for each tier together with helpful examples these could
be useful for practitioner level self-assessment Teachers are supported with extensive
materials available from the Washington State Office of the Superintendent of Public
Instruction wwwk12waus curriculuminstruct
Figure 3 An example of EALR for digital citizenship is shown below
Required
An Essential Academic Learning Requirement (EALR) is a broad statement of the learning that applies to Grades Kndash12 The Component is a statement that further defines and provides more specific information about the EALR There is at least one component for each EALR The Grade Level Expectation (GLE) is a statement containing the essential content or process to be learned and the cognitive demand required to learn it A revision of Bloomrsquos Taxonomy was used to categorize the cognitive demand required of the student Each GLE includes evidence of learning statements which are considered essential to the GLE
The GLE Numbering System identifies the EALR the component and the GLE in that order In the example at right the number 2 indicates the EALR the number 21 indicates the component and the number212 indicates the GLE Note Grade levels are not referenced in the numbering system The Evidence of Learning is a bulleted list of ways students can demonstrate learning considered essential to the GLE Educators are encouraged to identify additional ways in which the student can show proficiency as the educational technology GLEs are integrated across the curriculum Suggested The Examples provide specific illustrations of the learning However these examples are not exhaustive and educators are encouraged to find multiple ways by which learners can demonstrate what they know
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 17 | Coralesce Ltd wwwcoralescecom
Development Self Audit (DSA)
Sector Further Education
Country Scotland
Year 2007
Undertaken by Scottish Colleges as part of a baseline study of e-activity in 2007 the DSA
assessed e-maturity on a broad basis The development was funded by the Scottish Funding
Council and was informed by the practical activities and research of Dr Stephen Marshall
who developed an ideal model of e-maturity (eLearning Maturity Model)
The questions in the DSA cover five aspects
Learning (processes that impact on pedagogical aspects of e-learning)
Development (processes that surround the creation and maintenance of resources)
Support (processes surrounding oversight and management)
Evaluation (processes surrounding the evaluation and quality control)
Organisation (processes associated with planning and management)
As the DSA is based on the eMM developed in New Zealand it also maps the aspects against
the five dimensions of capability
Policy
Planning
Delivery
Management
Improvement
Using external assessment this process leads to the development of a grid which provides a
management level view of the e-maturity of a college an example is shown in figure 6
Figure 6 Management Level E-Maturity Grid
In addition to the completion of the grid the Scottish Funding Council also gathered
feedback on specific issues through the use of a questionnaire completed by institutions and
individuals which looked at the following themes
DSA
Po
licy
Pla
nn
ing
Delivery
Man
ag
em
en
t
Imp
rovem
en
t
Learning
L1 e-Learning is designed according to pedagogically sound principles N P L F P
L2 Learning skills are developed explicitly amp incrementally during delivery P F P P P
L3 Peer amp tutor interactions are provided explicitly during e-learning delivery P P N P P
L4 Courses take account of student workload and e-assessment demands P P N L P
L5 e-learning actively engage students P P P P N
L6 College defined (optional) na na na na na
Development
D1 Course development is directed and supported P P L F P
D2 Resources are maintained and re-used P F P P P
D3 Courses meet the needs of all students P P N P P
D4 The e-learning infrastructure is reliable robust and sufficient P P N L P
D5 The e-learning infrastructure is standardised and integrated P P P P N
D6 College defined (optional) na na na na na
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 18 | Coralesce Ltd wwwcoralescecom
Infrastructure
Teaching and Learning
Organisation
Strategic Leadership
e-Learning coverage by department subject area
The questions covered the extent to which e-learning was used across levels type of
learning and to what extent it was used in Work based Learning An important aspect that
the questionnaire covered under the heading of strategic leadership was the financial
aspects including affordability of software implementation and cost benefit analysis
eQSF A Quality Standards Framework for ICT in Learning
Sector Learning and Skills
Country Northern Ireland
Year 2009 10
eQSF was developed using European funding and provides a comprehensive evaluation and
action planning package to assess practice in the use of existing and new technologies The
framework consists of three pillars each with a series of standards and examples of
evidence
Values Pedagogy and Practice (see figure 4)
Structure of the Learning Process
Leadership and Management
The pillars break down into 13 quality standards each with a separate on-line template to
complete Each standard is graded on a scale of attainment from Absent to Advanced
However the statements tend to describe processes rather than outcomes and are quite
broad particularly in the section on Teaching and Learning
The online resource facilitates the self-assessment process provides opportunities to
identify evidence and results in an action plan for improvement The implementation
process involves establishing leadership and ownership reviewing practice action
planning implementing action plan evaluating progress and feedback
Positive feedback is available from the three pilot case studies that were developed
including North West College in NI which indicate the tool has been used at teacher
department and organisational level
http wwweqsfeu
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 19 | Coralesce Ltd wwwcoralescecom
Figure 4 Values Pedagogy amp Practice Standards
Standard Addressed
Yes No
Actions
Completed Current Attainment Level Signature
11
The organisation supports inclusiveness and accessibility through the use of ICT and sees ICT as promoting equality and diversity in the learning process
Absent Acquiring Achieved Advanced
out of
12
The organisation facilitates the adaptation of learning materials to meet a range of learner requirements and sees ICT as a mechanism for delivering such flexibility for the practitioner and learner
Absent Acquiring Achieved Advanced
out of
13
The organisation uses ICT to extend learning beyond the classroom and supports progression to further learning and career opportunities
Absent Acquiring Achieved Advanced
out of
14
The organisation is committed to the learnersrsquo outcomes being positively impacted by ICT
Absent Acquiring Achieved Advanced
out of
Comment
Adult and Community Learning e-Learning Positioning Statement
Sector Adult and Community Learning
Country England
Date 2005-2007
The ACL eLPs were designed by CEL (Centre for Excellence in Leadership) and NIACE
(National Institute for Adult amp Continuing Education) as an aid to both institutional
strategic planning in e-learning and to support institutional quality improvement processes
As well as its use within NIACEs e-learning management and leadership courses the tool
was also embedded in individual consultancies and within the NIACE Digital Learning
teams capital investment funding work The ACL eLPs tool was superseded by BECTAs
Technology Improvement Leadership Tool - Generator NIACE formed part of the
development team for Generator and continued to contribute to its development and
piloting with community-based adult learning providers The use of the Generator tool was
integrated into NIACEs e-learning capital funding projects (CAMEL) The e-Learning
positioning statement comprised 31 elements grouped under five themes Organisations
graded themselves against a five point scale ranging from not yet started to embedded The
ACL eLearning Positioning Statement was designed as an aid to both institutional strategic
planning in e-learning and to support institutional quality improvement processes
The framework reviewed Vision and Strategic Planning Teaching and Learning Staff
Development Infrastructure and Managing ILT and e-Learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 13 | Coralesce Ltd wwwcoralescecom
Figure 1 Example Question from online SRF Tool
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 14 | Coralesce Ltd wwwcoralescecom
Figure 2 Graphical summary of outputs from online tool
Level Aiming For Enhancing Learning through ICT - level 3
1 Leadership and Management 1a-
1
1a-
2
1b-
1
1b-
2
1b-
3
1b-
4
1b-
5
1b-
6
1c-1 1c-2 1c-3 1c-4
3 4 X 4 3 4 3 3 X X X X
2 Use of ICT in the curriculum 2a-
1
2a-
2
2a-3 2a-4 2a-5 2a-6 2a-7 2b-
1
2b-
2
2b-
3
2b-
4
X X X X X X X X X X X
3 Teaching and Learning 3a-
1
3a-
2
3a-3 3a-4 3a-5 3b-
1
3b-
2
3b-
3
3b-
4
3b-
5
3b-
6
3b-
7
3b-
8
3b-
9
X X X X X X X X X X X X X X
4 Assessment of digital
capability
4a-
1
4a-
2
4a-3
X X X
5 Professional Development 5a-
1
5a-
2
5a-3 5a-4
X X X X
6 Resources 6a-
1
6a-
2
6a-3 6a-4 6a-5 6a-6 6b-
1
6b-
2
6b-
3
6b-
4
X X X X X X X X X X
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 15 | Coralesce Ltd wwwcoralescecom
Washington State Educational Technology Standards and Technology Essential
Conditions Rubric
Sector School
Country USA
Year 2008
A very well constructed model based on research on the benefits and dangers of technology
The standards put the learner at the heart of the system focusing on their expectations
Developed for schools as with the Naace system in England it is classroom focused The
Washington State materials consist of standards for learners and for teachers and are
supplemented with a Technology Essential Rubric for schools to evaluate progress at an
institutional level
The standards emanated from state legislature (1997) which established common learning
goals for all students in Washington State which provided a foundation for the
development of content called Essential Academic Learning Requirements (EALR)
Educational Technology was added to the EALR in 2007 and articulates high expectations
for what students should be able to do across all subject areas The standards for
educational technology were categorised as
Technology Integration
Digital Citizenship (technical literacy moving to technical fluency)
The standards are detailed albeit a little out of date with each EALR broken down into
components each with a grade level expectation for each year of education up to K12 (see
figure 3) Each grade level expectation has a descriptor of evidence of learning As the
standard requires use across subject areas it also provides examples across different subject
areas and levels including basic and 21st century learning environment The standards
provide detailed and useful materials to support teachers and curriculum areas to evaluate
and improve practice They could also be adapted for learner self-evaluation
The self-assessment tool the Technology Essential Conditions Rubric for Schools is
relatively short (8 pages) but would support effective self-assessment and has the potential
for benchmarking It uses a simple grading system beginning progressing proficient and
looks at five key areas
Vision planning and policy
Technical Literacy for learners ndash related to EALRs
Integration standards for teachers ndashrelated to EALRs
Technical competencies of administrators
Communication and Connections
Integration standards for teachers are described in three tiers Teacher Focus on
Productivity Instructional Presentation and Student Productivity Powerful Student-
Centred 21st century Learning Environment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 16 | Coralesce Ltd wwwcoralescecom
EALR 2 mdash DIGITAL CITIZENSHIP
Students demonstrate a clear understanding
of technology systems and operations and
practice safe legal and ethical behavior
Component 21 Practice Safety
Demonstrate safe legal and ethical behavior
in the use of information and technology
GLE Grade 6
212 Practice ethical and respectful behavior
Evidence
of
Learning
Comply with district Acceptable Use Policy (AUP)
Practice ethical and appropriate use of all media and comply with copyright law
Participate and engage with the global community within expected norms of behavior and positive interaction
Recognize report and
counteract cyberbullying
Examples
Recognize different types of illegal and unethical technology use
Identify and cite Creative Commons images in online presentations
Cite all sources properly
Observable indicators are provided for each tier together with helpful examples these could
be useful for practitioner level self-assessment Teachers are supported with extensive
materials available from the Washington State Office of the Superintendent of Public
Instruction wwwk12waus curriculuminstruct
Figure 3 An example of EALR for digital citizenship is shown below
Required
An Essential Academic Learning Requirement (EALR) is a broad statement of the learning that applies to Grades Kndash12 The Component is a statement that further defines and provides more specific information about the EALR There is at least one component for each EALR The Grade Level Expectation (GLE) is a statement containing the essential content or process to be learned and the cognitive demand required to learn it A revision of Bloomrsquos Taxonomy was used to categorize the cognitive demand required of the student Each GLE includes evidence of learning statements which are considered essential to the GLE
The GLE Numbering System identifies the EALR the component and the GLE in that order In the example at right the number 2 indicates the EALR the number 21 indicates the component and the number212 indicates the GLE Note Grade levels are not referenced in the numbering system The Evidence of Learning is a bulleted list of ways students can demonstrate learning considered essential to the GLE Educators are encouraged to identify additional ways in which the student can show proficiency as the educational technology GLEs are integrated across the curriculum Suggested The Examples provide specific illustrations of the learning However these examples are not exhaustive and educators are encouraged to find multiple ways by which learners can demonstrate what they know
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 17 | Coralesce Ltd wwwcoralescecom
Development Self Audit (DSA)
Sector Further Education
Country Scotland
Year 2007
Undertaken by Scottish Colleges as part of a baseline study of e-activity in 2007 the DSA
assessed e-maturity on a broad basis The development was funded by the Scottish Funding
Council and was informed by the practical activities and research of Dr Stephen Marshall
who developed an ideal model of e-maturity (eLearning Maturity Model)
The questions in the DSA cover five aspects
Learning (processes that impact on pedagogical aspects of e-learning)
Development (processes that surround the creation and maintenance of resources)
Support (processes surrounding oversight and management)
Evaluation (processes surrounding the evaluation and quality control)
Organisation (processes associated with planning and management)
As the DSA is based on the eMM developed in New Zealand it also maps the aspects against
the five dimensions of capability
Policy
Planning
Delivery
Management
Improvement
Using external assessment this process leads to the development of a grid which provides a
management level view of the e-maturity of a college an example is shown in figure 6
Figure 6 Management Level E-Maturity Grid
In addition to the completion of the grid the Scottish Funding Council also gathered
feedback on specific issues through the use of a questionnaire completed by institutions and
individuals which looked at the following themes
DSA
Po
licy
Pla
nn
ing
Delivery
Man
ag
em
en
t
Imp
rovem
en
t
Learning
L1 e-Learning is designed according to pedagogically sound principles N P L F P
L2 Learning skills are developed explicitly amp incrementally during delivery P F P P P
L3 Peer amp tutor interactions are provided explicitly during e-learning delivery P P N P P
L4 Courses take account of student workload and e-assessment demands P P N L P
L5 e-learning actively engage students P P P P N
L6 College defined (optional) na na na na na
Development
D1 Course development is directed and supported P P L F P
D2 Resources are maintained and re-used P F P P P
D3 Courses meet the needs of all students P P N P P
D4 The e-learning infrastructure is reliable robust and sufficient P P N L P
D5 The e-learning infrastructure is standardised and integrated P P P P N
D6 College defined (optional) na na na na na
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 18 | Coralesce Ltd wwwcoralescecom
Infrastructure
Teaching and Learning
Organisation
Strategic Leadership
e-Learning coverage by department subject area
The questions covered the extent to which e-learning was used across levels type of
learning and to what extent it was used in Work based Learning An important aspect that
the questionnaire covered under the heading of strategic leadership was the financial
aspects including affordability of software implementation and cost benefit analysis
eQSF A Quality Standards Framework for ICT in Learning
Sector Learning and Skills
Country Northern Ireland
Year 2009 10
eQSF was developed using European funding and provides a comprehensive evaluation and
action planning package to assess practice in the use of existing and new technologies The
framework consists of three pillars each with a series of standards and examples of
evidence
Values Pedagogy and Practice (see figure 4)
Structure of the Learning Process
Leadership and Management
The pillars break down into 13 quality standards each with a separate on-line template to
complete Each standard is graded on a scale of attainment from Absent to Advanced
However the statements tend to describe processes rather than outcomes and are quite
broad particularly in the section on Teaching and Learning
The online resource facilitates the self-assessment process provides opportunities to
identify evidence and results in an action plan for improvement The implementation
process involves establishing leadership and ownership reviewing practice action
planning implementing action plan evaluating progress and feedback
Positive feedback is available from the three pilot case studies that were developed
including North West College in NI which indicate the tool has been used at teacher
department and organisational level
http wwweqsfeu
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 19 | Coralesce Ltd wwwcoralescecom
Figure 4 Values Pedagogy amp Practice Standards
Standard Addressed
Yes No
Actions
Completed Current Attainment Level Signature
11
The organisation supports inclusiveness and accessibility through the use of ICT and sees ICT as promoting equality and diversity in the learning process
Absent Acquiring Achieved Advanced
out of
12
The organisation facilitates the adaptation of learning materials to meet a range of learner requirements and sees ICT as a mechanism for delivering such flexibility for the practitioner and learner
Absent Acquiring Achieved Advanced
out of
13
The organisation uses ICT to extend learning beyond the classroom and supports progression to further learning and career opportunities
Absent Acquiring Achieved Advanced
out of
14
The organisation is committed to the learnersrsquo outcomes being positively impacted by ICT
Absent Acquiring Achieved Advanced
out of
Comment
Adult and Community Learning e-Learning Positioning Statement
Sector Adult and Community Learning
Country England
Date 2005-2007
The ACL eLPs were designed by CEL (Centre for Excellence in Leadership) and NIACE
(National Institute for Adult amp Continuing Education) as an aid to both institutional
strategic planning in e-learning and to support institutional quality improvement processes
As well as its use within NIACEs e-learning management and leadership courses the tool
was also embedded in individual consultancies and within the NIACE Digital Learning
teams capital investment funding work The ACL eLPs tool was superseded by BECTAs
Technology Improvement Leadership Tool - Generator NIACE formed part of the
development team for Generator and continued to contribute to its development and
piloting with community-based adult learning providers The use of the Generator tool was
integrated into NIACEs e-learning capital funding projects (CAMEL) The e-Learning
positioning statement comprised 31 elements grouped under five themes Organisations
graded themselves against a five point scale ranging from not yet started to embedded The
ACL eLearning Positioning Statement was designed as an aid to both institutional strategic
planning in e-learning and to support institutional quality improvement processes
The framework reviewed Vision and Strategic Planning Teaching and Learning Staff
Development Infrastructure and Managing ILT and e-Learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 14 | Coralesce Ltd wwwcoralescecom
Figure 2 Graphical summary of outputs from online tool
Level Aiming For Enhancing Learning through ICT - level 3
1 Leadership and Management 1a-
1
1a-
2
1b-
1
1b-
2
1b-
3
1b-
4
1b-
5
1b-
6
1c-1 1c-2 1c-3 1c-4
3 4 X 4 3 4 3 3 X X X X
2 Use of ICT in the curriculum 2a-
1
2a-
2
2a-3 2a-4 2a-5 2a-6 2a-7 2b-
1
2b-
2
2b-
3
2b-
4
X X X X X X X X X X X
3 Teaching and Learning 3a-
1
3a-
2
3a-3 3a-4 3a-5 3b-
1
3b-
2
3b-
3
3b-
4
3b-
5
3b-
6
3b-
7
3b-
8
3b-
9
X X X X X X X X X X X X X X
4 Assessment of digital
capability
4a-
1
4a-
2
4a-3
X X X
5 Professional Development 5a-
1
5a-
2
5a-3 5a-4
X X X X
6 Resources 6a-
1
6a-
2
6a-3 6a-4 6a-5 6a-6 6b-
1
6b-
2
6b-
3
6b-
4
X X X X X X X X X X
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 15 | Coralesce Ltd wwwcoralescecom
Washington State Educational Technology Standards and Technology Essential
Conditions Rubric
Sector School
Country USA
Year 2008
A very well constructed model based on research on the benefits and dangers of technology
The standards put the learner at the heart of the system focusing on their expectations
Developed for schools as with the Naace system in England it is classroom focused The
Washington State materials consist of standards for learners and for teachers and are
supplemented with a Technology Essential Rubric for schools to evaluate progress at an
institutional level
The standards emanated from state legislature (1997) which established common learning
goals for all students in Washington State which provided a foundation for the
development of content called Essential Academic Learning Requirements (EALR)
Educational Technology was added to the EALR in 2007 and articulates high expectations
for what students should be able to do across all subject areas The standards for
educational technology were categorised as
Technology Integration
Digital Citizenship (technical literacy moving to technical fluency)
The standards are detailed albeit a little out of date with each EALR broken down into
components each with a grade level expectation for each year of education up to K12 (see
figure 3) Each grade level expectation has a descriptor of evidence of learning As the
standard requires use across subject areas it also provides examples across different subject
areas and levels including basic and 21st century learning environment The standards
provide detailed and useful materials to support teachers and curriculum areas to evaluate
and improve practice They could also be adapted for learner self-evaluation
The self-assessment tool the Technology Essential Conditions Rubric for Schools is
relatively short (8 pages) but would support effective self-assessment and has the potential
for benchmarking It uses a simple grading system beginning progressing proficient and
looks at five key areas
Vision planning and policy
Technical Literacy for learners ndash related to EALRs
Integration standards for teachers ndashrelated to EALRs
Technical competencies of administrators
Communication and Connections
Integration standards for teachers are described in three tiers Teacher Focus on
Productivity Instructional Presentation and Student Productivity Powerful Student-
Centred 21st century Learning Environment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 16 | Coralesce Ltd wwwcoralescecom
EALR 2 mdash DIGITAL CITIZENSHIP
Students demonstrate a clear understanding
of technology systems and operations and
practice safe legal and ethical behavior
Component 21 Practice Safety
Demonstrate safe legal and ethical behavior
in the use of information and technology
GLE Grade 6
212 Practice ethical and respectful behavior
Evidence
of
Learning
Comply with district Acceptable Use Policy (AUP)
Practice ethical and appropriate use of all media and comply with copyright law
Participate and engage with the global community within expected norms of behavior and positive interaction
Recognize report and
counteract cyberbullying
Examples
Recognize different types of illegal and unethical technology use
Identify and cite Creative Commons images in online presentations
Cite all sources properly
Observable indicators are provided for each tier together with helpful examples these could
be useful for practitioner level self-assessment Teachers are supported with extensive
materials available from the Washington State Office of the Superintendent of Public
Instruction wwwk12waus curriculuminstruct
Figure 3 An example of EALR for digital citizenship is shown below
Required
An Essential Academic Learning Requirement (EALR) is a broad statement of the learning that applies to Grades Kndash12 The Component is a statement that further defines and provides more specific information about the EALR There is at least one component for each EALR The Grade Level Expectation (GLE) is a statement containing the essential content or process to be learned and the cognitive demand required to learn it A revision of Bloomrsquos Taxonomy was used to categorize the cognitive demand required of the student Each GLE includes evidence of learning statements which are considered essential to the GLE
The GLE Numbering System identifies the EALR the component and the GLE in that order In the example at right the number 2 indicates the EALR the number 21 indicates the component and the number212 indicates the GLE Note Grade levels are not referenced in the numbering system The Evidence of Learning is a bulleted list of ways students can demonstrate learning considered essential to the GLE Educators are encouraged to identify additional ways in which the student can show proficiency as the educational technology GLEs are integrated across the curriculum Suggested The Examples provide specific illustrations of the learning However these examples are not exhaustive and educators are encouraged to find multiple ways by which learners can demonstrate what they know
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 17 | Coralesce Ltd wwwcoralescecom
Development Self Audit (DSA)
Sector Further Education
Country Scotland
Year 2007
Undertaken by Scottish Colleges as part of a baseline study of e-activity in 2007 the DSA
assessed e-maturity on a broad basis The development was funded by the Scottish Funding
Council and was informed by the practical activities and research of Dr Stephen Marshall
who developed an ideal model of e-maturity (eLearning Maturity Model)
The questions in the DSA cover five aspects
Learning (processes that impact on pedagogical aspects of e-learning)
Development (processes that surround the creation and maintenance of resources)
Support (processes surrounding oversight and management)
Evaluation (processes surrounding the evaluation and quality control)
Organisation (processes associated with planning and management)
As the DSA is based on the eMM developed in New Zealand it also maps the aspects against
the five dimensions of capability
Policy
Planning
Delivery
Management
Improvement
Using external assessment this process leads to the development of a grid which provides a
management level view of the e-maturity of a college an example is shown in figure 6
Figure 6 Management Level E-Maturity Grid
In addition to the completion of the grid the Scottish Funding Council also gathered
feedback on specific issues through the use of a questionnaire completed by institutions and
individuals which looked at the following themes
DSA
Po
licy
Pla
nn
ing
Delivery
Man
ag
em
en
t
Imp
rovem
en
t
Learning
L1 e-Learning is designed according to pedagogically sound principles N P L F P
L2 Learning skills are developed explicitly amp incrementally during delivery P F P P P
L3 Peer amp tutor interactions are provided explicitly during e-learning delivery P P N P P
L4 Courses take account of student workload and e-assessment demands P P N L P
L5 e-learning actively engage students P P P P N
L6 College defined (optional) na na na na na
Development
D1 Course development is directed and supported P P L F P
D2 Resources are maintained and re-used P F P P P
D3 Courses meet the needs of all students P P N P P
D4 The e-learning infrastructure is reliable robust and sufficient P P N L P
D5 The e-learning infrastructure is standardised and integrated P P P P N
D6 College defined (optional) na na na na na
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 18 | Coralesce Ltd wwwcoralescecom
Infrastructure
Teaching and Learning
Organisation
Strategic Leadership
e-Learning coverage by department subject area
The questions covered the extent to which e-learning was used across levels type of
learning and to what extent it was used in Work based Learning An important aspect that
the questionnaire covered under the heading of strategic leadership was the financial
aspects including affordability of software implementation and cost benefit analysis
eQSF A Quality Standards Framework for ICT in Learning
Sector Learning and Skills
Country Northern Ireland
Year 2009 10
eQSF was developed using European funding and provides a comprehensive evaluation and
action planning package to assess practice in the use of existing and new technologies The
framework consists of three pillars each with a series of standards and examples of
evidence
Values Pedagogy and Practice (see figure 4)
Structure of the Learning Process
Leadership and Management
The pillars break down into 13 quality standards each with a separate on-line template to
complete Each standard is graded on a scale of attainment from Absent to Advanced
However the statements tend to describe processes rather than outcomes and are quite
broad particularly in the section on Teaching and Learning
The online resource facilitates the self-assessment process provides opportunities to
identify evidence and results in an action plan for improvement The implementation
process involves establishing leadership and ownership reviewing practice action
planning implementing action plan evaluating progress and feedback
Positive feedback is available from the three pilot case studies that were developed
including North West College in NI which indicate the tool has been used at teacher
department and organisational level
http wwweqsfeu
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 19 | Coralesce Ltd wwwcoralescecom
Figure 4 Values Pedagogy amp Practice Standards
Standard Addressed
Yes No
Actions
Completed Current Attainment Level Signature
11
The organisation supports inclusiveness and accessibility through the use of ICT and sees ICT as promoting equality and diversity in the learning process
Absent Acquiring Achieved Advanced
out of
12
The organisation facilitates the adaptation of learning materials to meet a range of learner requirements and sees ICT as a mechanism for delivering such flexibility for the practitioner and learner
Absent Acquiring Achieved Advanced
out of
13
The organisation uses ICT to extend learning beyond the classroom and supports progression to further learning and career opportunities
Absent Acquiring Achieved Advanced
out of
14
The organisation is committed to the learnersrsquo outcomes being positively impacted by ICT
Absent Acquiring Achieved Advanced
out of
Comment
Adult and Community Learning e-Learning Positioning Statement
Sector Adult and Community Learning
Country England
Date 2005-2007
The ACL eLPs were designed by CEL (Centre for Excellence in Leadership) and NIACE
(National Institute for Adult amp Continuing Education) as an aid to both institutional
strategic planning in e-learning and to support institutional quality improvement processes
As well as its use within NIACEs e-learning management and leadership courses the tool
was also embedded in individual consultancies and within the NIACE Digital Learning
teams capital investment funding work The ACL eLPs tool was superseded by BECTAs
Technology Improvement Leadership Tool - Generator NIACE formed part of the
development team for Generator and continued to contribute to its development and
piloting with community-based adult learning providers The use of the Generator tool was
integrated into NIACEs e-learning capital funding projects (CAMEL) The e-Learning
positioning statement comprised 31 elements grouped under five themes Organisations
graded themselves against a five point scale ranging from not yet started to embedded The
ACL eLearning Positioning Statement was designed as an aid to both institutional strategic
planning in e-learning and to support institutional quality improvement processes
The framework reviewed Vision and Strategic Planning Teaching and Learning Staff
Development Infrastructure and Managing ILT and e-Learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 15 | Coralesce Ltd wwwcoralescecom
Washington State Educational Technology Standards and Technology Essential
Conditions Rubric
Sector School
Country USA
Year 2008
A very well constructed model based on research on the benefits and dangers of technology
The standards put the learner at the heart of the system focusing on their expectations
Developed for schools as with the Naace system in England it is classroom focused The
Washington State materials consist of standards for learners and for teachers and are
supplemented with a Technology Essential Rubric for schools to evaluate progress at an
institutional level
The standards emanated from state legislature (1997) which established common learning
goals for all students in Washington State which provided a foundation for the
development of content called Essential Academic Learning Requirements (EALR)
Educational Technology was added to the EALR in 2007 and articulates high expectations
for what students should be able to do across all subject areas The standards for
educational technology were categorised as
Technology Integration
Digital Citizenship (technical literacy moving to technical fluency)
The standards are detailed albeit a little out of date with each EALR broken down into
components each with a grade level expectation for each year of education up to K12 (see
figure 3) Each grade level expectation has a descriptor of evidence of learning As the
standard requires use across subject areas it also provides examples across different subject
areas and levels including basic and 21st century learning environment The standards
provide detailed and useful materials to support teachers and curriculum areas to evaluate
and improve practice They could also be adapted for learner self-evaluation
The self-assessment tool the Technology Essential Conditions Rubric for Schools is
relatively short (8 pages) but would support effective self-assessment and has the potential
for benchmarking It uses a simple grading system beginning progressing proficient and
looks at five key areas
Vision planning and policy
Technical Literacy for learners ndash related to EALRs
Integration standards for teachers ndashrelated to EALRs
Technical competencies of administrators
Communication and Connections
Integration standards for teachers are described in three tiers Teacher Focus on
Productivity Instructional Presentation and Student Productivity Powerful Student-
Centred 21st century Learning Environment
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 16 | Coralesce Ltd wwwcoralescecom
EALR 2 mdash DIGITAL CITIZENSHIP
Students demonstrate a clear understanding
of technology systems and operations and
practice safe legal and ethical behavior
Component 21 Practice Safety
Demonstrate safe legal and ethical behavior
in the use of information and technology
GLE Grade 6
212 Practice ethical and respectful behavior
Evidence
of
Learning
Comply with district Acceptable Use Policy (AUP)
Practice ethical and appropriate use of all media and comply with copyright law
Participate and engage with the global community within expected norms of behavior and positive interaction
Recognize report and
counteract cyberbullying
Examples
Recognize different types of illegal and unethical technology use
Identify and cite Creative Commons images in online presentations
Cite all sources properly
Observable indicators are provided for each tier together with helpful examples these could
be useful for practitioner level self-assessment Teachers are supported with extensive
materials available from the Washington State Office of the Superintendent of Public
Instruction wwwk12waus curriculuminstruct
Figure 3 An example of EALR for digital citizenship is shown below
Required
An Essential Academic Learning Requirement (EALR) is a broad statement of the learning that applies to Grades Kndash12 The Component is a statement that further defines and provides more specific information about the EALR There is at least one component for each EALR The Grade Level Expectation (GLE) is a statement containing the essential content or process to be learned and the cognitive demand required to learn it A revision of Bloomrsquos Taxonomy was used to categorize the cognitive demand required of the student Each GLE includes evidence of learning statements which are considered essential to the GLE
The GLE Numbering System identifies the EALR the component and the GLE in that order In the example at right the number 2 indicates the EALR the number 21 indicates the component and the number212 indicates the GLE Note Grade levels are not referenced in the numbering system The Evidence of Learning is a bulleted list of ways students can demonstrate learning considered essential to the GLE Educators are encouraged to identify additional ways in which the student can show proficiency as the educational technology GLEs are integrated across the curriculum Suggested The Examples provide specific illustrations of the learning However these examples are not exhaustive and educators are encouraged to find multiple ways by which learners can demonstrate what they know
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 17 | Coralesce Ltd wwwcoralescecom
Development Self Audit (DSA)
Sector Further Education
Country Scotland
Year 2007
Undertaken by Scottish Colleges as part of a baseline study of e-activity in 2007 the DSA
assessed e-maturity on a broad basis The development was funded by the Scottish Funding
Council and was informed by the practical activities and research of Dr Stephen Marshall
who developed an ideal model of e-maturity (eLearning Maturity Model)
The questions in the DSA cover five aspects
Learning (processes that impact on pedagogical aspects of e-learning)
Development (processes that surround the creation and maintenance of resources)
Support (processes surrounding oversight and management)
Evaluation (processes surrounding the evaluation and quality control)
Organisation (processes associated with planning and management)
As the DSA is based on the eMM developed in New Zealand it also maps the aspects against
the five dimensions of capability
Policy
Planning
Delivery
Management
Improvement
Using external assessment this process leads to the development of a grid which provides a
management level view of the e-maturity of a college an example is shown in figure 6
Figure 6 Management Level E-Maturity Grid
In addition to the completion of the grid the Scottish Funding Council also gathered
feedback on specific issues through the use of a questionnaire completed by institutions and
individuals which looked at the following themes
DSA
Po
licy
Pla
nn
ing
Delivery
Man
ag
em
en
t
Imp
rovem
en
t
Learning
L1 e-Learning is designed according to pedagogically sound principles N P L F P
L2 Learning skills are developed explicitly amp incrementally during delivery P F P P P
L3 Peer amp tutor interactions are provided explicitly during e-learning delivery P P N P P
L4 Courses take account of student workload and e-assessment demands P P N L P
L5 e-learning actively engage students P P P P N
L6 College defined (optional) na na na na na
Development
D1 Course development is directed and supported P P L F P
D2 Resources are maintained and re-used P F P P P
D3 Courses meet the needs of all students P P N P P
D4 The e-learning infrastructure is reliable robust and sufficient P P N L P
D5 The e-learning infrastructure is standardised and integrated P P P P N
D6 College defined (optional) na na na na na
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 18 | Coralesce Ltd wwwcoralescecom
Infrastructure
Teaching and Learning
Organisation
Strategic Leadership
e-Learning coverage by department subject area
The questions covered the extent to which e-learning was used across levels type of
learning and to what extent it was used in Work based Learning An important aspect that
the questionnaire covered under the heading of strategic leadership was the financial
aspects including affordability of software implementation and cost benefit analysis
eQSF A Quality Standards Framework for ICT in Learning
Sector Learning and Skills
Country Northern Ireland
Year 2009 10
eQSF was developed using European funding and provides a comprehensive evaluation and
action planning package to assess practice in the use of existing and new technologies The
framework consists of three pillars each with a series of standards and examples of
evidence
Values Pedagogy and Practice (see figure 4)
Structure of the Learning Process
Leadership and Management
The pillars break down into 13 quality standards each with a separate on-line template to
complete Each standard is graded on a scale of attainment from Absent to Advanced
However the statements tend to describe processes rather than outcomes and are quite
broad particularly in the section on Teaching and Learning
The online resource facilitates the self-assessment process provides opportunities to
identify evidence and results in an action plan for improvement The implementation
process involves establishing leadership and ownership reviewing practice action
planning implementing action plan evaluating progress and feedback
Positive feedback is available from the three pilot case studies that were developed
including North West College in NI which indicate the tool has been used at teacher
department and organisational level
http wwweqsfeu
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 19 | Coralesce Ltd wwwcoralescecom
Figure 4 Values Pedagogy amp Practice Standards
Standard Addressed
Yes No
Actions
Completed Current Attainment Level Signature
11
The organisation supports inclusiveness and accessibility through the use of ICT and sees ICT as promoting equality and diversity in the learning process
Absent Acquiring Achieved Advanced
out of
12
The organisation facilitates the adaptation of learning materials to meet a range of learner requirements and sees ICT as a mechanism for delivering such flexibility for the practitioner and learner
Absent Acquiring Achieved Advanced
out of
13
The organisation uses ICT to extend learning beyond the classroom and supports progression to further learning and career opportunities
Absent Acquiring Achieved Advanced
out of
14
The organisation is committed to the learnersrsquo outcomes being positively impacted by ICT
Absent Acquiring Achieved Advanced
out of
Comment
Adult and Community Learning e-Learning Positioning Statement
Sector Adult and Community Learning
Country England
Date 2005-2007
The ACL eLPs were designed by CEL (Centre for Excellence in Leadership) and NIACE
(National Institute for Adult amp Continuing Education) as an aid to both institutional
strategic planning in e-learning and to support institutional quality improvement processes
As well as its use within NIACEs e-learning management and leadership courses the tool
was also embedded in individual consultancies and within the NIACE Digital Learning
teams capital investment funding work The ACL eLPs tool was superseded by BECTAs
Technology Improvement Leadership Tool - Generator NIACE formed part of the
development team for Generator and continued to contribute to its development and
piloting with community-based adult learning providers The use of the Generator tool was
integrated into NIACEs e-learning capital funding projects (CAMEL) The e-Learning
positioning statement comprised 31 elements grouped under five themes Organisations
graded themselves against a five point scale ranging from not yet started to embedded The
ACL eLearning Positioning Statement was designed as an aid to both institutional strategic
planning in e-learning and to support institutional quality improvement processes
The framework reviewed Vision and Strategic Planning Teaching and Learning Staff
Development Infrastructure and Managing ILT and e-Learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 16 | Coralesce Ltd wwwcoralescecom
EALR 2 mdash DIGITAL CITIZENSHIP
Students demonstrate a clear understanding
of technology systems and operations and
practice safe legal and ethical behavior
Component 21 Practice Safety
Demonstrate safe legal and ethical behavior
in the use of information and technology
GLE Grade 6
212 Practice ethical and respectful behavior
Evidence
of
Learning
Comply with district Acceptable Use Policy (AUP)
Practice ethical and appropriate use of all media and comply with copyright law
Participate and engage with the global community within expected norms of behavior and positive interaction
Recognize report and
counteract cyberbullying
Examples
Recognize different types of illegal and unethical technology use
Identify and cite Creative Commons images in online presentations
Cite all sources properly
Observable indicators are provided for each tier together with helpful examples these could
be useful for practitioner level self-assessment Teachers are supported with extensive
materials available from the Washington State Office of the Superintendent of Public
Instruction wwwk12waus curriculuminstruct
Figure 3 An example of EALR for digital citizenship is shown below
Required
An Essential Academic Learning Requirement (EALR) is a broad statement of the learning that applies to Grades Kndash12 The Component is a statement that further defines and provides more specific information about the EALR There is at least one component for each EALR The Grade Level Expectation (GLE) is a statement containing the essential content or process to be learned and the cognitive demand required to learn it A revision of Bloomrsquos Taxonomy was used to categorize the cognitive demand required of the student Each GLE includes evidence of learning statements which are considered essential to the GLE
The GLE Numbering System identifies the EALR the component and the GLE in that order In the example at right the number 2 indicates the EALR the number 21 indicates the component and the number212 indicates the GLE Note Grade levels are not referenced in the numbering system The Evidence of Learning is a bulleted list of ways students can demonstrate learning considered essential to the GLE Educators are encouraged to identify additional ways in which the student can show proficiency as the educational technology GLEs are integrated across the curriculum Suggested The Examples provide specific illustrations of the learning However these examples are not exhaustive and educators are encouraged to find multiple ways by which learners can demonstrate what they know
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 17 | Coralesce Ltd wwwcoralescecom
Development Self Audit (DSA)
Sector Further Education
Country Scotland
Year 2007
Undertaken by Scottish Colleges as part of a baseline study of e-activity in 2007 the DSA
assessed e-maturity on a broad basis The development was funded by the Scottish Funding
Council and was informed by the practical activities and research of Dr Stephen Marshall
who developed an ideal model of e-maturity (eLearning Maturity Model)
The questions in the DSA cover five aspects
Learning (processes that impact on pedagogical aspects of e-learning)
Development (processes that surround the creation and maintenance of resources)
Support (processes surrounding oversight and management)
Evaluation (processes surrounding the evaluation and quality control)
Organisation (processes associated with planning and management)
As the DSA is based on the eMM developed in New Zealand it also maps the aspects against
the five dimensions of capability
Policy
Planning
Delivery
Management
Improvement
Using external assessment this process leads to the development of a grid which provides a
management level view of the e-maturity of a college an example is shown in figure 6
Figure 6 Management Level E-Maturity Grid
In addition to the completion of the grid the Scottish Funding Council also gathered
feedback on specific issues through the use of a questionnaire completed by institutions and
individuals which looked at the following themes
DSA
Po
licy
Pla
nn
ing
Delivery
Man
ag
em
en
t
Imp
rovem
en
t
Learning
L1 e-Learning is designed according to pedagogically sound principles N P L F P
L2 Learning skills are developed explicitly amp incrementally during delivery P F P P P
L3 Peer amp tutor interactions are provided explicitly during e-learning delivery P P N P P
L4 Courses take account of student workload and e-assessment demands P P N L P
L5 e-learning actively engage students P P P P N
L6 College defined (optional) na na na na na
Development
D1 Course development is directed and supported P P L F P
D2 Resources are maintained and re-used P F P P P
D3 Courses meet the needs of all students P P N P P
D4 The e-learning infrastructure is reliable robust and sufficient P P N L P
D5 The e-learning infrastructure is standardised and integrated P P P P N
D6 College defined (optional) na na na na na
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 18 | Coralesce Ltd wwwcoralescecom
Infrastructure
Teaching and Learning
Organisation
Strategic Leadership
e-Learning coverage by department subject area
The questions covered the extent to which e-learning was used across levels type of
learning and to what extent it was used in Work based Learning An important aspect that
the questionnaire covered under the heading of strategic leadership was the financial
aspects including affordability of software implementation and cost benefit analysis
eQSF A Quality Standards Framework for ICT in Learning
Sector Learning and Skills
Country Northern Ireland
Year 2009 10
eQSF was developed using European funding and provides a comprehensive evaluation and
action planning package to assess practice in the use of existing and new technologies The
framework consists of three pillars each with a series of standards and examples of
evidence
Values Pedagogy and Practice (see figure 4)
Structure of the Learning Process
Leadership and Management
The pillars break down into 13 quality standards each with a separate on-line template to
complete Each standard is graded on a scale of attainment from Absent to Advanced
However the statements tend to describe processes rather than outcomes and are quite
broad particularly in the section on Teaching and Learning
The online resource facilitates the self-assessment process provides opportunities to
identify evidence and results in an action plan for improvement The implementation
process involves establishing leadership and ownership reviewing practice action
planning implementing action plan evaluating progress and feedback
Positive feedback is available from the three pilot case studies that were developed
including North West College in NI which indicate the tool has been used at teacher
department and organisational level
http wwweqsfeu
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 19 | Coralesce Ltd wwwcoralescecom
Figure 4 Values Pedagogy amp Practice Standards
Standard Addressed
Yes No
Actions
Completed Current Attainment Level Signature
11
The organisation supports inclusiveness and accessibility through the use of ICT and sees ICT as promoting equality and diversity in the learning process
Absent Acquiring Achieved Advanced
out of
12
The organisation facilitates the adaptation of learning materials to meet a range of learner requirements and sees ICT as a mechanism for delivering such flexibility for the practitioner and learner
Absent Acquiring Achieved Advanced
out of
13
The organisation uses ICT to extend learning beyond the classroom and supports progression to further learning and career opportunities
Absent Acquiring Achieved Advanced
out of
14
The organisation is committed to the learnersrsquo outcomes being positively impacted by ICT
Absent Acquiring Achieved Advanced
out of
Comment
Adult and Community Learning e-Learning Positioning Statement
Sector Adult and Community Learning
Country England
Date 2005-2007
The ACL eLPs were designed by CEL (Centre for Excellence in Leadership) and NIACE
(National Institute for Adult amp Continuing Education) as an aid to both institutional
strategic planning in e-learning and to support institutional quality improvement processes
As well as its use within NIACEs e-learning management and leadership courses the tool
was also embedded in individual consultancies and within the NIACE Digital Learning
teams capital investment funding work The ACL eLPs tool was superseded by BECTAs
Technology Improvement Leadership Tool - Generator NIACE formed part of the
development team for Generator and continued to contribute to its development and
piloting with community-based adult learning providers The use of the Generator tool was
integrated into NIACEs e-learning capital funding projects (CAMEL) The e-Learning
positioning statement comprised 31 elements grouped under five themes Organisations
graded themselves against a five point scale ranging from not yet started to embedded The
ACL eLearning Positioning Statement was designed as an aid to both institutional strategic
planning in e-learning and to support institutional quality improvement processes
The framework reviewed Vision and Strategic Planning Teaching and Learning Staff
Development Infrastructure and Managing ILT and e-Learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 17 | Coralesce Ltd wwwcoralescecom
Development Self Audit (DSA)
Sector Further Education
Country Scotland
Year 2007
Undertaken by Scottish Colleges as part of a baseline study of e-activity in 2007 the DSA
assessed e-maturity on a broad basis The development was funded by the Scottish Funding
Council and was informed by the practical activities and research of Dr Stephen Marshall
who developed an ideal model of e-maturity (eLearning Maturity Model)
The questions in the DSA cover five aspects
Learning (processes that impact on pedagogical aspects of e-learning)
Development (processes that surround the creation and maintenance of resources)
Support (processes surrounding oversight and management)
Evaluation (processes surrounding the evaluation and quality control)
Organisation (processes associated with planning and management)
As the DSA is based on the eMM developed in New Zealand it also maps the aspects against
the five dimensions of capability
Policy
Planning
Delivery
Management
Improvement
Using external assessment this process leads to the development of a grid which provides a
management level view of the e-maturity of a college an example is shown in figure 6
Figure 6 Management Level E-Maturity Grid
In addition to the completion of the grid the Scottish Funding Council also gathered
feedback on specific issues through the use of a questionnaire completed by institutions and
individuals which looked at the following themes
DSA
Po
licy
Pla
nn
ing
Delivery
Man
ag
em
en
t
Imp
rovem
en
t
Learning
L1 e-Learning is designed according to pedagogically sound principles N P L F P
L2 Learning skills are developed explicitly amp incrementally during delivery P F P P P
L3 Peer amp tutor interactions are provided explicitly during e-learning delivery P P N P P
L4 Courses take account of student workload and e-assessment demands P P N L P
L5 e-learning actively engage students P P P P N
L6 College defined (optional) na na na na na
Development
D1 Course development is directed and supported P P L F P
D2 Resources are maintained and re-used P F P P P
D3 Courses meet the needs of all students P P N P P
D4 The e-learning infrastructure is reliable robust and sufficient P P N L P
D5 The e-learning infrastructure is standardised and integrated P P P P N
D6 College defined (optional) na na na na na
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 18 | Coralesce Ltd wwwcoralescecom
Infrastructure
Teaching and Learning
Organisation
Strategic Leadership
e-Learning coverage by department subject area
The questions covered the extent to which e-learning was used across levels type of
learning and to what extent it was used in Work based Learning An important aspect that
the questionnaire covered under the heading of strategic leadership was the financial
aspects including affordability of software implementation and cost benefit analysis
eQSF A Quality Standards Framework for ICT in Learning
Sector Learning and Skills
Country Northern Ireland
Year 2009 10
eQSF was developed using European funding and provides a comprehensive evaluation and
action planning package to assess practice in the use of existing and new technologies The
framework consists of three pillars each with a series of standards and examples of
evidence
Values Pedagogy and Practice (see figure 4)
Structure of the Learning Process
Leadership and Management
The pillars break down into 13 quality standards each with a separate on-line template to
complete Each standard is graded on a scale of attainment from Absent to Advanced
However the statements tend to describe processes rather than outcomes and are quite
broad particularly in the section on Teaching and Learning
The online resource facilitates the self-assessment process provides opportunities to
identify evidence and results in an action plan for improvement The implementation
process involves establishing leadership and ownership reviewing practice action
planning implementing action plan evaluating progress and feedback
Positive feedback is available from the three pilot case studies that were developed
including North West College in NI which indicate the tool has been used at teacher
department and organisational level
http wwweqsfeu
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 19 | Coralesce Ltd wwwcoralescecom
Figure 4 Values Pedagogy amp Practice Standards
Standard Addressed
Yes No
Actions
Completed Current Attainment Level Signature
11
The organisation supports inclusiveness and accessibility through the use of ICT and sees ICT as promoting equality and diversity in the learning process
Absent Acquiring Achieved Advanced
out of
12
The organisation facilitates the adaptation of learning materials to meet a range of learner requirements and sees ICT as a mechanism for delivering such flexibility for the practitioner and learner
Absent Acquiring Achieved Advanced
out of
13
The organisation uses ICT to extend learning beyond the classroom and supports progression to further learning and career opportunities
Absent Acquiring Achieved Advanced
out of
14
The organisation is committed to the learnersrsquo outcomes being positively impacted by ICT
Absent Acquiring Achieved Advanced
out of
Comment
Adult and Community Learning e-Learning Positioning Statement
Sector Adult and Community Learning
Country England
Date 2005-2007
The ACL eLPs were designed by CEL (Centre for Excellence in Leadership) and NIACE
(National Institute for Adult amp Continuing Education) as an aid to both institutional
strategic planning in e-learning and to support institutional quality improvement processes
As well as its use within NIACEs e-learning management and leadership courses the tool
was also embedded in individual consultancies and within the NIACE Digital Learning
teams capital investment funding work The ACL eLPs tool was superseded by BECTAs
Technology Improvement Leadership Tool - Generator NIACE formed part of the
development team for Generator and continued to contribute to its development and
piloting with community-based adult learning providers The use of the Generator tool was
integrated into NIACEs e-learning capital funding projects (CAMEL) The e-Learning
positioning statement comprised 31 elements grouped under five themes Organisations
graded themselves against a five point scale ranging from not yet started to embedded The
ACL eLearning Positioning Statement was designed as an aid to both institutional strategic
planning in e-learning and to support institutional quality improvement processes
The framework reviewed Vision and Strategic Planning Teaching and Learning Staff
Development Infrastructure and Managing ILT and e-Learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 18 | Coralesce Ltd wwwcoralescecom
Infrastructure
Teaching and Learning
Organisation
Strategic Leadership
e-Learning coverage by department subject area
The questions covered the extent to which e-learning was used across levels type of
learning and to what extent it was used in Work based Learning An important aspect that
the questionnaire covered under the heading of strategic leadership was the financial
aspects including affordability of software implementation and cost benefit analysis
eQSF A Quality Standards Framework for ICT in Learning
Sector Learning and Skills
Country Northern Ireland
Year 2009 10
eQSF was developed using European funding and provides a comprehensive evaluation and
action planning package to assess practice in the use of existing and new technologies The
framework consists of three pillars each with a series of standards and examples of
evidence
Values Pedagogy and Practice (see figure 4)
Structure of the Learning Process
Leadership and Management
The pillars break down into 13 quality standards each with a separate on-line template to
complete Each standard is graded on a scale of attainment from Absent to Advanced
However the statements tend to describe processes rather than outcomes and are quite
broad particularly in the section on Teaching and Learning
The online resource facilitates the self-assessment process provides opportunities to
identify evidence and results in an action plan for improvement The implementation
process involves establishing leadership and ownership reviewing practice action
planning implementing action plan evaluating progress and feedback
Positive feedback is available from the three pilot case studies that were developed
including North West College in NI which indicate the tool has been used at teacher
department and organisational level
http wwweqsfeu
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 19 | Coralesce Ltd wwwcoralescecom
Figure 4 Values Pedagogy amp Practice Standards
Standard Addressed
Yes No
Actions
Completed Current Attainment Level Signature
11
The organisation supports inclusiveness and accessibility through the use of ICT and sees ICT as promoting equality and diversity in the learning process
Absent Acquiring Achieved Advanced
out of
12
The organisation facilitates the adaptation of learning materials to meet a range of learner requirements and sees ICT as a mechanism for delivering such flexibility for the practitioner and learner
Absent Acquiring Achieved Advanced
out of
13
The organisation uses ICT to extend learning beyond the classroom and supports progression to further learning and career opportunities
Absent Acquiring Achieved Advanced
out of
14
The organisation is committed to the learnersrsquo outcomes being positively impacted by ICT
Absent Acquiring Achieved Advanced
out of
Comment
Adult and Community Learning e-Learning Positioning Statement
Sector Adult and Community Learning
Country England
Date 2005-2007
The ACL eLPs were designed by CEL (Centre for Excellence in Leadership) and NIACE
(National Institute for Adult amp Continuing Education) as an aid to both institutional
strategic planning in e-learning and to support institutional quality improvement processes
As well as its use within NIACEs e-learning management and leadership courses the tool
was also embedded in individual consultancies and within the NIACE Digital Learning
teams capital investment funding work The ACL eLPs tool was superseded by BECTAs
Technology Improvement Leadership Tool - Generator NIACE formed part of the
development team for Generator and continued to contribute to its development and
piloting with community-based adult learning providers The use of the Generator tool was
integrated into NIACEs e-learning capital funding projects (CAMEL) The e-Learning
positioning statement comprised 31 elements grouped under five themes Organisations
graded themselves against a five point scale ranging from not yet started to embedded The
ACL eLearning Positioning Statement was designed as an aid to both institutional strategic
planning in e-learning and to support institutional quality improvement processes
The framework reviewed Vision and Strategic Planning Teaching and Learning Staff
Development Infrastructure and Managing ILT and e-Learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 19 | Coralesce Ltd wwwcoralescecom
Figure 4 Values Pedagogy amp Practice Standards
Standard Addressed
Yes No
Actions
Completed Current Attainment Level Signature
11
The organisation supports inclusiveness and accessibility through the use of ICT and sees ICT as promoting equality and diversity in the learning process
Absent Acquiring Achieved Advanced
out of
12
The organisation facilitates the adaptation of learning materials to meet a range of learner requirements and sees ICT as a mechanism for delivering such flexibility for the practitioner and learner
Absent Acquiring Achieved Advanced
out of
13
The organisation uses ICT to extend learning beyond the classroom and supports progression to further learning and career opportunities
Absent Acquiring Achieved Advanced
out of
14
The organisation is committed to the learnersrsquo outcomes being positively impacted by ICT
Absent Acquiring Achieved Advanced
out of
Comment
Adult and Community Learning e-Learning Positioning Statement
Sector Adult and Community Learning
Country England
Date 2005-2007
The ACL eLPs were designed by CEL (Centre for Excellence in Leadership) and NIACE
(National Institute for Adult amp Continuing Education) as an aid to both institutional
strategic planning in e-learning and to support institutional quality improvement processes
As well as its use within NIACEs e-learning management and leadership courses the tool
was also embedded in individual consultancies and within the NIACE Digital Learning
teams capital investment funding work The ACL eLPs tool was superseded by BECTAs
Technology Improvement Leadership Tool - Generator NIACE formed part of the
development team for Generator and continued to contribute to its development and
piloting with community-based adult learning providers The use of the Generator tool was
integrated into NIACEs e-learning capital funding projects (CAMEL) The e-Learning
positioning statement comprised 31 elements grouped under five themes Organisations
graded themselves against a five point scale ranging from not yet started to embedded The
ACL eLearning Positioning Statement was designed as an aid to both institutional strategic
planning in e-learning and to support institutional quality improvement processes
The framework reviewed Vision and Strategic Planning Teaching and Learning Staff
Development Infrastructure and Managing ILT and e-Learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 20 | Coralesce Ltd wwwcoralescecom
The tool encouraged Adult and Community Learning providers to identify their position
and develop an action plan to improve The tool not only looked at developing staff
confidence and competence in the use of ICT it also encouraged reflection on the return on
investment and the development of digital literacy skills of learners
BECTA Generator
Sector FE HE Learning and Skills
Country England
Year 2008 - 2011
A self-assessment tool developed through partnership between LSN Jisc and BECTA which
enabled learning providers to identify the impact that technology had in 9 key business
areas The tool could be used for a whole organisation or across one or more curriculum
areas The aim of the tool was to enable learning providers to effectively target their
technology resources and better meet organisational and learner goals
The tool was complex using 9 business processes grouped into three categories
Engage and Empower Learners
1 Informed Demand
2 Equitable Access
3 Proactive Support
Enhanced Learning Experience
4 Tailored content resource
5 Flexible Delivery and pathways
6 Personalised assessment
E-Confident and effective provider
7 Confident leadership and workforce
8 Innovative use of resources
9 Self-improving organisation
Each business process had a set of outcome based questions to be completed see figure 5
for example of informed demand questions There was the option of carrying out a quick
review or a full review in greater depth Providers also had the option of submitting one
review per year for benchmarking against the sector
The nine business processes were reviewed against seven enablers of technology shown in
figure 6 The areas were reviewed on a continuum beginning developing performing and
pioneering
The tool was capable of reviewing two or more curriculum areas within an organisation and
plotting on a spider gram (figure 7)
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 21 | Coralesce Ltd wwwcoralescecom
Figure 5 Example of outcome based question for informed demand element of Generator
Tool
Figure 6 Seven enablers of technology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 22 | Coralesce Ltd wwwcoralescecom
Figure 7 Review of two separate curriculum areas against the business process ndash informed
demand
Review and Plan Technology in Action (RAPTA)
Sector Learning and Skills
Country England
Date 2012-2013
The self-assessment tool is based on the premise that technology is an integral element of
outstanding teaching learning and assessment in FE and Learning and Skills provision The
RAPTA tool is a key quality improvement approach primarily designed to be used as an
integral part of Self-Assessment Review The tool uses the Ofsted Common Inspection
Framework as its structure in order to support integration with other self-assessment
activity However the tool lacks focus on improvement and does not naturally lead to the
production of an action plan
The tool consists of a set of questions for teaching and learning and leadership and
management with supporting explanation However there is a lack of emphasis on
outcomes for learners and it does not communicate an understanding of the process of
learning The tool was developed as a result of the Generator Tool developed by BECTA and
therefore is not based on research on the impact of technology on teaching and learning
RAPTA provides some good guidance about evidence gathering and making judgements
but this is very academic in style and not easy to understand
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 23 | Coralesce Ltd wwwcoralescecom
E-Learning Maturity Model (eMM)
Sector Higher Education
Country New Zealand
Year 2009 10
A mature evaluation framework developed by Stephen Marshall underpinned by theory is
designed to support organisational change and is a system which can be used for
benchmarking The eMM provides a quality improvement framework that HE institutions
could use to compare their capability to develop implement and sustain e-learning
The tool is based on research undertaken by Stephen Marshall focused over a period of 18
months in 2009 10 during which time four HE institutions engaged in a process of change
influenced by technology using the eMM e-Learning capability was benchmarked and the
information generated was used to stimulate change within the four institutions when used
at the beginning and end of the research project
The system is very long and is largely focused on onlineblended learning but does have a
strong focus on culture organisation change
The framework consists of 35 inter-related processes organised into five key aspects
Learning
Development (the creation and management of e-learning resource)
Support
Evaluation
Organisation
The eMM system is externally validated as it supports external assessment rather than self-
evaluation Each of the 35 processes is further divided into a number of very clear
statements which are assessed against five dimensions of capability thereby providing a
view of holistic capability The five dimensions of capability are shown in figure 8 below
Figure 8 Five Dimensions of eMM Processes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 24 | Coralesce Ltd wwwcoralescecom
The statements and capabilities are assessed against a four point scale not adequate
through to fully adequate Assessment takes the form of external examination of evidence
interviews and surveys Results are presented in a grid providing a very clear visual
representation of the organisationrsquos e-learning capacity In addition each of the
dimensions of an individual process can be benchmarked against other participating
institutions
Assessment does not link to an inspection framework
Towards Maturity
Sector Industry
Country UK
Date 2004 - 2014
Towards Maturity is a mature (10 years old) self-assessment and benchmarking tool that
aims to answer the following question
What are successful organisations doing to deliver business results with learning
technologies and how can we learn from them
The self-assessment is a straightforward to use but lengthy online tool with a strong focus
on learners learning and outcomes The questions are designed to promote reflection It
provides a very effective and successful evaluation tool for e-learning It uses a set of KPIs
and identification of best practice to provide sophisticated benchmarking Whilst it is
focused on learning in the workplace it is a model which could be adapted for use in the
education and training sector
The model consists of six sections
Defining need (strategic and operational alignment of training)
Learner context (access motivation)
Work context (integration with technical infrastructure and HR processes)
Building Capacity (designing learning assessing achievement supporting learning)
Ensuring engagement (involving leaners engaging trainers)
Demonstrate value (measure effectiveness and communicate benefits to key
stakeholders)
Organisations self-assess themselves against the model and receive a report where they can
benchmark themselves against industry averages using the following rating scale
Novice
Sporadic
Developing
Established
Embedded
The tool is free to use in June and July each year to ensure that a reasonable dataset is
produced for benchmarking purposes later in the cycle a fee is charged
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 25 | Coralesce Ltd wwwcoralescecom
The reports are based on nine KPIrsquos and provide detailed descriptors in the form of a grid
measured against three levels beginning progressing and proficient Numerical indicators
are also used to report statistical data Whilst the system is a very effective tool it does rely
on the views of managers who are completing the tool on behalf of their organisations but it
is recognised that there is sometimes a mismatch between the views of staff and managers
This tool could be adapted for use in the education and training sector
52 Individual Self-Assessment Competency
Frameworks
UNESCO Competency Framework for Teachers
Sector All
Country Global
Date 2011
The international competency framework is underpinned by research contributed to by
Cisco and Microsoft It supports teachers to develop students to become collaborative
problem solving creative learners and includes three aspects
Technology Literacy
Knowledge Deepening
Knowledge Creation
Within the three strands there are six dimensions
Understanding ICT in Education
Curriculum and Assessment
Pedagogy
ICT
Organisation and Administration
Teacher Professional Learning
Developed by UNESCO the framework embraces the UNESCO and the UNrsquos aims of
poverty reduction improved quality of life and improved quality of education Together the
aspects and dimensions create 18 modules which provide a framework (figure 9) for teacher
education institutions and providers to adapt to develop effective CPD programmes
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 26 | Coralesce Ltd wwwcoralescecom
Policy awareness Policy understanding Policy innovation
Figure 9 UNESCO ICT Competency Framework for Teachers
Whilst it is a very sophisticated system and is not easily adaptable the knowledge creation
part of the framework could be adapted to form part of an individual self-assessment tool
which is integrated into the organisational self-assessment tool
The Open University Digital Literacy Framework
Sector Various
Country England
Date 2014
The digital literacy framework developed by the Open University is a very clear and detailed
framework which identifies competencies expected at levels 0 1 2 3 and M across a range
of areas
Understand and engage in digital practices
Find information
Critically evaluate information online interactions and online tools
Manage and communicate information
Collaborate and share digital content
Comprehensive and high quality support materials are provided for students There is a
self-assessment tool which is presented in 4 separate lsquo10 minutersquo sections with links to a
very large and wide-ranging set of bite-sized learning materials on the lsquoBeing Digital Sitersquo
eg lsquoWho to Trust On-linersquo (5 minutes) lsquoCommunicating On-linersquo (10 minutes) In addition
much useful guidance and support is provided for teachers eg A guidance document
lsquoDeveloping Digital Literacies through Learning and Teachingrsquo 2013 and the lsquoInformation
Literacy Toolkitrsquo with activities and resources for teachers to use
This framework could be a used as a key foundation for the individual self-assessment for
staff and potentially for students
THE UNESCO ICT COMPETENCY FRAMEWORK FOR TEACHERS
KNOWLEDGE
DEEPENING
KNOWLEDGE
CREATION
UNDERSTANDING ICT IN EDUCATION
CURRICULUM AND ASSESSMENT
Basic knowledge Knowledge application Knowledge society skills
PEDAGOGY Integrate technology Complex problem solving
Self-management
ICT
ORGANISATION AND ADMINISTRATION
Basic tools
Standard classroom
Complex tools
Collaborative groups
Pervasive tools
Learning organisations
TEACHER PROFESSIONAL LEARNING
Digital literacy
Manage and guide
Teacher as model learner
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 27 | Coralesce Ltd wwwcoralescecom
Digilit Leicester
Sector Schools - Secondary
Country England
Year 2012 2013
The Digilit Leicester project was a two year collaboration between Leicester County Council
De Montfort University and 23 Secondary Schools This collaborative approach was key to
its success
The digital literacy framework was developed in consultation with schools and staff and
mapped to classroom practice The initiative built on Building Skills for the Future (BSF)
ICT priorities The partners worked collaboratively to drive development across six strands
of digital literacy for secondary school staff
Finding evaluating and organising
Creating and sharing
Assessment and feedback (see figure 10)
Communication collaboration and participation
E-Safety and on-line identity
Technology supported professional development
Surveys were developed from the framework and completed by teachers and managers
between April and July 2013 The rating scale used the following descriptors
Entry
Core
Developer
Pioneer
As a result the following definition of digitally literate educators was produced
ldquoTo be digitally literate educators must be able to utilise technology to enhance and
transform classroom practices and to enrich their own professional development and
identity The digitally literate educator will be able to think critically about why how and
when technology supplements teaching and learningrdquo
The framework is very clear and concise and puts learning and learners at the centre with a
strong focus on outcomes City-wide school and departmental reports were produced as
part of the survey therefore if required the tool could support benchmarking
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 28 | Coralesce Ltd wwwcoralescecom
Figure 10 Screenshot of Online Survey - Assessment and Feedback section
Assessing practitioner e-maturity Developing a benchmarking tool to measure
practitioner ICT capability in Further Education BECTA Pilot Study
Sector FE
Country Europe
Date 2010
The International Benchmarking Project was commissioned by BECTA to help explore how
teaching staff in further education colleges perceive their skills and use of technology for
teaching and learning the extent to which these assessments are shared with staff in similar
circumstances working in other countries and how these assessments can be used to assist
institutions in the sector to make more efficient and effective use of technology to provide
improved outcomes for learners
For BECTA and partners this initial trial phase of the project was designed to support
learning about practitioner perceptions and the implications for their organisations and test
the methodology
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 29 | Coralesce Ltd wwwcoralescecom
The survey instrument was based on experience gained in the annual surveys of managers
and practitioners in English colleges and drew on international work including UNESCO
standards on the use of technology for learning and teaching
This survey focused on teaching staff in general further education institutions Data was
collected from English FE Colleges and from four other European tertiary systems Austria
Denmark Portugal and Sweden
The online survey instrument was designed to map teachersrsquo confidence in using ICT within
a framework of six themes
Me - What can I do with ICT and how do I acquire and update those skills
My influences - To what extent is my use of ICT shaped by my colleagues or by
external influences
My learners - What ICT skills do my learners possess and what expectations do they
have of using them in their learning
My curriculum - To what extent do the curriculum content and the assessment
regime allow me to use ICT
My organisation - What does my organisation require me to do with ICT and what
developments does it encourage
The outcomes - Does ICT have a beneficial impact on my work as a teacher Does
ICT have a beneficial impact on the results of my learners
6 Comparative Analysis and Evaluation of
Organisational and Individual Self-Assessment
Competency Frameworks
Each self-assessment framework has been evaluated against the following criteria impact on
teaching and learning suitability for self-assessment quality improvement suitability for
the sector and sustainability
Teaching and Learning
The three school frameworks examined Naace DigiLit Leicester and Washington Essential Technology Rubric are informed by research on the effective use of learning technologies and focus on the classroom based delivery with no reference to on-line learning The frameworks use a similar model and look at the effective integration of technology and the development of learnersrsquo digital literacies This is expressed particularly well in the Washington Essential Technology Rubric which has dual standards of Technology Integration and Digital Citizenship moving from technical literacy to technical fluency The Naace framework does not review the use of technology to support assessment monitoring and feedback however this is covered more comprehensively in the Digilit Leicester framework and in the ACL e-Learning Positioning Statement which also includes a section on the use of technology in Initial Assessment The school frameworks reviewed have a strong focus on outcomes for learners The Naace framework details nine aspects of pupil learning experiences including progress in digital literacy confidence and independence effective and safe use of ICT creativity and learning
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 30 | Coralesce Ltd wwwcoralescecom
skills and contribution of pupils However the positioning of the section on teaching and learning in the middle of a particularly long document could mean that it is overshadowed by the detailed exploration of strategic management issues The UNESCO framework applies to all sectors is informed by research and focuses on the development of teachers It has desired outcomes for learners at the centre which are expressed as Technology Literacy Knowledge Deepening and Knowledge Creation The UNESCO framework covers similar areas to the school based Washington Essential Technology Rubric The BECTA Pilot study also focuses on development of teacher competence and built on other frameworks including the UNESCO framework Towards Maturity an industry framework for workforce development has a central focus on learning and is led by the desired outcomes for learners It provides an effective example of where benchmarking and ongoing feedback is used to develop a dynamic self-assessment tool The five Further Education and Skills sector frameworks examined (eQSF Development Self Audit ACL e-Positioning Statement Generator and RAPTA) differ in the extent to which they appear to have been informed by research on the effective use of technologies The most recent frameworks eQSF and RAPTA sought to have teaching and learning at the centre but both are limited in other ways Whilst the ACL e-Positioning statement is the precursor to Generator and RAPTA it has teaching and learning as a central core and explores the development of ICT literacy of learners and includes outreach provision The RAPTA model has a twin focus on lsquoinspiring encouraging and challenging learnersrsquo and lsquoencouraging independent learningrsquo but does not address the development of digital literacy skills or effective integration of technology into teaching and learning activities in the classroom or training environment The eQSF framework however is appropriate in its coverage of teaching and learning but describes processes rather than outcomes The HE framework the E-Learning Maturity Model is underpinned by research and expresses the importance of teaching and learning by making these the subject of the first two sections of a five section framework However other frameworks which focus on online learning and assessment have a narrow focus on teaching and learning
Suitability for Self-assessment
The scope of the frameworks reviewed is similar with each having a varying degree of emphasis on the different aspects The table in Appendix 2 provides an overview of the aspects covered in each framework examined Generally frameworks cover a core set of questions on
pedagogy
outcomes for learners
strategic planning
professional development for practitioners
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 31 | Coralesce Ltd wwwcoralescecom
Others frameworks also include information technology infrastructure quality assurance and financial management The Naace framework offered the widest scope including questions on aspects of general management whilst e-Learning Maturity Model and Generator are wide ranging and lengthy frameworks The Generator framework also offered a quick review option Other models such as RAPTA and the Washington State Technology Essential Conditions Rubric provide a narrower focus prioritising what they considered to be the key areas to develop to drive improvement Washington State for example evaluated in five areas ndash vision planning and policy technical literacy for learners integration standards for teachers technical competencies of administrators (leaders and managers) communication and connections RAPTA as a result of using the Common Inspection Framework focuses on those aspects of leadership and management considered most significant in ensuring high quality outcomes for learners The focus for Digilit Leicester and Towards Maturity is even narrower with Digilit Leicester evaluating five strands ndash finding evaluating and organising creating and sharing assessment and feedback communication collaboration and participation E-safety and on-line identity technology supported professional development The frameworks examined differ in the extent to which they involve relevant stakeholders in self-assessment Some involve a single senior manager making judgements on behalf of the whole organisation as in Towards Maturity This can reduce the rigour of findings Towards Maturity have identified through a study of 2000 learners that there can be a significant mismatch between the views of managers and learners on some areas an example being the willingness to use e-learning approaches where learners were much more positive than perceived by managers In contrast the DigiLit Leicester framework involves both teachers and managers Some of the more recent tools such as RAPTA and eQSF encourage the appropriate involvement of learners teachers and managers but this is not built into the tool The language used in evaluation descriptors differ across the frameworks and therefore communicate different approaches to development Some describe a journey for example
eQSF ndash lsquoabsent acquiring achieved advancedrsquo
E-Learning Maturity Model ndash lsquodelivery planning definition management and optimisationrsquo
ACL e-positioning statement ndash lsquonot yet started early stages developing established and embeddedrsquo
Towards Maturity ndash lsquobeginning progressing and proficientrsquo
DigiLit Leicester ndash lsquoentry core developer pioneerrsquo
Naace - uses a four point numerical scale like the Ofsted scale The usability of the frameworks across a range of stakeholders is varied with some questions and grade descriptors such as the DigiLit Leicester communicating clearly to a range of users However others namely those used by the E-Learning Maturity Model would only be understood by managers trained in the process The descriptions used in Towards Maturity and DigiLit Leicester would support practitioners and managers who lack
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 32 | Coralesce Ltd wwwcoralescecom
confidence as they are easy they are easy to understand and are developmental rather than numerical All the school frameworks examined are easy to read and understand The Digilit Leicester framework is concise and seeks to identify the most significant areas to address in order to drive progress in the effective use of learning technologies In contrast some of the other frameworks are not easy to understand for example RAPTA where the supporting guidance is written in an academic style which limits accessibility and the E-Learning Maturity Model which is discussed above There are a number of approaches used to ensure rigour Some frameworks use evaluation grids using a competency based approach which describe expectations for different aspects and for each grade Good examples are the Naace framework and the ACL e-Learning positioning statement where the descriptors used in the grid are particularly clear This approach also supports the clear presentation of results in a graphical format with colour coding to show performance level However frameworks like this tend to be long and time consuming to complete Some frameworks make use of external assessors for example the E-Maturity Model and Naace where the host organisation makes a charge With Naace those schools achieving a good standard across the board receive the ICT mark Beyond this they offer a 3rd Millennium Learning Award which involves peer referencing In contrast Towards Maturity does not use external assessors but uses a set of nine KPIs which link the use of online and blended learning to reduce cost study time and business impact
use of E-Learning
of formal learning E-Enabled
Cost reduction
Volume of E-Learning delivered
Study time reduced
Business impact
All the above are self-reported but business results can be verified
Benchmarked reports are provided by several of the frameworks and others have the
potential for benchmarking In Towards Maturity the provision of benchmarked reports to
users is key to the effective self-assessment that results Generator also provided
benchmarking but against self-reported assessments and did not use external assessors
Where frameworks simply consist of a set of questions (and are therefore shorter and less
daunting) such as RAPTA the quality of the self-assessment carried out is dependent on
the skills of those making judgements As a support RAPTA provides some good guidance
on making judgements and the suggested use of a lsquowhatrsquo lsquoso whatrsquo and lsquowhat nextrsquo proforma
to record judgements This approach focuses on impact and the importance of using learner
views is also stressed by RAPTA
RAPTA seeks to stimulate reflection amongst users in order to maximise the learning
achieved by completing the tool Towards Maturity although longer also uses questions
designed to make users think and consider this to be a crucial saying that users need to get
something immediate out of completing a survey
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 33 | Coralesce Ltd wwwcoralescecom
The frameworks such as Naace and DigiLit Leicester are underpinned by learning clear learner expectations and the practice and skills of teachers provide opportunities to map to other standards such as those from inspection bodies and teacher professional standards Quality Improvement
A number of examples of good support for quality improvement were identified throughout
the research The online Naace Self Review Framework results in an automatically
generated action plan as does the ACL e-Learning positioning statement There is an
excellent range of support materials for schools and teachers on the Naace website eQSF
also provides an online action planning tool Other sources of high quality support
materials identified are those produced to support the Australian Flexible Learning Quality
Framework and the Open University Digital Literacy Framework Towards Maturity
provides individual reports for users which benchmark their performance against industry
averages and provides concrete ideas from the best performers on how to improve Very
effective use is also made of its extensive database to publish bespoke reports which provide
further general advice on improvement eg The New Learning Agenda in 2014 which
highlighted nine key features for successful training and highlighting that learners needs to
be treated as customers
Suitability for the whole sector
The eQSF framework has been piloted in a number of different settings in the Further
Education and Skills sector and RAPTA because of its use of the Common Inspection
Framework was also applicable to all Whilst the e-Learning Positioning Statement focused
on Adult and Community Learning providers it could be easily adapted for the wider
Learning and Skills sector Other frameworks which also tended to be more detailed were
very sector specific
The frameworks which focus on self-assessment of individual competencies such as DigiLit
Leicester the BECTA Pilot Study and the UNESCO ICT Competency Framework provide
models which could be adapted for the FE and Learning and Skills Sector
Sustainability
One of the features of the 12 tools reviewed is that ongoing usability is often tied to external
funding When a grant awarding body closes the tools disappear from view and often
replacement is staggered and there is a loss of intelligence and lsquoknow-howrsquo in tool creation
The two tools that have had significant sustainability are Towards Maturity (10 years) and
Naace (6 years) as they have developed their own self- funding method for maintenance
7 CONCLUSIONS
In reviewing the frameworks the following conclusions have been drawn to support the
development of a self-assessment tool that builds on the previous work supported by the
Learning and Skills Improvement Service (LSIS) in the form of the RAPTA tool
Teaching and Learning
The self-assessment tool should be explicitly linked to how learning technologies support
teaching and learning and should be based on evidential research
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 34 | Coralesce Ltd wwwcoralescecom
The tool should be underpinned by
A clear and up-to-date model of expected impact such as the one used by Washington
State or UNESCO It should be broad enough to encompass all aspects of teaching
and learning in all settings (eg classroom workplace online)
Digital literacy expectations for learners in different sectors and at different levels of
study
Expectations for practitioners around effective integration of technology and digital
literacy and fluency
Learner and practitioner expectations should be referenced against other appropriate
standards such as the Ofsted Common Inspection Framework and the New Professional
Standards for Teachers
The self-assessment tool should facilitate access by a range of stakeholders by being concise
opening with questions relating to teaching and learning followed by questions on the other
key areas which support teaching and learning It is recommended that the forthcoming
consultation process considers both the length and the scope of the tool
Evaluation descriptors should be easy to understand and developmental in tone as in the
Towards Maturity and Digilit Leicester frameworks
Self-Assessment
Both the self-assessment tool and its use in pilots should model guide and support good
self-assessment practice
Education and Training providers use a range of self-assessment approaches and it is
crucial that the tool can be effectively integrated into existing processes rather used as a
lsquobolt-onrsquo Developing a broad understanding of this issue needs to be a key aim of the
consultation process
Many providers have developed materials additional to the Common Inspection Framework
for example competency style grids for teaching and learning or descriptors of best practice
both of which are also used in observation schemes and appraisal Some use detailed and
structured pro-formas for self-assessment reports others just the CIF headings of outcomes
for learners teaching learning and assessment and leadership and management The self-
assessment tool needs to be designed so that it supports the differing self-assessment
approaches employed This could be achieved for example by a layered approach - by
having a set of lsquokey questionsrsquo each of which is then underpinned by a detailed grid showing
expectations at different levels
The tool should facilitate the involvement of learners practitioners managers senior
leaders governors and employers for example
By having separate but related banks of questions which can be integrated into other
quality assurance processes eg learner surveys and focus groups employer surveys
Using clear and accessible language
It is recommended that survey questions are designed to stimulate reflection as seen in the
Towards Maturity framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 35 | Coralesce Ltd wwwcoralescecom
An option for ensuring cross-sector rigour is the use of external assessors eg Naace and the
E-Maturity model However the cost of this could prove to be a barrier to some
organisations An alternative option would be to develop a peer referencing approach
which would also have the potential to support a cross-sector collaboration
It is recommended that the Towards Maturity approach to benchmarking including the use
of key performance indicators and which does not rely on the use of external assessors is
further investigated particularly as a tool for self-assessment of individuals In addition the
Towards Maturity style of benchmarking would facilitate continuous improvement by
applying sector learning when updating the self-assessment tool It also provides a possible
model for sustainability with a combination of free and charged-for benchmarking reports
However this approach may not develop sufficient focus on development improvement if
used as a basis for an organisationrsquos self-assessment tool
Quality Improvement
The self-assessment tool must provide effective support for quality improvement through
for example providing automatically generated action plans Additionally it needs to be
used to support development The self-assessment tool should be used in a way that
supports and builds confidence in those whose digital skills are low This could be achieved
by employing collaborative approaches working with teams of practitioners to set up
supported experiments and by ensuring immediate access to training and support It is
recommended that this area is a key focus in the pilot phase of the project
For the more confident users of the tool there is a wealth of high quality support material
freely available online which could be utilised and linked to the tool eg Open University
digital literacy materials and the support site for the Australian Flexible Learning
Framework ndash wwwnationalvetcontenteduau
Suitability for the whole sector
Producing a self-assessment tool that works well for the whole sector will be a challenge but
achievable if a principle based approach is used and if exemplars from different settings are
routinely used Working collaboratively with different parts of the sector as the tool is developed will be key here
Sustainability
The design of the tool should take account of ongoing use after the pilot period and should
generate its own funds to ensure maintenance in the sector
8 NEXT STEPS
The analysis and evaluation of the institutional and individual self-assessment frameworks
has provided a robust overview of the diverse tools available both nationally and
internationally The frameworks examined have covered a range of sectors schools
education and training and higher education and many have elements which will be useful
in developing a self-assessment tool for the sector
The benchmark research evaluated frameworks against a number of criteria central focus on teaching and learning addresses learner outcomes leads to quality improvement and
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 36 | Coralesce Ltd wwwcoralescecom
sharing of best practice links to existing quality standards engages all staff is suitable for the whole sector and can be sustained and readily updated The tools reviewed encompassed a range of themes and used a variety of grading systems including numerical and descriptive grading They also presented different approaches to assessment and validation In addition to the wide variety of questions asked and grading systems used there was an enormous variety in the length of the framework some with as many as 54 questions (Naace) and others presenting a more streamlined approach The frameworks were written for a number of audiences within institutions and some specifically focused on the digital literacy competency of practitioners These key findings will now shape our approach to consultation with the education and training sector which will focus on the following themes Evaluation Criteria
Were these the correct criteria to assess the self-assessment frameworks against
Themes
Outcomes for learners (learner experience progress and achievement development of learnersrsquo digital literacy skills)
Teaching learning and assessment (pedagogy resources e-assessment)
Strategic leadership and management
Practitioner development (including practitioner digital literacy)
Engaging the Sector
Length of and audience for the self-assessment tool
Single or multiple contributors
Grading Methodology
Grading methodology (descriptive or numerical)
Benchmarking
Validation processes
Quality Improvement
Integration with other organisational self-assessment activity
Inclusion of features which support organisational development
Practitioner Development
Would a practitioner self-assessment tool be valuable and should it be consistent with and link to the institutional self-assessment tool
Should a practitioner based tool be informed by and contribute to teacher education frameworks
These consultation questions will be explored in a series of focus groups which will run in August and September 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 37 | Coralesce Ltd wwwcoralescecom
Appendix 1 Quality Frameworks for E-Learning
The QA-QE Special Interest Group is a group of practitioners both academic and professional support staff interested in using technology to enhance the quality of learning teaching and assessment Their toolkit provides a framework and a set of tools based around standard quality assurance procedures in HE which can be used when employing technology in teaching learning and assessment (Technology Enhanced learning - TEL) The Toolkit aims to encourage a shift from a culture of quality assurance and reporting to one of quality enhancement and course improvement and supports practitioners in ensuring that all aspects of the course design implementation and maintenance processes which may be impacted by the use of TEL are addressed by their institutionrsquos quality assurance procedures http qaqe-signet page_id=24 Skills for Health in collaboration with Strategic Health Authorities in England have developed a quality framework for e-learning The framework outlines key quality principles which should be considered when developing an e-learning programme for use by the healthcare workforce This framework which includes a useful checklist will be particularly helpful to commissioners of e-learning for use by the health workforce those education and e-learning specialists providers providing e-learning developments to the health sector and trainers and educators involved in the design delivery and evaluation of e-learning http
wwwskillsforhealthorguk about-us news launch-of-the-quality-framework-for-e11learning The Australian Flexible Learning Framework provides a set of technical e-Standards for online learning This is supported by an e-Learning Quality Model developed by the Flexible Learning Advisor Group The website wwwflexiblelearningnetacu provides a good example of a practitioner support website containing useful support materials and links to additional sites such as the wwwnationalvetcontenteduau These resources include learning design tools and a useful document lsquoCommonly used E-Learning Applications and tools in VET ECBCheck is a quality improvement scheme that supports organisations to measure how
successful their e-learning programmes are and allows for continuous improvement though
peer collaboration and bench marking ECBCheck provides a participative quality environment
allowing members to benefit by having access to tools and guidelines for their own practice the
website http wwwecb-checkorg
New Zealandrsquos eLearning guidelines (eLg) support individuals and organisations wanting to develop online teaching and learning The 2014 version considers new technologies developed since the guidelines were first established 10 years ago This is supported by the launch of a new eLg website providing an interactive platform for people interested in engaging with the comprehensive material and those keen to contribute to its future development http elgacnz new-zealandE28099s-elearning-guidelines-revamp The Quality Matters Rubric is a set of 8 general standards and 41 specific standards used to evaluate the design of online and blended courses A scoring system and set of online tools facilitate the evaluation by a team of reviewers Unique to the Quality Matters Rubric is the concept of alignment This occurs when critical course components - Learning Objectives Assessment and Measurement Instructional Materials Learner Interaction and Engagement and Course Technology work together to ensure students achieve desired outcomes https wwwqualitymattersorg rubric
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 38 | Coralesce Ltd wwwcoralescecom
Appendix 2 Comparison of the focus of Self-Assessment Digital Literacy Frameworks
Themes Naace Washington State K12 Tech Conditions Rubric
Digilit Leicester
DSA eQSF ACL e-Learning Position Statement
Generator RAPTA eMM Towards Maturity
UNESCO Competency Framework
OU Digital Literacy
Strategic Leadership
radic radic radic radic radic radic radic radic
Infrastructure - Technology
radic radic radic radic radic radic radic radic
Infrastructure ndash Organisational and Estates
radic radic radic
Teaching and Learning ndashPedagogy
radic radic radic radic radic radic radic radic radic radic
Teaching and Learning ndashResources
radic radic radic radic radic radic radic
Teaching and Learning - Online e-assessment
radic radic radic radic radic radic radic
Learner experience outcomes and progress
radic radic radic radic radic radic radic radic radic
Learner experience ndash development of digital literacy
radic radic radic radic radic
Organisational - Professional Development
radic radic radic radic radic radic radic
Organisational ndash development of staff digital literacy
radic radic radic radic radic radic radic radic
Organisational ndash Financial Management
radic radic radic radic
Organisational ndash Quality Improvement
radic radic radic radic
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 39 | Coralesce Ltd wwwcoralescecom
Appendix 3 Glossary
BECTA British Educational Communications and Technology Agency
BIS Department of Business Innovation and Skills
BSF Building Skills for the Future
CEL Centre for Excellence in Leadership
DSA Development Self-Audit
eMM E-Learning Maturity Model
eQSF A Quality Standards Framework for ICT in Learning
FELTAG Further Education Learning Technology Action Group
LSN Learning and Skills Network
Naace National Association of Advisers for Computers in Education
NIACE National Institute for Adult amp Continuing Education
RAPTA Review and Plan Technology in Action
SEF Self Evaluation Framework
SRF Self Review Framework
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014
Review of Learning Technology Self-Assessment Baseline Report July 2014
P a g e 40 | Coralesce Ltd wwwcoralescecom
Appendix 4 Bibliography
Atkins L Fraser J and Hall R (2013) DigiLit Leicester 2013 Survey Results Leicester Leicester
City Council (CC BY-NC 30)
Australian Flexible Learning Framework 2013 Available from http e-
standardsflexiblelearningnetau
BIS (2014) Further Education Workforce Strategy
BIS (2014) Government Response to the recommendations from FELTAG
Digital and Information Literacy Framework Open University Available from http
wwwopenacuk libraryservices subsites dilframework
Dr Bergeson T Davidson C McGuigan C K-12 Educational Technology Learning Standards Dec
2008 Available from Educational Technology Office Educational Technology Director Office of
Superintendent of Public Instruction
eQSF A Quality Standards Framework for ICT in Learning Available from http eqsfeu
Further Education Learning Technology Action Group (2014) FELTAG Recommendations Paths
forward to a digital future for Further Education and Skills FELTAG London
Marshall S Kay D McGonigle R Pepler G Schmoller S Patterson W (2007) DSA Introductory Handbook and Workbook Sero Consulting Ltd Available from wwwserocouk sfc
Marshall S E-Learning and Higher Education Understanding and Supporting Organisational
Change AKo Aoteroa National Centre for Tertiary Teaching Available from http
akoaotearoaacnz organisational-‐change-‐e-‐learning
Now On (date not known) The Educational Technology Journal District Technology Self-
Assessment Form Available from http fromnowonorg techsurveyhtml
RAPTA Review and Plan Technology in Action 2013 Available from http
wwwexcellencegatewayorguk node 27128
The Self Review Framework Jan 2015 Available from wwwnaacecouk SRF
Towards Maturity Improving the Impact of Learning Technologies at Work 2014 Available from
http wwwtowardsmaturityorg
UNESCO ICT Competency Framework for Teachers Published in 2011 by the United Nations
Educational Scientific and Cultural Organisation 7 place de Fontenoy 75352 PARIS 07 SP
New Professional Standards for Teachers Education and Training Foundation May 2014