Upload
paul-bacsich
View
758
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Draf
Citation preview
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 1
Success Criteria for Virtual Schools
- Pick&Mix
Professor Paul Bacsich
Sero Consulting Ltd and Matic Media Ltd
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 2
Topics
1. Introduction, disclaimers and acknowledgements
2. History of Pick&Mix in universities
3. Why is he telling us this?
4. Pick&Mix
5. More recent history: application to colleges and high schools
6. Reflections on this process
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 3
1. Introduction, disclaimers and acknowledgements
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012
Who is this talk for?
Want to know about comparing your virtual school with other ones?
Want to know about the “tradecraft” of benchmarking and quality reviews?
Want to adapt or update a benchmarking system?
Want to learn some of the underlying principles of such schemes?
4
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 5
Pick&Mix is an “Open Educational Methodology” for benchmarking online learning, developed
by Paul Bacsich and available for all to develop and modify, but the names
“Pick&Mix” and “ELDDA” are reserved for the “main sequence” of development
Thanks to many, including UK HE Academy, JISC, EU Lifelong Learning Programme
(Re.ViCa and VISCED), Manchester Business School, University of Leicester and
Sero Consulting Ltd for support
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012
2. History of Pick&Mix in universities
6
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 7
Benchmarking online learning
At national level, started in 2004-05 in UK and New Zealand– Soon spread to Australia– Not closely linked initially to quality agenda
At European level, developments include E-xcellence and UNIQUe– Some earlier work from OBHE, ESMU etc– Later, developments in other projects– Increasingly, links made to quality agenda and to
critical success factors
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 8
Pick&Mix history
Initial version developed in early 2005 in response to a request from Manchester Business School for an international (largely US) competitor study
Since then, refined by literature search, discussion, feedback, presentations, workshops, concordance studies and four phases of use – sixth and seventh phases now
Forms the basis of the wording of the Critical Success Factors scheme for the EU Re.ViCa project and now being used to develop a similar scheme for Virtual Schools in VISCED
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 9
Benchmarking e-learning (UK)
Foreseen in HEFCE 2005 e-learning strategy for universities in England (later for Wales via HEFCW)
Higher Education Academy (HEA) oversaw itFour phases – 82 institutions – 5 methodologiesTwo consultant teams – one run by myself My team benchmarked over 40 institutions using
4 methodologiesIncluding 24 using Pick&Mix – now well over 30
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 10
What HEFCE wanted
“Possibly more important is for us [HEFCE] to help individual institutions understand their own positions on e-learning, to set their aspirations and goals for embedding e-learning – and then to benchmark themselves and their progress against institutions with similar goals, and across the sector”
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 11
Methodologies in UK HEOf the five methodologies used in the UK from 2005 on, only
one survives that is actively under development, refined annually, public domain and available for supervised or self-applied use in institutions, via funded projects and commercially:
Pick&Mix
In other countries’ HE systems:– eMM is in a similar situation in New Zealand– Quality Matters is widespread in US– ACODE is used somewhat in Australia– On the continent of Europe there are a few methodologies but
fostered purely by EU-funded projects
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 12
Pick&Mix overview
Focussed on online learning, not general pedagogyDraws on several sources and methodologies – UK and
internationally (including US – especially Quality on the Line) and from college sector
Not linked to any particular style of online learning (e.g. distance or on-campus or blended)
Oriented to institutions with notable activity in online learning
Suitable for desk research as well as “in-depth” studiesSuitable for single- and multi-institution studies
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012
BECAUSE IT CAN BE USED FOR COLLEGES AND SCHOOLS ALSO
3. But why is he telling us this?
13
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 14
4. Pick&Mix
Criteria and metrics
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 15
Criteria
Criteria are “statements of practice” which are scored into a number of performance levels from bad/nil to excellent
It is crucial that these statements are in the public domain – to allow analysis & refinement
The number of criteria is also crucial: 24Pick&Mix originally had a core of 20 – based on
analysis from the literature (ABC, BS etc) and experience in many senior mgt scoring meetings
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 16
Pick&Mix: 20 core criteriaRemoved any not specific to online learning
– Including those in general quality/accreditation schemesCareful about any which are not provably success factorsLeft out of the core were some criteria where there was
not yet UK consensus Institutions will wish to add some to monitor their KPIs and
objectives. Recommended no more than 6.– Pick&Mix now has over 70 supplementary criteria to choose from– more can be constructed or taken from other schemes
These 20 have stood the test of four phases of benchmarking with only minor changes of wording– originally 18 - two were split to make 20
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 17
Pick&Mix Scoring
Use a 6-point scale (1-6)– 5 (cf Likert, MIT90s levels) plus 1 more for
“excellence”Contextualised by “scoring commentary”There are always issues of judging
progress especially “best practice”The 6 levels are mapped to 4 colours in a
“traffic lights” system – red, amber, olive, green
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 18
Pick&Mix System: summary
Has taken account of “best of breed” schemes
Output and student-oriented aspectsMethodology-agnostic but uses underlying
approaches where useful (e.g. Chickering & Gamson, Quality on the Line, MIT90s)
Requires no long training course to understand
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 19
P01 “Adoption” (Rogers)
1. Innovators only2. Early adopters taking it up3. Early adopters adopted; early majority
taking it up4. Early majority adopted; late majority taking
it up5. All taken up except laggards, who are now
taking it up (or retiring or leaving)6. First wave embedded, second wave under
way (e.g. BYOD-learning after e-learning)
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 20
P10 “Training”
1. No systematic training for e-learning2. Some systematic training, e.g. in some projects
and departments3. Institution-wide training programme but little
monitoring of attendance or encouragement to go4. Institution-wide training programme, monitored
and incentivised5. All staff trained in VLE use, training appropriate to
job type – and retrained when needed6. Staff increasingly keep themselves up to date in a
“just in time, just for me” fashion except in situations of discontinuous change
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 21
Supplementary criteria - examples
IT reliabilityMarket researchCompetitor research IPRHelp DeskManagement of student expectationsStudent satisfactionWeb 2.0 pedagogy
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012
Adaptation to colleges and high schools (both virtual and blended)
5. More recent history
22
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012
Virtual post-secondary institutions
The EU Re.ViCa project 2007-09 (Review of Virtual Campuses) did a great deal of work to refine and “Europeanise” the Pick&Mix criteria, focussing on – Critical Success Factors for virtual campuses
A slightly revised scheme was produced, oriented to institutions where distance learning was the prevalent approach
23
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 24
Distance Learning Benchmarking Club 2009-11
In summary: Universities, encouraged by a UK-funded project, formed a group to benchmark themselves:– University of Leicester (UK)– Royal Swedish Institute of Technology (KTH)– Lund University (Sweden)– University College Gotland (Sweden)– Thompson Rivers University (Canada)
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012
VISCED 2011-12
A further iteration is in final stages to produce a scheme of Critical Success Factors for virtual schools– With initial focus to European virtual schools– Since these are smaller and few have ever
failed, this is a hard task
25
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012
Draft key success factors for European virtual schools
1. Usability (for all participants)
2. Strategy (for online learning)
3. Recruitment and training (fused?)
4. Evaluation (of programmes) – often informal but effective
5. Reliability (of system)
6. Leadership (with knowhow, down and up – fused?)
7. Organisation (formal and informal)
8. Learning outcomes (often individualised)d
9. Use of resources (not key in universities)
10. Market research?
26
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012
6. Reflections on this process
27
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 28
Too many concepts
Benchmarking
StandardsQuality
Accreditation/approval
/kitemarking
(Critical) Success Factors
Online learning is only a small part of the quality process – how can agencies and assessors handle five variants of the concept across many separate methodologies?
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 29
My view for ENQA – the pyramid
Critical Success Factors, a selection of:
Benchmarking, which split into sub-criteria:
Quality, split into
Detailed pedagogic guidelines ----------
Criteria are placed at different layers in the pyramid
depending on their “level”
Leadership level
Senior managers
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012
Adaptability
The transition of a basically university-originated methodology to schools is feasible, particularly for larger high schools (virtual or blended)
The sectors are not as different as territorial experts like to believe
But the methodology has to be theoretically sound and well-researched
There are other extensions - eg for OER !
30
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012
Merging US and EU practice?But could we cross the transatlantic divide?There were some discussions between EU
institutions and QM over a common HE system (UNESCO would love that)
For schooIs, we need to look at iNACOL guidelines – meeting after the Symposium
Since Pick&Mix was heavily influenced by US experience (Ehrmann; Quality on the Line) there should be hope – at first sight it looks promising
A hierarchical (pyramid) approach will be key
31
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 32
A view after 7 years of Pick&MixMethodologies do not survive without regular
updating by a design authority– this is difficult in a leaderless group context
Forking of methodologies needs dealt with by folding updates back to the core system– otherwise survival is affected
Complex methodologies do not survive well A public criterion system allows confidence,
transparency, and grounding in institutionsSectoral boundaries can be overcome
iNACOL Virtual Schools Symposium 2012, New Orleans – October 2012 33
ReferencesThe Pick&Mix system is detailed at http://www.matic-media.co.uk/benchmarking/PnM-2pt6-beta3-full.xlsx
A key paper on the international aspects is:
“Benchmarking E-Learning in UK Universities: Lessons from and for the International Context”, in Proceedings of the ICDE conference M-2009, Open Praxis – http://www.openpraxis.com/files/Bacsich%20et%20al..pdf
A specific chapter on the UK HE benchmarking programme methodologies is:
“Benchmarking e-learning in UK universities – the methodologies”, in Mayes, J.T., Morrison, D., Bullen, P., Mellar, H., and Oliver, M.(Eds.) Transformation in Higher Education through Technology-Enhanced Learning, York: Higher Education Academy, 2009 – http://www.heacademy.ac.uk/assets/documents/learningandtech/Transforming-07.pdf