Evaluation and Evaluation and Benchmarking of ICT in Benchmarking of ICT in

SchoolsSchools

Professor Paul Bacsich Professor Paul Bacsich

UKUK

17 17 October 2003October 2003

AcknowledgementAcknowledgement

The Education and Manpower Bureau The Education and Manpower Bureau

would like to offer special thanks to would like to offer special thanks to

Professor Paul BacsichProfessor Paul Bacsich for his powerpoint for his powerpoint presentation on presentation on

Evaluation and Benchmarking of ICT in SchoolsEvaluation and Benchmarking of ICT in Schools

EvaluationEvaluation

In 4 easy lessonsIn 4 easy lessons

EvaluationEvaluation

• Programme-wide with deep drill-Programme-wide with deep drill-downdown

• Based on numbers and qualitative Based on numbers and qualitative info, with interpretationinfo, with interpretation

• Triangulation and stakeholdersTriangulation and stakeholders

Flashlight (after Ehrmann)Flashlight (after Ehrmann)

See See wwwwww.tltgroup.tltgroup.org.org And the Flashlight presentation at And the Flashlight presentation at

www.roundtable.acwww.roundtable.ac.uk/FLIntro.ppt.uk/FLIntro.ppt

The Flashlight OathThe Flashlight Oath

""Above all, I will do no harm. I know the value of Above all, I will do no harm. I know the value of people's time, and so I will be ruthless in making people's time, and so I will be ruthless in making my surveys and interviews focused, brief, and my surveys and interviews focused, brief, and useful…so that those who have answered my useful…so that those who have answered my questions will be even more willing to collaborate questions will be even more willing to collaborate with the next researcher or evaluator who with the next researcher or evaluator who contacts them."contacts them."

"My study's process as well as its findings will "My study's process as well as its findings will stimulate reflection and learning…"stimulate reflection and learning…"

Programme evaluation in UKProgramme evaluation in UK

Based on Flashlight Based on Flashlight Developed in the Annenberg/CPB trials of e-Developed in the Annenberg/CPB trials of e-

learning in HE c. 1995learning in HE c. 1995 and the OU evaluation consensus and the OU evaluation consensus

(Bates, Mason, Laurillard et al)(Bates, Mason, Laurillard et al) First substantial trial in the JANUS evaluations First substantial trial in the JANUS evaluations

of 1993-5of 1993-5 Refined through TERG (SHU) into a tool for Refined through TERG (SHU) into a tool for

evaluation of NLNevaluation of NLN Next set of slides are c/o Ash, Carter and MistryNext set of slides are c/o Ash, Carter and Mistry

NLN-e1 FindingsNLN-e1 Findings

ICT is only appreciated by students when ICT is only appreciated by students when there is a reason for itthere is a reason for it

ICT is an increasingly effective tool that is ICT is an increasingly effective tool that is used with increasing sophistication and used with increasing sophistication and appreciationappreciation

Increasing ICT skills at schools is pressurising Increasing ICT skills at schools is pressurising teacher skills at FEteacher skills at FE

Involvement in evaluation can facilitate Involvement in evaluation can facilitate colleges. organisational developmentcolleges. organisational development

Evaluation successesEvaluation successes

A further £84M funding for the skills A further £84M funding for the skills sector sector

DFeS are pushing the toolkit to DFeS are pushing the toolkit to evaluate coursesevaluate courses

Furthered thought in the identification Furthered thought in the identification of good practiceof good practice

Helping to raise awareness of the Helping to raise awareness of the issues in the pedagogy of ICT teachingissues in the pedagogy of ICT teaching

What is UK doing for the What is UK doing for the second phase of NLN second phase of NLN

evaluationevaluation Meta-analysis of current and recent Meta-analysis of current and recent

surveyssurveys Quantitative data gathering from Quantitative data gathering from

across the sectoracross the sector Twelve In-depth longitudinal studies Twelve In-depth longitudinal studies

developed by developed by externalexternal evaluators evaluators Focus on the Focus on the whole organisationwhole organisation

rather than the courserather than the course

What did Phase one tell us What did Phase one tell us about conducting internal about conducting internal

evaluations?evaluations? Tools and processes are often Tools and processes are often

secondary to good managementsecondary to good management The process of reviewing and The process of reviewing and

assessing work can be threatening assessing work can be threatening and evoke a deeply political responseand evoke a deeply political response

Evaluations serve multiple functions Evaluations serve multiple functions not always those the evaluators not always those the evaluators envisageenvisage

Conducting the evaluation Conducting the evaluation Data collection Data collection

Leave your baggage at the doorLeave your baggage at the door Pilot everythingPilot everything ALWAYS double check and share ALWAYS double check and share

insights with your informantsinsights with your informants Look from multiple viewpointsLook from multiple viewpoints Don’t over-gather the same Don’t over-gather the same

information from the same sourcesinformation from the same sources

BenchmarkingBenchmarking

Not an easy subjectNot an easy subject

Benchmarking – but who Benchmarking – but who with?with?

Similar countries/regions – but who is?Similar countries/regions – but who is?

Leaders, to provide aspirational role Leaders, to provide aspirational role

modelsmodels

Laggards in case they catch up!Laggards in case they catch up!

CriteriaCriteria

PopulationPopulation And population densityAnd population density

Type of language (alphabet, Type of language (alphabet, ideographic, etc)ideographic, etc)

Level of use of ICT in general lifeLevel of use of ICT in general life (Pedagogic) culture(Pedagogic) culture

SuggestionsSuggestions SingaporeSingapore Scotland not EnglandScotland not England Canadian (province level)Canadian (province level) Denmark not FinlandDenmark not Finland Australia (states)Australia (states) Thailand (laggard but trying to catch up)Thailand (laggard but trying to catch up) South Korea?South Korea? US – are we sure?US – are we sure?

ICT findingsICT findings

Key Key ratiosratios

Key Performance IndicatorsKey Performance Indicators

Students per PCStudents per PC Bandwidth per student PCBandwidth per student PC

Or per student (aspirational?)Or per student (aspirational?) Media sophistication of softwareMedia sophistication of software

Text, graphics, web, audio, videoText, graphics, web, audio, video Production to consumption ratioProduction to consumption ratio

Spot emerging trends even if Spot emerging trends even if smallsmall

Laptops/tablets are less intrusive and fit the Laptops/tablets are less intrusive and fit the paradigm of “ubiquitous computing”paradigm of “ubiquitous computing”

Binding of home PCs into school systems: Binding of home PCs into school systems: ”extranets””extranets”

Hosting: moving servers out of schools back Hosting: moving servers out of schools back to server farmsto server farms

New “worldware” apps?New “worldware” apps? Email integration with SMS on mobile Email integration with SMS on mobile

phones??phones?? Anything coming in from HE (Grid?) or Anything coming in from HE (Grid?) or

industry (cooperative working) or real life industry (cooperative working) or real life (games?)(games?)