4 July 2004 CAA Conference 1 of 19

Assessments & Testing

Implementing large scale and rapid growth assessment programmes

Suzana LopesCommercial Director

4 July 2004 CAA Conference 2 of 19

Assessments & Testing

Presentation Outline

Pearson Company Overview

Case Studies:

Large Scale Assessment Programme DSA – Driving Standards Agency (UK) DVTA – Driver and Vehicle Testing Agency

(Northern Ireland)

Rapid Growth Assessment Programme City and Guilds

4 July 2004 CAA Conference 3 of 19

Assessments & Testing

Pearson Company Overview

Pearson plcPearson plc

£4 billion annual sales£4 billion annual sales

World’s # 1 education World’s # 1 education businessbusiness

Over 30,000 employeesOver 30,000 employees

Pearson VUE - The UK Market Leader in e-Assessment Pearson VUE - The UK Market Leader in e-Assessment

Acquired Promissor in January 2006 to expand assessment serviceAcquired Promissor in January 2006 to expand assessment service

Flexible and robust content development and delivery solutionsFlexible and robust content development and delivery solutions

Test centres in 145 countries and easy to deploy customer site testingTest centres in 145 countries and easy to deploy customer site testing

A commitment to quality and outstanding serviceA commitment to quality and outstanding service

4 July 2004 CAA Conference 4 of 19

Assessments & Testing

Case Study: The Driving Theory Test

• Contract awarded 31 October 2003 by Joint Authorities

• Our contract is for delivery of theory tests

• Nearly 1.8 million tests delivered annually

• Learner tests for: Car Motorcycle Truck Buses Driving instructor tests Driving examiner tests

4 July 2004 CAA Conference 5 of 19

Assessments & Testing

What needed to be achieved

• Establish a contact centre to handle: Telephone bookings Mail and fax bookings

• Build DDA compliant test centres at 151 locations England Scotland Northern Ireland Wales

• Build and publish the tests• Develop bespoke booking and reporting systems• Real-time eligibility checks• Hire, train and transfer staff from previous vendor

4 July 2004 CAA Conference 6 of 19

Assessments & Testing

What Needed To Be Achieved - The Timeframe

Nov 03 Dec 03 Jan 04 Feb 04 Mar 04 Apr 04 May 04 Jun 04 Jul 04 Aug 04 Sep 04

Contract awarded31 Oct 2003

Model centre launched1 March 2004

Contact centre opened1 Jul 2004

Registration andscheduling begins

26 Jul 2004

Testing ends withprevious contractor

3 Sep 2004

Testing beginswith Pearson4 Sep 2004

4 July 2004 CAA Conference 7 of 19

Assessments & Testing

Test Centres

Requirement: 80% of the UK population to be no more than 20 miles from their nearest test centre

• Difficult to find DDA-compliant (accessible) locations in some areas

• Northern Scotland was particularly a problem

• Decision was made to use coaches as mobile test centres

• Eventually found locations for 151 test centres Space that worked with the test centre plans Safe and appropriate Easily accessible by public transport

• Flexible opening hours

4 July 2004 CAA Conference 8 of 19

Assessments & Testing

Publishing The Tests

Requirement: Publish tests in English and Welsh - with voiceovers in 20 languages and available in British Sign Language

• First understand the presentation and scoring requirements

• Multiple choice and hazard perception components

• Thousands of the items built into a variety of tests

• Multiple forms of each test

• Hundreds of video clips for HPT test with specialized scoring

• Inventory the items and materials carefully

• Identify missing components

• Build the tests

• Test, test and test

4 July 2004 CAA Conference 9 of 19

Assessments & Testing

Test Centre Design and Test Delivery

Requirement: Design highly secure test centre with invigilator station

CCTV and audio monitoring and biometrics during check in process

• Produced working prototype of test centre meeting all requirements

• Opened on 1 March 2004

• Loaded Hazard Perception Test video clips on 30 stations to demonstrate smooth operation

• Test registration and scheduling

• Standardised procedures at the centres

• Hired contractors to kit out 151 test centres

and mobile centres

• Hired and trained staff

4 July 2004 CAA Conference 10 of 19

Assessments & Testing

Example Test Centre Floor Plan

4 July 2004 CAA Conference 11 of 19

Assessments & Testing

Critical Success Factors

• Excellent Project Management

• Working groups Software Testing centres Contact centre Test publishing Security

• Senior business group

• Experienced test development and delivery staff Members from Pearson VUE, DSA and DVTA

• Weekly update meetings

4 July 2004 CAA Conference 12 of 19

Assessments & Testing

Rapid Growth Assessment Case Study: City & Guilds

• 125 year old not for profit company• Royal Charter• UK’s largest vocational awarding body• 1.2 million learners• 8500 centres • 600 awards• 102 Countries • Operating in 22 sectors

4 July 2004 CAA Conference 13 of 19

Assessments & Testing

Overview

• 2001 City & Guilds decided to launch assessment and learning portal

• Plan to move centres to online registration and booking and develop computer based tests

• Global search for test partner

• Detailed specification

• Reliability, scalability and flexibility ranked highly among the attributes required

• 6 companies competed

• Promissor awarded the contract in 2002

4 July 2004 CAA Conference 14 of 19

Assessments & Testing

Attributes Of The Market

• Almost entirely paper based.

• Candidate choice was limited but growing

• Centres were cost focussed

• Constant policy and funding changes

• Backdrop of slow uptake of on line services

across industry

4 July 2004 CAA Conference 15 of 19

Assessments & Testing

Test Development

• Deployment team established• CBT item writers appointed by

City and Guilds• Training in Promissor Builder

Easy to Use – non expert interface

Web based tool – collaborative

Track and view content changes

Store multiple classification schemes

Create tests based on balanced content criteria

Allows randomized tests, test pooled from an item pool and adaptive test

4 July 2004 CAA Conference 16 of 19

Assessments & Testing

Test Delivery

• Customers consulted about design of system

• Regional training events held for customers

• Online training provided to use registration system

• Documentation provided for surveillance and invigilation at centre

• Technical support line

• Customer service support

Centre Approval

Promissor provide

Centre Criteria

Test Centre Installation

Test Centre Demo Test

Promissor make LIVE

4 July 2004 CAA Conference 17 of 19

Assessments & Testing

www.catglobal.com

Solution Overview

(ENTERPRISE)SITE MANAGER

www.catglobal.com

Candidate Checks-in

Booking

Test Administrator

Test Administrator

Test Administrator

CandidateTakes Test

Test Published for Delivery

Item Bank

BUILDERCandidateRegistration

Registration

C&GSystem

Scheduling

Candidates scheduled

on-line

Results

C&GSystem

Test CentreSet-up

Test Centre

4 July 2004 CAA Conference 18 of 19

Assessments & Testing

Rapid Growth

• 2002: 200 tests delivered

• 2003: Plan to move 80% of objective assessment to CBT

65k tests delivered

• 2004: 187k tests delivered

2006

• 100k test per month

• Over 250 computer based tests developed

• 6000 registered centres

GOLA - Monthly Running Totals 2002 to 2006

839199

708219

394273

253073

127761

64120

15294394

0

100000

200000

300000

400000

500000

600000

700000

800000

Oct-02 Jan-03 Apr-03 Jul-03 Oct-03 Jan-04 Apr-04 Jul-04 Oct-04 Jan-05 Apr-05 Jul-05 Oct-05 Jan-06

Month

Tes

t V

olu

me

4 July 2004 CAA Conference 19 of 19

Assessments & Testing

Critical Success Factors

• Senior management support

• Efficiency and value has to be proven internally as well as externally

• On-screen must be mandatory

• Technology right first time

• Training must be easily available

• Extensive support is needed