State of ReadinessNSC 2011

Umalusi Presentation to thePortfolio Committee

13 September 2011

Vijayen D Naidoo

Quality Assurance of the 2011 National Senior Certificate Examination

Regulatory Framework

QAA• NQF Act Section 27 (h)

The QC must develop and implement policy and criteria for assessment for the qualifications on its sub-framework.

Section17 of the GENFETQA Act(5) The Council must, with the concurrence of the Director-General and

after consultation with the relevant assessment body or education

institution, approve the publication of the results of learners if the

Council is satisfied that the assessment body or education institution

has—

(i) conducted the assessment free from any irregularity that may

jeopardise the integrity of the assessment or its outcomes;

(ii) complied with the requirements prescribed by the Council for

conducting assessments;

(iii) applied the standards prescribed by the Council which a learner is

required to comply with in order to obtain a certificate; and

(iv) complied with every other condition determined by the Council.

Framework for QA of Learner Achievement Based on established and existing practices in

assessment for certification

• Quality assures both prescribed components of External assessment (examinations) and Site based/ internal / continuous assessment

• Use of systems, processes, and procedures to evaluate, inspect, monitor and report on examination systems, processes and procedures of public and private assessment bodies.

Framework for Quality Assurance of Assessment Evaluation and /or accreditation of assessment bodies Periodic inspection of assessment bodies Ongoing monitoring of assessment bodies Quality assurance of external examinations through: Moderation of examination question papers Monitoring the conduct of examinations Moderation of marking Standardization of assessment outcomes

Framework for Quality Assurance of Assessment

• Quality assurance of internal assessment through: Monitoring the implementation of internal assessment Moderation of learner tasks and educator portfolios Statistical moderation of internal assessment results.

Approval of Results - Policy• Compliance with minimum requirements

• Shift from Moderation to Verification (Moderation for verification)

• Emphasis on Assessment Body reports The monitoring of the conduct of examinations and the

moderation of marking as well as the monitoring of the implementation of internal assessment and the moderation of internal assessment is undertaken by Umalusi to verify the veracity of the assessment body reports in respect of these process.

• Broadened definition of irregularities.

PURPOSE to ensure that the question papers are of the required standard To ensure that the question papers are relatively:

- fair

- reliable

- representative of an adequate sample of the curriculum

- representative of relevant conceptual domains

- representative of relevant levels of cognitive challenge External moderators (panels)

Ensuring Standards: Moderating question papers

Scope of the quality assurance processes

Number of NSC 2011 question papers moderatedTarget Number of

question papers moderated

Approved

DBE Nov 2011 129 129 92% (P2 languages still to be approved)

DBE Mar 2012 129 129 92%

IEB Nov 2011 76 76 100%

ERCO Nov 2011 40 40 88% (5 papers at third moderation)

STRENGTHENING THE APPROVAL PROCESS Additional layer of QA - final sign off Participated in pre- testing of sample question papers Post-exam analysis

Moderation of question papers

Internal assessment refers to any assessment conducted by the provider, the outcomes of which count towards the achievement of the qualification

panels of moderators / subject specialists Moderation at all levels Directives for internal assessment Prescribed criteria

- Educator Portfolios

- Learner portfolios (body of evidence)

Moderation of internal /continuous assessment

Ascertain the appropriateness and standard of the assessment tasks being developed within schools

Ascertain the degree to which assessment bodies/provinces are attempting to ensure standardisation across

Determine the extent and quality of internal moderation and educator development

Determine the reliability and validity of the assessment outcomes

Moderation of internal assessment cont

Moderation of SBA• Previous model - moderation of learner and

educator portfolios at the end of the year. • Implementation of new model in 2010 –focus

on tasks rather than on portfolios Ongoing moderations across 9 PDE’s, IEB & ERCO Improve feed back to educators Improvement in SBA implementation

Planned Completed Still to happen

Term 2/3Number of subjects: 7 subjects sampled across PDE’s plus Life Orientation across all PDE’sSubjects: Accounting, History, Geography, Life Orientation, Life Sciences, Mathematics, Physical Science, English FAL(each assessment body was allocated 3 subjects + LO)

Term 2/3: •DBE Moderation was conducted (4-6 July and 13-15 July). Umalusi monitored the DBE moderation. •Umalusi moderation /verification was conducted (11-21 July) •Mpumalanga & North West re-scheduled

Term 4 Number of subjects: 7 subjects sampled across PDE’s plus Life Orientation across all PDE’sSubjects: Accounting, History, Geography, Life Orientation, Life Sciences, Mathematics, Physical Science, English FAL

3-19 October

ERCOIEB

3-4 November3-5 December

Moderation of SBA • Non-adherence to sampling – EC• Provincial moderation still compliance/audit

focused• Quality of tasks and assessment must be

improved. • Training of Educators

The focus is on: Policy development and implementation Systems, processes and procedures Plan for assessment and moderation Availability and training of staff

Monitoring Administration of Exams

Monitoring of Examinations

Planned Completed Still to happen

State of readinessMonitored the following DBE support visits to identified provinces.

MpumalangaKZNEastern CapeLimpopoNorth West

•Free State (22 September)•Follow up visits to identified PDE’s

Monitoring Conduct of ExamsDeployment of monitors and staff to examination and marking centres.32 monitors (3- 4 per province) 9 Conveners and 23 Monitors: 4 days for writing the examinations2 days for marking4 staff members3 visits each for writing phase1 visit each for marking phase

Planning 17 October to 17 December

PURPOSE Moderation of marking determines the standard and quality

of marking and ensures that marking is conducted in accordance with agreed practices

Umalusi engages the following during the moderation of marking

1. Pre-marking/memorandum discussion: centralised memo discussions recommended - this will ensure consistency across marking centres

2. Off-site moderation of marking

3. On-site moderation of marking

Verification of Marking

Verification of Marking

Planned Completed Still to happen

Centralised verification of marking in 25 NSC selected subjects:

Post the paper being written

Verification of the Resulting Process Planned Completed Still to happen

Subject structures Subject structures verified.

Candidate registration System verified during state of readiness visits.

Generation of mark Sheets On-site during September 2011

Capturing of Marks On-site during September 2011

Standardisation data Data sets received and input/comments submitted to DBE.

Awaiting test data sets from DBE.

Standardisation booklets Booklets received and input/comments submitted to DBE.

Awaiting test data sets from DBE.

Capturing of adjustments Awaiting test data sets from DBE.

Statistical moderation & resulting Awaiting test data sets from DBE.

Standardisation and Resulting Planned Date

Qualitative Input meeting 19th December 2011

Pre-standardisation meetings 21st & 22nd December 2011

Standardisation meeting 23rd December 2011

Verification of resulting processes 23-29 December 2011

Approval of results 30 December 2011

Monitoring the State of Readiness New approach One Assessment body (DBE) with 9 provincial

arms vs. 9 separate assessment bodies. Avoid duplication (parallel processes)and burden

on provincial authorities “Lift” the system – Umalusi to assume a

verification role. “Shadowing” the DBE team to establish the

veracity of the DBE monitoring processes and to authenticate DBE reports.

The DBE Monitoring Process Audit of PDE systems PDE’s required to complete a self-evaluation

instrument and develop improvement plan. Monitoring visits (evaluation/support) to Head Office,

District & Exam Centers- monitored against Audit report and improvement plan for Examinations and Assessment.

Submission of composite report to relevant authorities (Umalusi included)

Comprehensive approach that monitors exam systems (including SBA)

Umalusi findings-KZN Compliance / Strong points

Mature /consolidated exam system Management and Operational plansMark capture system

ConcernsStaffing – vacant posts; contract workers Security at nodal points ( outlying areas)High number of marking centers Irregularities

Umalusi findings-EC Compliance / Strong points

Mature /consolidated exam system Management and Operational plans

Concerns

Staffing – vacant posts; contract workers Security – storage facilities at Butterworth Private Centers/independent schools Large number of marking centers Labour issues

Umalusi findings-Mpumalanga Compliance / Strong points

Management and Operational plans Key management posts filled Exam processes in place

• Concerns Coordination and communication between HO and districts Printing of Question papers Private centers Large number of marking centers Security and Appointment of markers

Umalusi findings-Limpopo Compliance / Strong points

Management and Operational plans 50 posts filled Exam processes in place

• Concerns Printing of Question papers Large number of distribution points Security Appointment of markers District monitoring Script control

Umalusi findings-North West Compliance / Strong points

Management and Operational plans Assessment Management coordinating committee Exam processes in place Printing

• Concerns Use of schools as distribution points Monitoring compromised – shortage of vehicles

Concerns: Other Provinces Gauteng

Independent Schools• Western Cape

Highly centralised –minimal district involvement Security of question papers – distribution and storage

• Northern Cape

Security – storage of question papers; return of answer books

Umalusi Overall Evaluation Systems and Processes are in place to ensure the

administration of the 2011 NSC examinations.

Identified risks will have to be closely monitored. Includes follow up visits.

Umalusi remains confident that we will be able to ensure a credible 2011 NSC examination.

Thank you!