Developing the APTIS Testing System - piap.cl · Developing the APTIS Testing System Professor...

Developing the APTIS Testing System

Professor Barry O’SullivanSenior Advisor, English Language Assessment

Overview

Localisation

Developing theories

Computerisation

Conclusions

DEVELOPING THEORIESconceptualising the test

System Design

Thinking about Test Performance

PERFORMANCETask Language

Target Language

Task Performance Parameters

Social Factors

Personal Characteristics

Cognitive Characteristics

Then There’s Scoring…

Machine or Human

Answer Key or Rating Scale

Selection & Training Consistency

Accuracy

Interpretation

Validation Theory

Decisions, Decisions…

Why are we testing?

How Many Decisions?

TEST SCORE

Task Language•Vocabulary &

Grammar•Genre, Mode, etc.

Target Language•Vocabulary &

Grammar•Function, Mode etc.

Task Performance Parameters•Time & Support•Knowing Assessment

Criteria•Physical Conditions, etc.

Social Factors•Interlocutor [who;

number]•Relative Age, gender

etc.•Familiarity & Status•Etc.

Personal Characteristics•Physical•Psychological•Experiential

Cognitive Characteristics•Maturity•Ability•Etc.

Scoring System•Raters & Rating•Scales & Keys•Analysis & Grading•Etc.

Why all the Fuss?

Curriculumexplicitimplicit

Deliveryroom design

materialsteachers

Assessmentjudgemental

developmental

Decisions

FOCUS ON READINGtheories of language progression

The Reading Model

Word/Sentence

Paragraph

Text [within]

Text [external]

Inter Text [full text level]

Inter Text [paragraph level]

A

B

C

How it Works

Introducing

CORE[Grammar & Vocabulary]

Speaking

Writing

Listening

Reading

LOCALISATIONthe british council approach

Theory to Practice – Test Design

Scoring System

Appropriate Content

Appropriate Task

Appropriate Language

Appropriate Scoring

Appropriate Topic

Test Taker

Test Task

LOCALISATION

Defining Localisation

The process of ensuring that all construct-irrelevant

variance related to the test taking population is

controlled for in the design and delivery of a test

Testing a well defined population within a well defined context in order to make decisions that will apply only to that context

Localisation – Case Study

Curriculum

Delivery Assessment

Case Study - Approach

Context

Population 15 & 17 yrsSecondary SchoolMixed ability, GenderVarious L1 [multilingual]Similar CultureSimilar CurriculumSimilar Delivery

Localisation

Language Reviewed[inc. instructions]

Reading TopicsListening TopicsWriting Topics/TasksSpeaking TopicsSpeaking Images

COMPUTERISATIONthe growing importance of the machine

The Computer & Test Practice

Item DevelopmentSpecificationWritingTriallingBankingArchiving

Test ScoringRater TrainingTest Break UpScoringAnalysing/AwardingScore Reporting

Test TakingKnowledgeReceptive SkillsProductive Skills

ResearchLanguageData AnalysisTask PerformanceRating Performance

System ManagementBack OfficeReporting to Clients

Test DeliveryComputerTabletOther…

The Computer & Test Practice

Item DevelopmentSpecificationWritingTriallingBankingArchiving

CONCLUSIONSwhat’s the point?

The Message?

Every decision impacts on ALL stakeholders, though especially on the test taker

Without a clearly defined ABILITY MODEL the test is suspect

Without a clearly defined VALIDATION MODEL the test is suspect

It is close to impossible to deliver any test without technology

Professor Barry O’Sullivan

barry.o’sullivan@britishcouncil.org