An English Proficiency Test for Today’s Student Using Today’s TechnologyMarcie Mealia, 6.23.2015

3

PTE Academic

• Launched in 2009

• Developed to meet the need for a test that more

accurately measures the communicative skills of international students in an academic environment as

securely and efficiently as possible

Accepted by over 3,000 programs worldwide

• Available in 50+ countries and 230+ locations

• Uses automated scoring technology

4

Automated Scoring TechnologyAn efficient, quick process while providing consistent and objective results

• Ensures consistent item presentation across time, people, and locations

• Enables efficiency, objectivity, fairness, and consistency

• Dependable decision-making process

Development of the Data

Over 18 months, test data were collected from more than 10,000 test takers from 38 cities in 21 countries who participated in PTE Academic’s field tests. These test takers came from 158 different countries and spoke 126 different native languages, including (but not limited to) Cantonese, French, Gujarati, Hebrew, Hindi, Indonesian, Japanese, Korean, Mandarin, Marathi, Polish, Spanish, Urdu, Vietnamese, Tamil, Telugu, Thai and Turkish.

The data from the field test were used to train the automated scoring engines for both the written and spoken PTE Academic items.

Presentation Title runs here l 00/00/005

6

How Automated Scoring Technology WorksSpeaking Sections (Ordinate Technology)

PTE Academic Database

Local

Computer / Server

Scoring

Administrator’s data

management

“Vancouver is famous for its ethnic diversity.”

7

Automated Scoring TechnologyAssessed skills in Spoken scoring

• Sentence Mastery

• Content

• Vocabulary

• Accuracy

• Pronunciation

• Intonation

• Fluency

• Expressiveness

• Pragmatics

Intelligent Essay Essayer (IEA)Latent Semantic Analysis (LSA)

Test takers are asked to write 200–300 word essays and 50–70 word summaries.

Response is submitted for scoring

System evaluates the meaning of the response, as well as mechanical aspects of the writing. The system compares the response with the large set of training responses, computes similarities, and assigns a score based on content,

The system assesses each trait (grammar, structure and coherence, etc.,) in the test taker’s response, compares it with the large set of training responses, and then ranks the response according to that trait.

Presentation Title runs here l 00/00/008

9

How Automated Scoring Technology WorksWriting Sections

For a specific prompt, LSA engine is trained on 200+ essays and scored by expert human raters

LSA Training

Vector-based

Comparison

Score of the most similar essay is used

to predict what a human score would be by using modern

regression and classification

methods

Various variables added

Coherence

When a new essay is submitted, it is compared with all 200+ pre-scored essays

Word Flow Word Choice

10

Automated Scoring TechnologyAssessed skills in Written scoring

• Word choice

• Grammar & Mechanics

• Progression of ideas

• Organization

• Style, Tone

• Paragraph structure

• Development, Coherence

• Point of view

• Task completion

11

Automated Scoring TechnologyReliability and Validity

• LSA is as accurate as skilled human graders

• Agrees with a single human reader as often as two human raters agree with each other

Measure LSA to human raters

Human raters to human raters

Mean exact score agreement 57 - 59% - 74 43 - 56% - 87

Mean exact + adjacent agreement 95 - 98% - 100 88 - 97% - 100

Presentation Title runs here l 00/00/0012

Presentation Title runs here l 00/00/0013

Test Taker ProcessBefore test day

• Book test online or over phone– Prompted to set up account in Pearson VUE– Test dates can be made available based on demand

• Prepare for PTE Academic– Courses– Scored Practice Test– Sample Questions– Free materials including:○ Test tutorial○ Skills video○ Top tips○ Offline practice test

Presentation Title runs here l 00/00/0014

Test Taker ProcessOn test day

• Arrive to test center 30 minutes prior to test

• Show your ID and sign in digitally

• Get your digital photograph taken

• Scan your palm digitally–Must scan palm each and every time a test takers leaves or

enters testing room

• Check all belongings into locker

• One single 3 hour sitting

Presentation Title runs here l 00/00/0015

Test Taker ProcessAfter test day

• Receive email notification once your scores are available (within 5 business days)

• Login to your Pearson VUE account

• Assign score report(s) to institutions online through secure server– Ability to send unlimited number of score assignments at no

extra cost – Institutions do not accept a paper or pdf copy of score

report

Presentation Title runs here l 00/00/0016

PTE Academic Score Report

• Delivered through secure, computer-based server

• Scored against Global Scale of English–Granular score scale from

10-90

• Comprised of both communicative and enabling skills

Presentation Title runs here l 00/00/0017

PTE Academic Access

• Available in 50+ countries in 230+ locations

• Test sessions run 360 days per year around the world

• 51 new test centers opened in 2014

Presentation Title runs here l 00/00/0018

Test Taker BenefitsA fast, flexible, and convenient process

• Objective and accurate results

• Results available within 5 business days (typically 48 hours)

• One single 3 hour sitting

• Available in 50+ countries worldwide

• Test sessions can be made available based on demand

• Provides most detailed score report compared to any other English proficiency test

• Ability to send unlimited score report assignments

Presentation Title runs here l 00/00/0019

“Once I had taken my test, I received my results within 48 hours; a very fast service which I greatly appreciated.” - Mr. Muhammad Saleem, UK, August 2014

“People tend to be more biased for [other tests]. They tend to judge you sometimes. You don't get the same score twice, so that way I think a computer based test is more fair and more true to your capabilities.” – Test taker interviews, February 2014

“It’s easier for me as I use computers a lot and I am confident, instead of having an examiner in front of me and asking me questions, which makes me nervous.” - Mr. Muhammad Nazim Rasheed, UK, August 2014

Presentation Title runs here l 00/00/0020