Meeting the Challenge: the Development of TETET-
Test of English for Thai Engineers and Technologists
Asst.Prof. Natjiree Jaturapitakkul, Ph.D.
School of Liberal Arts, KMUTT, Thailand
Outline of Presentation
• Rationale for TETET development
• Details of TETET
• Development of TETET
• Q&A
Rationale
To construct an English test that is
-suitable for KMUTT students
(engineering and technology fields)
-able to assess student’s actual ability in
communication in their fields before
graduation (not TOEIC)
- particularly for THAI engineers and
technologists
What is TETET?
A computer-based standardized test designed to
measure engineers and technologists’ English skills
needed in workplace environments in Thailand
Who should take TETET?
• TETET is ideal for:
– University students in
engineering and technology
faculties
– People working or wanting
to work in engineering and
technology positions
What does TETET involve?
• A computer-based test covering the 4 language skills
• Lasting 2.5 – 3 hours
• Engineering- oriented content but not biased towards any
particular discipline
• Covering the main work requirements of Thai engineers and
technologists
What is the content of TETET?
• Reading
– Survival reading, Internet, technical manuals,
• Writing
– Reports, memorandum, e-mail
• Listening
– Meetings, conversations, telephones
• Speaking
– Questions and answers, longer business communication
Reading in TETET
– Survival reading
• To measure basic reading abilities
– Internet
• To measure reading comprehension and evaluation
– Technical manuals
• To measure reading of instructions
• To measure reading of communications
Writing in TETET
• Memorandum
- To measure ability to use common phrases
• Reports
– To measure grammatical accuracy
– To measure ability to write extended texts
Listening in TETET
• Meetings
– To measure ability to identify both gist and specific meanings with
multiple speakers
• Conversations
– To measure the ability to draw inferences from conversations
• Telephones (AVRS)
– To measure the ability to understand specific instructions
Speaking in TETET
• Questions and answers
– To measure the ability to engage in interactional speaking
• Longer business communication
– To measure the ability to produce comprehensible longer
discourse
Score reporting on TETET
• Scores are determined by the number of correct
answers.
• TETET candidates receive 5 grades:
– A grade for overall proficiency
– A grade for proficiency in each skill
• Grades range from 0 (poor) to 7 (excellent)
Example of TETET Score Report
Listening 3 Has an adequate understanding of predictable listening situations such as general instructions, but comprehension is severely reduced in more complex situations.
Reading 3 Has an adequate understanding of simple texts for immediate survival needs, such as finding information in telephone directories, and is able to find specific facts from texts.
Writing 4 Copes well with familiar written situations by producing both short and long texts, and performs adequately in less predictable contexts such as writing a report where there may be some fluency and accuracy problems.
Speaking 2 Copes adequately with familiar speech situations, but speech may be slow, full of pauses and difficult to comprehend at times.
Overall Band 3 Has partial command of the language in simple workplace situations, but Score likely to make many mistakes, and has great problems in dealing with
complicated or unfamiliar situations.
How does TETET compare to
TOEIC and TOEFL?
• TETET bandscale
– 0
– 1
– 2
– 3
– 4
– 5
– 6
– 7
• TOEFL score
– 0-200
– 340
– 400
– 450
– 500
– 550
– 600
– 640-677
• TOEIC score
– 0-140
– 145-250
– 255-360
– 365-470
– 475-580
– 585-690
– 695-875
– 880-990
8 Steps in developing TETET
Defining the TLU domain & Outline of test
Constructing the table of TETET specifications
Developing the draft of the TETET
Going through a priori validation process by experts
Revising the test and trying out the test in
a pilot study (paper-based)
Doing item analysis and item revise
Obtaining a final version of TETET (CBT) + Band scores description
Conducting concurrent validation processes
Figure1: A summary flowchart of TETET development
Step 1: Defining the TLU Domain and Outline of Test
4 methods employed in defining TLU and test construct
Referring to purposes of the study
Studying theoretical background i.e. Strategies used in
different skills, Bloom’s Taxonomy, ESP theory
Studying previous research studies/ existing standardized
tests i.e. TOEIC, TOEFL, IELTS,
Doing need analysis by interviewing the stakeholders involved i.e. senior university administrators, human
resource managers, engineers, engineering (both under & graduate) students
(Interview questions concern experts’ expectations, test components, test content, test format, test taking time, test length and recommendations).
Step 3: Developing the draft of TETET (paper-based)
Concerns
1. Topic and text do not bias towards any disciplines
2. Level of text specificity (mild-semi level of ESP)
3. Copyright concern
4. Authentic texts/ materials
5. Text length
Step 4: Going through a priori validation process by experts
3 lecturers and 2 engineering experts validated content by
using an IOC Index form.
H, M and L were used to identify the degree to which the
item (question) measured the ability indicated in the objectives.
H = High degree of congruence with the objective
M = Moderate degree of congruence with the objective
L = Low degree of congruence with the objective
* Items which obtained the high degree of congruence with the objectives less than 75% were revised or discarded.
Step 5: Revising the test and trying out (a pilot study with
paper-based)
Data and comments from all experts were
used to modify the test and used for the pilot
study.
43 engineering undergraduate students at
KMUTT participated in the pilot study
(2 versions of TETET).
Step 6: Doing item analysis and item review
Some items were revised and discarded
according to the following criteria:
Item Difficulty index (IF) should be between 0.2-
0.8
Item Discrimination index (ID) should be higher
than 0.2.
Step 7: Obtaining a final version of TETET (CBT) + creating band scores description
See MS word – TETET Full descriptors
Step 8: Conducting concurrent validation process
Conducting a main study with 42 students who also took
TOEIC test.
TETET has a very strong correlation with TOEIC (r = 0.888).
TETET provides coverage of all aspects of English used by engineers and technologists.
TETET also has very high internal consistency (KR-21
scores of 0.915, higher than TOEIC).
There are strong correlations between the different
sections of TETET and between different versions of the test.
Correlations between different sections and versions of TETET
Reading Listening Writing Speaking
Total TETET
0.788 0.931 0.928 0.850
Speaking 0.474 0.753 0.719
Writing 0.677 0.849
Listening 0.643
Useful websites
More details about KMUTT Testing Unit
http://arts.kmutt.ac.th/crs/testing.htm
Preview of TETET
http://sola.kmutt.ac.th/tetet/index.php?q=vdo