EVALUATING A TBLT SPANISH IMMERSION PROGRAM

COLLOQUIUM: Evaluating task-based language programs

Greg Burwell – BPA – greg.burwell@dhs.gov

Marta Gonzalez-Lloret –UHM – marta@hawaii.eduKatie Nielson – UMD (CASL) – knielson@casl.umd.edu

Overview

Background

Student assessment

Pilot study

Ordinate study

Student evaluation

Discussion

TBLT at the Border Patrol Academy (BPA)

BPA Training Environment Spanish at the BPA (before and after) Students, instructors, and supervisors

sought practical training Developed in 2006-2007

Needs Analysis Genuine audio/video Extensive input from actual Border Patrol

agents

TBLT at the Border Patrol Academy (BPA)

First implemented in December of 2007 First cohort of 15 students graduated in

March 2008 As of August 2009, 2,813 students have

completed the program To date, the program has started 180

classes with over 3,000 students 100% pass rate

TBLT at the BPA- Curriculum 8-week program following Long’s (1985, 2005a) stages

for designing and implementing a TBLT program

7 target tasks

Several target task

types

Multiple pedagogic tasks

TBLT at the BPA - CurriculumTarget Task Types:

Extract subject

Interview

subject

Determine

citizenship status

Inspect Vehicle

TBLT at the BPA - Curriculum

Input (video, audio)

Output (guided)

Role-play tasks

COMPLEXITY

+ Lab Activities + La Chamba interactive video game

TBLT at BPA- Assessment

•Computer-based language test •Performance-based assessment where trainees interact with native Spanish speaking role-players as they complete that module’s target task•One-on-one conference with instructor

Weekly formative

assessment:

•Critical scenarios with professional role-players•Students role-play key course tasks over two days•Criterion-referenced scoring•Each scenario scored by two instructors

End-of-course summative

assessment:

TBLT at BPA - Assessment

TBLT at BPA- Assessment Multiple versions of end-of-module

assessments Questions considered in each iteration:

Are these elements required to complete the task?

Which are critical? How do we assess language?

Global accuracy, fluency, and pronunciation ratings

Inclusion of “using appropriate Spanish” with each required task

Removal of any mention of language

TBLT at BPA - Assessment

TBLT at BPA - Assessment

TBLT at BPA - Assessment

TBLT at BPA - Assessment Instructor training

Video evaluation Role-play evaluation

Reliability Rubric modification

Perceived grade inflation Language versus task accomplishment Non-obligatory steps removed Average grade changed from 94.10% to

95.48%

ParticipantsGrammar-based group (GB) (n=20)TBLT group (TBLT) (n=20)

Oral narrative based on picture storyHypotheses:

• TBLT will perform better on measures of fluency, lexical complexity and structural complexity

• GB will perform better on measures of grammatical accuracy

Pilot Study – Oral narration task

Pilot study- cont Compared on:

Fluency [syllables/second] (Gilabert, 2004) Lexical complexity

[lexical words/words] (Robinson, 1995; Gilabert, 2004)

[1st time use of lexical word/words] Ratio of lexical/functional words (Gilabert, 2004)

Structural complexity [words/utterance] (Ortega, 1999)

Grammatical accuracy TLU of noun-modifier agreement (Ortega, 1999) TLU of noun-verb agreement

Pilot study- Results - Fluency

GB TBLT0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

1.60

1.80

2.00sy

llabl

es/

seco

nd

Pilot study- Results – Lexical complexity

Lexical complexity 1 Lexical complexity 2 Lexical complexity 30.00

0.20

0.40

0.60

0.80

1.00

1.20

GBTBLT

Lexical words/words

1st use lexical w/words Ratio of lexical/ functional words

Pilot study- Results- Structural complexity

GB TBLT0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

GBTBLT

#of w

ords

/ utte

ranc

e

Pilot study- Results –Grammatical Accuracy

Grammatical accuracy 1 -(S-N agrement) Grammatical accuracy 2 (S-V agreement)0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

GBTBLT

TLU of noun-modifier agreement TLU of noun-verb agreement

Pilot study - Discussion

Very small n-size due to difficulty

obtaining samples

Significant effect for fluency and

structural complexity (.05)

Students from TBLT course tend to produce more fluent and complex speech that is less grammatically accurate

but contains a greater variety of lexical items.

ParticipantsFirst 256 TBLT students

Performance on computerized oral proficiency interviewHypotheses:

• TBLT will improve students’ overall proficiency• TBLT will affect overall proficiency regardless of

language abilities at the start of the course

Ordinate study

Ordinate Study - continued Ordinate (Versant) Computerized Oral

Proficiency Exam Used at BPA as a placement test Scored between 20 – 80

Overall Sentence Mastery Vocabulary Fluency Pronunciation

Students took the test before and after the 8-week TBLT program

Ordinate Study - Mean Scores

Overal

l

Sente

nce M

astery

Vocab

ulary

Fluen

cy

Pronu

nciati

on0

1020304050

Pre-testPost-test

Ordinate Study – Change Scores

* Overall:Mean: 7.52SD: 5.42

* Sentence mastery:

Mean: 7.55SD: 8.21

* Vocabulary:Mean: 7.16 SD: 10.99

* Fluency:Mean: 5.86SD: 7.06

* Pronunciation:

Mean 3.53SD 4.92

All change scores

significant at .01

Ordinate Study - Correlation

Ordinate Study - Discussion

Hypothesis #1Overall proficiency will

improve as a result of TBLT

Yes: Post-test means improve an average of 7.5 points overall on an

unrelated, standardized proficiency test

Students’ evaluations

Electronic (anonymous) surveys 2 Groups:

Pre-experienced

Current students of the TBLT Spanish

program n=4

1

ExperiencedCurrent BA Agents trained with TBLT Spanish program

n=14

Students’ evaluation (pre-experienced)

In general students liked the best… The work done with the role-players. The direct application of the class to their job.

“Role playing is very helpful. I feel that it helped me learn more spanish (sic) and become more comfortable speaking it.”

“I enjoyed the hands on experience the most. Being able to read and write is important when learning a new language, but being able to incorporate it into your daily duties is beneficial.”

“I have enjoyed that we get to have a lot of hands on in this program. Also helps a lot for us to be able to apply all that we have learned in basic and build apon (sic) basic.”

Students’ evaluation

Students’ evaluations Students suggestions:

More conversational Spanish (not for the job)

More grammar in class More role-play work with Spanish speakers

Agents’ evaluation (experienced)

The Spanish program (topics, activities, lab) was interesting.

I completed most of the lab exercises.

The topics I studied are now useful in the field.

The vocabulary & expressions I studied are now useful in the field.

The class exercises are now useful in the field.

I apply most of what I learned in class in the field.

The lab exercises helped me learn Spanish.

The Spanish lessons gave me confidence to use Spanish in the filed.

The Spanish lessons reflect well what happens in the filed.

I played the video game La chamba at least twice a month.

The video game helped me learn important Spanish words & expressions.

0.00 0.50 1.00 1.50 2.00 2.50 3.00

Agents’ evaluations What they found most useful from the

Spanish course was: Commands and expressions Knowing the procedures and forms in

Spanish What they would have liked to learn:

More conversation (outside scenarios) More vocabulary Grammar and verb conjugation

TBLT at BPA

Students are all

passing

Overall student

proficiency improves after the

TBLT course

Student speech is

more fluent and complex

than in the old course

What does this

mean?

Students find the course

practical

•Difficulty of training instructors properly on methods and principles of TBLT•Instructors reluctant to adopt new approach•Lack of trained personnel to collect and analyze data•Lack of support for follow-up research

Difficulties with evaluating TBLT at the BPA

THANK YOUGRACIAS

MAHALO

Questions?