Evaluating Task-Based Language Programs Colloquium – TBLT 2009

Evaluating Task-Based Language Programs

Colloquium – TBLT 2009

NOTE: This PowerPoint presentation has been modified by removal of all high-resolution graphics, to reduce the storage and downloading requirements (sorry, no pretty photos!).

The colloquium

Why bother with TBLT program evaluation?

Three presentations + clarification questions (2:00-3:30):

Re-framing the evaluation of task-based language education

Evaluating a TBLT Spanish immersion program

Evaluation of TBLT in Flanders

Open audience-panel discussion (3:30-3:50)

John M. Norris

University of Hawai‘i at Mānoa

TBLT 2009

Re-framing the evaluation of task-based language education

Please cite as:

Norris, J. M. (2009, September). Reframing the evaluation of task-based language education. Paper presented at the refereed colloquium “Evaluating task-based language programs”, at the 3rd International Conference on Task-Based Language Teaching, Lancaster, UK (September 14, 2009).

TBLL v. TBLT: Disconnects between inquiry and practice

What is task-based language learning (TBLL)?

1. Societal need for change in language education…

Value

Outcomes

Methods2. Emerging notions of L2 acquisition…

Processes

Impediments

Indicators


Proposals

Practices

Findings

Hypotheses

Observations

1. Societal need for change in language education…

Value

Outcomes

Methods2. Emerging notions of L2 acquisition…

Processes

Impediments

Indicators

Discussions

Opportunity for a

researched language pedagogy


Pedagogic principles, such as…

Promote learning by doing, experiential learningUse task as the unit of analysis for instruction &

assessmentProvide rich L2 input

Elaborate (rather than simplify) L2 input

Respect learner-internal syllabuses

Promote collaborative-cooperative interaction

Enable inductive/chunk learning

Provide focus on form, negative feedback

(e.g., Doughty & Long, 2003; Ellis, 2003)

Task-BasedLanguageTeaching

Cognitive Psychology

Philosophy of Education

Curriculum theory

SLA

Planning and policy

Learners

Assessment

Curriculum

Instruction

Teacher development

Materials

Rationales and Principles

L2 Education Programs

Sociocultural theory

Needs

Applied to

Inform???

What is task-based language teaching (TBLT)?

What is the role of task-based inquiry?


Task-BasedLanguageLearning

Generate theory

Discover robust, if small, truths

Test hypothese

s

Improve teaching practice

Inform curriculum,

course design

Understand what works,

where, when, &

why

HOLISTIC

GENERALIZABLE

SITUATED

DISCRETE




Generate theory


Test hypothese

s


Inform curriculum,

course design


where, when, &

why

HOLISTIC

GENERALIZABLE

SITUATED

DISCRETE




Generate theory


Test hypothese

s


Inform curriculum,

course design


where, when, &

why

HOLISTIC

GENERALIZABLE

SITUATED

DISCRETE

Challenges for task-based inquiry

1. The scope of task-based research does not match the scope of our claims about (for or against) TBLT.

2. The focus of theoretical task-based research does not relate to the situated realities of task-based teaching.

Needs

Curriculum

Materials

Instruction

Teachers

Learners

Assessment

TBLT Education Programs

Goals, outcomes

Scope, sequence

Resources

Practices

History, training

Individual differences

Intended uses, users

Framing TBLT inquiry through program evaluation

Research emphasizes theoretical, conclusion-oriented inquiry

Evaluation operationalizes decision-oriented inquiry

Inquiry through evaluation

Cronbach & Suppes (1969)

Evaluation is the gathering of information about any of the variety of elements that constitute educational programs, for a variety of purposes that include primarily understanding, demonstrating, improving, and judging program value; evaluation brings evidence to bear on the problems of programs, but the nature of that evidence is not restricted to one particular methodology.

Norris (2006) MLJ Perspectives

Inquiry frame and focus

Inquiry question

prioritization

Inquiry impetus


“The evaluator will be wise not to declare allegiance to either a quantitative-scientific-summative methodology or a qualitative-naturalistic-descriptive methodology.” (p. 7)

Cronbach et al. (1980) Paradigms

Epistemology 1

Methodology 1

Epistemology 2

Methodology 2



Cronbach et al. (1980) Paradigms

Epistemology 1

Methodology 1

Epistemology 2

Methodology 2



Cronbach et al. (1980) Pragmatism

Who?

Method 1

Why?

Method 2

What?

When?

Method 5

Method 4

Method 3


1. Participation – stakeholders, representatives, primary intended users

2. Prioritization – challenges, questions in immediate need of answers

3. Instrumentation – what data will answer the questions?

4. Collection – how can we get data in available time/resources?

5. Interpretation – what do findings mean in context?

6. Utilization – what decisions & actions are taken?

Language educators are ultimately responsible for what happens in

language education.

Participation by language educators is essential throughout evaluation if

contextual relevance is sought.

A focus on specific intended uses for evaluation findings is essential from the outset, if evaluation is to make

any difference.


Patton (1997) Utilization-focused

evaluation

Context: Intended

uses

Context: Intended

users

Questions + Methods

UnderstandImproveEducate

Demonstrate worthHold accountable

Empower (Test theory)

TeachersAdministrators

Curriculum writersLearners

Parents/publicFunders

(Researchers)Values clarificationImplementationProcess-product

Outcomes

EvaluatingTBLT

Programsin situ

Learner needs, institutional resources, program goals and outcomes, curriculum, materials, instruction, assessment, teachers, teacher

development, learners, etc.

Context: Program features

Learning from evaluation examples: the brief history of TBLT

Learning from TBLT evaluation

Evaluating the Communicational Teaching Project – Prabhu’s “Bangalore Project” (See Prabhu, 1987)

ContextEnglish L2

Education inBangalore,

India; Seeking

Improvementvia Innovation

L2 learningby processing

meaning; Unconscious

grammarconstructionby learners

Project/task-based work;

4 experimentalschools;

Implemented1979-1984

Theory Program


Evaluating the Communicational Teaching Project – Phase 1 (See Beretta & Davies, 1985)

“To assess, through appropriate tests, whether there is any demonstrable difference in terms of attainment in English between classes of children who have been taught on the CT project and their peers who have received normal instruction in the respective schools.”

Beretta & Davies (1985)

Initial inquiry, final year of the project:


Evaluating the Communicational Teaching Project – Phase 1 (See Beretta & Davies, 1985)

Purpose

Test theory

Demonstrate method effectiveness

Methods

Quasi-experimentation

Class/method comparison

Outcome achievement assessments

Findings

Structures test: Control > CTP

Contextual grammar: Control = CTP

Dictation: Control = CTP

List/Read comp: CTP > Control

Task-based test: CTP > Control

Claims???

Task-based learners achieved as much or more than traditional on all but the least functional outcomes

Task-based instruction is successful

Warranted claims???

What do we really know???


Evaluating the Communicational Teaching Project – Phase 2 (See Beretta, 1986, 1990, 1992)

Purpose

Understand program implementation

Illuminate relation with apparent outcomes

Methods

Retrospective interview protocols

Teacher level of concern questionnaires

Document analysis

Findings

Lack of comparability (intact classes, no baseline data)

More qualified teachers in CTP classes

Implementation of CTP highly variable (over time, between classes, with structures)

More confident teachers = better results


Evaluating the Communicational Teaching Project – Lessons Learned (See Beretta, 1992)

Theory testing, methods comparisons, “what works” claims are rarely feasible in real educational programs

Teachers (beliefs, training, commitment, time) play a key role in implementing programs: what they actually do must be understood

Even poorly executed evaluations (e.g., post-hoc) can shed light on how programs function and help explain why learning does or does not occur

Apparent differences in learning achievement, behaviors, etc. can only be explained by observation of multiple factors as they are experienced in real program contexts


Evaluating a university French curriculum – Student perspectives (See Towell & Tomlinson, 1999)

ContextFrench FL

Education, UKuniversity, Salford;

Restructuring advanced

FL teaching

Input, text,task;

Learning through form-

functionmapping in

tasks

Task-basedsyllabus;

Multiple levelsat university;Implemented

1988-96

Theory Program


“Curriculum design, evaluation, application and enhancement is a slow process, and subject to a number of extraneous influences which make it impossible to measure with totally scientific precision…use of diaries and questionnaires on the first occasion enabled a number of lessons to be learned and these helped considerably in creating a second application where the testimony of the student population through a detailed questionnaire shows the success of the operation.”

Towell & Tomlinson (1999)

Multiple iterations of development, implementation, evaluation, revision:



Program: Initial TBLT long group projects

Methods:

•Learner diaries

•Learner surveys

•Assessments/exams

Findings:

•Projects too long (6 wks)

•Training in group work

•Staged task objectives

•Gains in text/task learning

•Developing accuracy?

STAGE

1

Program: Revised TBLT staged, short projectsMethods:

•Learner surveys

•Focus groups

•Assessments/exams

Findings:

•Increased satisfaction

•Higher learning of skills

•Improved oral translation

•Written translation?

•Developing accuracy?

STAGE

2


Student/learner perspective on teaching with tasks sheds important light on the realities of implementation (how + how well)

Learners can change how they learn—acculturating to TBLT—especially when tasks, instructions, assessments are intentionally designed and staged to do so

Triangulated learner feedback (diaries, self-assessments, questionnaires, exams) can lead to effective improvements in curriculum and task design, and in turn to higher evaluations

Building evaluation activities into curricular delivery from the outset (e.g., student diaries), enables longitudinal insights about change, development, response to instruction



Evaluating TBLT for EAP – Developmental evaluation in Thailand (See McDonough & Chaikitmongkol, 2007)

Context

English FLeducation,

Thaiuniversity;

Improving EAPinstruction

Integrated-skills,communication;

Life-longlearning;

Learner needs + interestorientation

Task-basedsyllabus;English

department;Implemented

12 months

Theory Program


“…relatively few empirical studies have documented how teachers and learners react to entirely task-based courses, as opposed to the use of individual tasks…The purposes of this case study were (a) to identifyteacher and learner reactions to the course and (b) to describe how their concerns, if any, were addressed.”

McDonough & Chaikitmongkol (2007)

Inquiry for developing and improving TBLT experiences:



Methods:

•Learner task evaluations (repeated)

•Learning notebooks

•Class observations

•Student course evaluations

•Teacher/student interviews

•Observer field notes

Findings:

•Increased learner independence, language skills, learning strategies

•Decreased grammar obsession

•Non-specific real-world relevance

•Need time to adjust (T&L)

•More support, guidance from teachers

•Too much to cover, disparate materials

Uses:

•Intro unit on language learning

•Teacher’s guide to instruction + workshop

•Enhanced task guidelines, built-in feedback opportunities

•Reduced number of tasks

•Consolidated materials



Cycles of evaluation planned into TBLT innovation, and carried throughout, can lead to increased likelihood of effectiveness

TBLT based on learner needs can work well in EAP contexts, especially when evaluation is used to support on-going effectiveness of delivery from the outset

Teachers and learners both require support in implementing TBLT, especially during early phases of introducing task-based instruction

Systematic evaluation (a) from multiple stakeholder perspectives and (b) focused on multiple program elements (materials, preparedness, outcomes) enables balanced change


Dutch SLEducation inFlanders;

Nationwide; K-16;Ensuring

EducationalAccess, Equity

Large-scale Task-Based LT

Innovation;Improving Functional

DSL Abilities

School-basedTeacher-Training

Programs;EnablingChange,

1994-2003


Context Theory Program

Evaluating TBLT teacher training – Cyclical evaluation in Belgium (See Van den Branden, 2006)

Van den Branden (2006): “…the teacher tries to act as a true interactional partner, negotiating meaning and content with the students, eliciting and encouraging their output, focusing on form when appropriate and offering them a rich, relevant and communicative input” (p. 217).

Evaluation PROBLEMS Evaluation USES

Teacher cognition

Teacher action

Teaching context

•What do they theorize about TBLT?

•How do they learn about TBLT?

•Are they willing to change with syllabus?

•How do they adopt/adapt TBLT in practice?•What are the social constraints on T-Dev?

•How can T-Dev be optimized?

Understand teachers

Illuminate context

Improve T-dev program

Encourage teacher agency

Ensure teaching success

Enable TBLT learning

Demonstrate outcomes


Program: Theoretical inservice trainingMethods:

•Teacher survey

•Training observation

Findings:

•Transmission model

•Short term (3 hrs.)

•“Try that with my students”…Post-coursal depression!

STAGE

1

Program: TB training + syllabus supportMethods:

•Teacher logs, interviews, classroom observationsFindings:

+awareness of TBLT +student enthusiasm ?teacher adoption -teacher control -task complexity -groupwork

STAGE

2

Program: Training + coaching + agencyMethods:

•Coaching obs, classroom obs, coach/ teacher interviewsFindings:

+conscious decisions +TBLT adaptation +self-evaluation ?teacher control -transfer -groupwork

STAGE

3


Program: Sustained implementation of TBLT T-Dev with coaching, support

Methods:

•Pre-post student learning outcomes, teacher surveys, classroom observationsFindings:

+incorporation of TBLT correlated with higher Dutch L2 proficiency outcomes

+3-year gains in DSL higher in TBLT intensive adopting schools

?mixed incorporation of TBLT across schools, teachers

STAGE

4



Long-term evaluation of TBLT sheds light on how ideas are implemented, how participants change, and what support is needed

Teachers can learn to engage with TBLT, but change takes time, requires individualized support, and must be valued

Persistent follow-through on evaluation findings (use) underlies effective innovation

Multi-directional evaluation (political, social, school, individual) increases our capacity to explain why task-based ideas work or do not

Evaluating TBLT teacher training – Cyclical evaluation in Belgium (See Van den Branden, 2006)

What have we learned? Reframing evaluation in TBLTFrom summative to intentional

From assessment-driven to multi-methodologicalFrom external to participatory

From method-testing to program-illuminatingFrom one-shot to longitudinal, cyclical


From theoretical conclusions to educational decisions

Research, evaluation, and the future of task-based education

TBLL research

Sociocultural, cognitive, and other theories provide useful starting points for thinking about language teaching and learning, and offer principles for building educational programs

Task-based language learning research helps in that it raises our awareness about particular factors that we should pay attention to in the instructed L2 learning process

Task-based language learning research cannot tell us much about how or why language education programs work; findings from TBLL research should not be interpreted as direct implications for TBLT education

TBLT evaluation

Intentional

Evaluative

Inquiry

Answers questions & informs decisions of local interest

Sheds light on how TBLT ideas work in practice

Provides truths situated in rich contexts of programs

Relates outcomes to TBLT delivery and other factors

Focuses on scope that is meaningful to teachers, learners

Tests and informs innovation on the ground, in situ

Empowers participants to learn, and learn to change

Forces an honest accounting of TBLT

TBLT evaluation

Challenges

For TBLT

Evaluation

Resources: It takes time and money to do evaluation well and to sustain it within L2 educational programs.

Training: Effective evaluation calls upon skills that may not be easily available among personnel at hand.

Dissemination: There are few venues for publishing evaluation reports, thereby limiting learning.

Actual uses: There are many possible uses/needs for evaluation that we are not sufficiently attuned to, yet.

Cheers! (Mahalo!)

http://www2.hawaii.edu/~jnorris

References

Beretta, A. (1986). Program-fair language teaching evaluation. TESOL Quarterly, 20, 431-445.Beretta, A. (1990). Implementation of the Bangalore Project. Applied Linguistics, 11(4), 321-340.Beretta, A. (1992). What can be learned from the Bangalore evaluation? In J. C. Alderson and A. Beretta (eds.), Evaluating second language education (pp. 250-273). Cambridge: Cambridge University Press.Beretta, A., & Davies, A. (1985). Evaluation of the Bangalore Project. ELT Journal, 39(2), 121-127.Cronbach, L. J., & Associates. (1980). Toward reform of program evaluation. San Francisco: Jossey-Bass.Cronbach, L., & Suppes, (1969). Research for tomorrow's schools: Disciplined inquiry for education. New York: Macmillan, 1969.Doughty, C., & Long, M. H. (2003). Optimal psycholinguistic environments for distance foreign language learning. Language Learning & Technology, 7, 50-80.Ellis, R. (2003). Task-based language learning and teaching. New York: Oxford University Press.McDonough, K., & Chaikitmongkol, W. (2007). Teachers’ and learners’ reactions to a task-based EFL course in Thailand. TESOL Quarterly, 41(1), 107-132.Norris, J. M. (2006). The why (and how) of student learning outcomes assessment in college FL education. Modern Language Journal, 90(4), 590-597.Norris, J. M. (2009). Task-based teaching and testing. In M. Long and C. Doughty (Eds.), Handbook of language teaching (pp. 578-594). Cambridge: Blackwell.Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage.Prabhu, N. S. (1987). Second language pedagogy. Oxford University Press, Oxford.Towell, R., & Tomlinson, P. (1999). Language curriculum development research at university level. Language Teaching Research, 3(1), 1-32.Van den Branden, K. (2006). Training teachers: Task-based as well? In K. Van den Branden (ed.), Task-based language teaching in practice (pp. 217-273). Cambridge: Cambridge University Press.

Documents

Evaluating Task-Based Language Programs Colloquium – TBLT 2009