Evaluating Task-Based Language Programs
Colloquium – TBLT 2009
NOTE: This PowerPoint presentation has been modified by removal of all high-resolution graphics, to reduce the storage and downloading requirements (sorry, no pretty photos!).
The colloquium
Why bother with TBLT program evaluation?
Three presentations + clarification questions (2:00-3:30):
Re-framing the evaluation of task-based language education
Evaluating a TBLT Spanish immersion program
Evaluation of TBLT in Flanders
Open audience-panel discussion (3:30-3:50)
John M. Norris
University of Hawai‘i at Mānoa
TBLT 2009
Re-framing the evaluation of task-based language education
Please cite as:
Norris, J. M. (2009, September). Reframing the evaluation of task-based language education. Paper presented at the refereed colloquium “Evaluating task-based language programs”, at the 3rd International Conference on Task-Based Language Teaching, Lancaster, UK (September 14, 2009).
TBLL v. TBLT: Disconnects between inquiry and practice
What is task-based language learning (TBLL)?
1. Societal need for change in language education…
Value
Outcomes
Methods2. Emerging notions of L2 acquisition…
Processes
Impediments
Indicators
What is task-based language learning (TBLL)?
Proposals
Practices
Findings
Hypotheses
Observations
1. Societal need for change in language education…
Value
Outcomes
Methods2. Emerging notions of L2 acquisition…
Processes
Impediments
Indicators
Discussions
Opportunity for a
researched language pedagogy
What is task-based language learning (TBLL)?
Pedagogic principles, such as…
Promote learning by doing, experiential learningUse task as the unit of analysis for instruction &
assessmentProvide rich L2 input
Elaborate (rather than simplify) L2 input
Respect learner-internal syllabuses
Promote collaborative-cooperative interaction
Enable inductive/chunk learning
Provide focus on form, negative feedback
(e.g., Doughty & Long, 2003; Ellis, 2003)
Task-BasedLanguageTeaching
Cognitive Psychology
Philosophy of Education
Curriculum theory
SLA
Planning and policy
Learners
Assessment
Curriculum
Instruction
Teacher development
Materials
Rationales and Principles
L2 Education Programs
Sociocultural theory
Needs
Applied to
Inform???
What is task-based language teaching (TBLT)?
What is the role of task-based inquiry?
Task-BasedLanguageTeaching
Task-BasedLanguageLearning
Generate theory
Discover robust, if small, truths
Test hypothese
s
Improve teaching practice
Inform curriculum,
course design
Understand what works,
where, when, &
why
HOLISTIC
GENERALIZABLE
SITUATED
DISCRETE
What is the role of task-based inquiry?
Task-BasedLanguageTeaching
Task-BasedLanguageLearning
Generate theory
Discover robust, if small, truths
Test hypothese
s
Improve teaching practice
Inform curriculum,
course design
Understand what works,
where, when, &
why
HOLISTIC
GENERALIZABLE
SITUATED
DISCRETE
What is the role of task-based inquiry?
Task-BasedLanguageTeaching
Task-BasedLanguageLearning
Generate theory
Discover robust, if small, truths
Test hypothese
s
Improve teaching practice
Inform curriculum,
course design
Understand what works,
where, when, &
why
HOLISTIC
GENERALIZABLE
SITUATED
DISCRETE
Challenges for task-based inquiry
1. The scope of task-based research does not match the scope of our claims about (for or against) TBLT.
2. The focus of theoretical task-based research does not relate to the situated realities of task-based teaching.
Needs
Curriculum
Materials
Instruction
Teachers
Learners
Assessment
TBLT Education Programs
Goals, outcomes
Scope, sequence
Resources
Practices
History, training
Individual differences
Intended uses, users
Framing TBLT inquiry through program evaluation
Research emphasizes theoretical, conclusion-oriented inquiry
Evaluation operationalizes decision-oriented inquiry
Inquiry through evaluation
Cronbach & Suppes (1969)
Evaluation is the gathering of information about any of the variety of elements that constitute educational programs, for a variety of purposes that include primarily understanding, demonstrating, improving, and judging program value; evaluation brings evidence to bear on the problems of programs, but the nature of that evidence is not restricted to one particular methodology.
Norris (2006) MLJ Perspectives
Inquiry frame and focus
Inquiry question
prioritization
Inquiry impetus
Inquiry through evaluation
“The evaluator will be wise not to declare allegiance to either a quantitative-scientific-summative methodology or a qualitative-naturalistic-descriptive methodology.” (p. 7)
Cronbach et al. (1980) Paradigms
Epistemology 1
Methodology 1
Epistemology 2
Methodology 2
Inquiry through evaluation
“The evaluator will be wise not to declare allegiance to either a quantitative-scientific-summative methodology or a qualitative-naturalistic-descriptive methodology.” (p. 7)
Cronbach et al. (1980) Paradigms
Epistemology 1
Methodology 1
Epistemology 2
Methodology 2
Inquiry through evaluation
“The evaluator will be wise not to declare allegiance to either a quantitative-scientific-summative methodology or a qualitative-naturalistic-descriptive methodology.” (p. 7)
Cronbach et al. (1980) Pragmatism
Who?
Method 1
Why?
Method 2
What?
When?
Method 5
Method 4
Method 3
Inquiry through evaluation
1. Participation – stakeholders, representatives, primary intended users
2. Prioritization – challenges, questions in immediate need of answers
3. Instrumentation – what data will answer the questions?
4. Collection – how can we get data in available time/resources?
5. Interpretation – what do findings mean in context?
6. Utilization – what decisions & actions are taken?
Language educators are ultimately responsible for what happens in
language education.
Participation by language educators is essential throughout evaluation if
contextual relevance is sought.
A focus on specific intended uses for evaluation findings is essential from the outset, if evaluation is to make
any difference.
Inquiry through evaluation
Patton (1997) Utilization-focused
evaluation
Context: Intended
uses
Context: Intended
users
Questions + Methods
UnderstandImproveEducate
Demonstrate worthHold accountable
Empower (Test theory)
TeachersAdministrators
Curriculum writersLearners
Parents/publicFunders
(Researchers)Values clarificationImplementationProcess-product
Outcomes
EvaluatingTBLT
Programsin situ
Learner needs, institutional resources, program goals and outcomes, curriculum, materials, instruction, assessment, teachers, teacher
development, learners, etc.
Context: Program features
Learning from evaluation examples: the brief history of TBLT
Learning from TBLT evaluation
Evaluating the Communicational Teaching Project – Prabhu’s “Bangalore Project” (See Prabhu, 1987)
ContextEnglish L2
Education inBangalore,
India; Seeking
Improvementvia Innovation
L2 learningby processing
meaning; Unconscious
grammarconstructionby learners
Project/task-based work;
4 experimentalschools;
Implemented1979-1984
Theory Program
Learning from TBLT evaluation
Evaluating the Communicational Teaching Project – Phase 1 (See Beretta & Davies, 1985)
“To assess, through appropriate tests, whether there is any demonstrable difference in terms of attainment in English between classes of children who have been taught on the CT project and their peers who have received normal instruction in the respective schools.”
Beretta & Davies (1985)
Initial inquiry, final year of the project:
Learning from TBLT evaluation
Evaluating the Communicational Teaching Project – Phase 1 (See Beretta & Davies, 1985)
Purpose
Test theory
Demonstrate method effectiveness
Methods
Quasi-experimentation
Class/method comparison
Outcome achievement assessments
Findings
Structures test: Control > CTP
Contextual grammar: Control = CTP
Dictation: Control = CTP
List/Read comp: CTP > Control
Task-based test: CTP > Control
Claims???
Task-based learners achieved as much or more than traditional on all but the least functional outcomes
Task-based instruction is successful
Warranted claims???
What do we really know???
Learning from TBLT evaluation
Evaluating the Communicational Teaching Project – Phase 2 (See Beretta, 1986, 1990, 1992)
Purpose
Understand program implementation
Illuminate relation with apparent outcomes
Methods
Retrospective interview protocols
Teacher level of concern questionnaires
Document analysis
Findings
Lack of comparability (intact classes, no baseline data)
More qualified teachers in CTP classes
Implementation of CTP highly variable (over time, between classes, with structures)
More confident teachers = better results
Learning from TBLT evaluation
Evaluating the Communicational Teaching Project – Lessons Learned (See Beretta, 1992)
Theory testing, methods comparisons, “what works” claims are rarely feasible in real educational programs
Teachers (beliefs, training, commitment, time) play a key role in implementing programs: what they actually do must be understood
Even poorly executed evaluations (e.g., post-hoc) can shed light on how programs function and help explain why learning does or does not occur
Apparent differences in learning achievement, behaviors, etc. can only be explained by observation of multiple factors as they are experienced in real program contexts
Learning from TBLT evaluation
Evaluating a university French curriculum – Student perspectives (See Towell & Tomlinson, 1999)
ContextFrench FL
Education, UKuniversity, Salford;
Restructuring advanced
FL teaching
Input, text,task;
Learning through form-
functionmapping in
tasks
Task-basedsyllabus;
Multiple levelsat university;Implemented
1988-96
Theory Program
Learning from TBLT evaluation
“Curriculum design, evaluation, application and enhancement is a slow process, and subject to a number of extraneous influences which make it impossible to measure with totally scientific precision…use of diaries and questionnaires on the first occasion enabled a number of lessons to be learned and these helped considerably in creating a second application where the testimony of the student population through a detailed questionnaire shows the success of the operation.”
Towell & Tomlinson (1999)
Multiple iterations of development, implementation, evaluation, revision:
Evaluating a university French curriculum – Student perspectives (See Towell & Tomlinson, 1999)
Learning from TBLT evaluation
Program: Initial TBLT long group projects
Methods:
•Learner diaries
•Learner surveys
•Assessments/exams
Findings:
•Projects too long (6 wks)
•Training in group work
•Staged task objectives
•Gains in text/task learning
•Developing accuracy?
STAGE
1
Program: Revised TBLT staged, short projectsMethods:
•Learner surveys
•Focus groups
•Assessments/exams
Findings:
•Increased satisfaction
•Higher learning of skills
•Improved oral translation
•Written translation?
•Developing accuracy?
STAGE
2
Learning from TBLT evaluation
Student/learner perspective on teaching with tasks sheds important light on the realities of implementation (how + how well)
Learners can change how they learn—acculturating to TBLT—especially when tasks, instructions, assessments are intentionally designed and staged to do so
Triangulated learner feedback (diaries, self-assessments, questionnaires, exams) can lead to effective improvements in curriculum and task design, and in turn to higher evaluations
Building evaluation activities into curricular delivery from the outset (e.g., student diaries), enables longitudinal insights about change, development, response to instruction
Evaluating a university French curriculum – Student perspectives (See Towell & Tomlinson, 1999)
Learning from TBLT evaluation
Evaluating TBLT for EAP – Developmental evaluation in Thailand (See McDonough & Chaikitmongkol, 2007)
Context
English FLeducation,
Thaiuniversity;
Improving EAPinstruction
Integrated-skills,communication;
Life-longlearning;
Learner needs + interestorientation
Task-basedsyllabus;English
department;Implemented
12 months
Theory Program
Learning from TBLT evaluation
“…relatively few empirical studies have documented how teachers and learners react to entirely task-based courses, as opposed to the use of individual tasks…The purposes of this case study were (a) to identifyteacher and learner reactions to the course and (b) to describe how their concerns, if any, were addressed.”
McDonough & Chaikitmongkol (2007)
Inquiry for developing and improving TBLT experiences:
Evaluating TBLT for EAP – Developmental evaluation in Thailand (See McDonough & Chaikitmongkol, 2007)
Learning from TBLT evaluation
Methods:
•Learner task evaluations (repeated)
•Learning notebooks
•Class observations
•Student course evaluations
•Teacher/student interviews
•Observer field notes
Findings:
•Increased learner independence, language skills, learning strategies
•Decreased grammar obsession
•Non-specific real-world relevance
•Need time to adjust (T&L)
•More support, guidance from teachers
•Too much to cover, disparate materials
Uses:
•Intro unit on language learning
•Teacher’s guide to instruction + workshop
•Enhanced task guidelines, built-in feedback opportunities
•Reduced number of tasks
•Consolidated materials
Evaluating TBLT for EAP – Developmental evaluation in Thailand (See McDonough & Chaikitmongkol, 2007)
Learning from TBLT evaluation
Cycles of evaluation planned into TBLT innovation, and carried throughout, can lead to increased likelihood of effectiveness
TBLT based on learner needs can work well in EAP contexts, especially when evaluation is used to support on-going effectiveness of delivery from the outset
Teachers and learners both require support in implementing TBLT, especially during early phases of introducing task-based instruction
Systematic evaluation (a) from multiple stakeholder perspectives and (b) focused on multiple program elements (materials, preparedness, outcomes) enables balanced change
Evaluating TBLT for EAP – Developmental evaluation in Thailand (See McDonough & Chaikitmongkol, 2007)
Dutch SLEducation inFlanders;
Nationwide; K-16;Ensuring
EducationalAccess, Equity
Large-scale Task-Based LT
Innovation;Improving Functional
DSL Abilities
School-basedTeacher-Training
Programs;EnablingChange,
1994-2003
Learning from TBLT evaluation
Context Theory Program
Evaluating TBLT teacher training – Cyclical evaluation in Belgium (See Van den Branden, 2006)
Van den Branden (2006): “…the teacher tries to act as a true interactional partner, negotiating meaning and content with the students, eliciting and encouraging their output, focusing on form when appropriate and offering them a rich, relevant and communicative input” (p. 217).
Evaluation PROBLEMS Evaluation USES
Teacher cognition
Teacher action
Teaching context
•What do they theorize about TBLT?
•How do they learn about TBLT?
•Are they willing to change with syllabus?
•How do they adopt/adapt TBLT in practice?•What are the social constraints on T-Dev?
•How can T-Dev be optimized?
Understand teachers
Illuminate context
Improve T-dev program
Encourage teacher agency
Ensure teaching success
Enable TBLT learning
Demonstrate outcomes
Learning from TBLT evaluation
Program: Theoretical inservice trainingMethods:
•Teacher survey
•Training observation
Findings:
•Transmission model
•Short term (3 hrs.)
•“Try that with my students”…Post-coursal depression!
STAGE
1
Program: TB training + syllabus supportMethods:
•Teacher logs, interviews, classroom observationsFindings:
+awareness of TBLT +student enthusiasm ?teacher adoption -teacher control -task complexity -groupwork
STAGE
2
Program: Training + coaching + agencyMethods:
•Coaching obs, classroom obs, coach/ teacher interviewsFindings:
+conscious decisions +TBLT adaptation +self-evaluation ?teacher control -transfer -groupwork
STAGE
3
Learning from TBLT evaluation
Program: Sustained implementation of TBLT T-Dev with coaching, support
Methods:
•Pre-post student learning outcomes, teacher surveys, classroom observationsFindings:
+incorporation of TBLT correlated with higher Dutch L2 proficiency outcomes
+3-year gains in DSL higher in TBLT intensive adopting schools
?mixed incorporation of TBLT across schools, teachers
STAGE
4
Learning from TBLT evaluation
Learning from TBLT evaluation
Long-term evaluation of TBLT sheds light on how ideas are implemented, how participants change, and what support is needed
Teachers can learn to engage with TBLT, but change takes time, requires individualized support, and must be valued
Persistent follow-through on evaluation findings (use) underlies effective innovation
Multi-directional evaluation (political, social, school, individual) increases our capacity to explain why task-based ideas work or do not
Evaluating TBLT teacher training – Cyclical evaluation in Belgium (See Van den Branden, 2006)
What have we learned? Reframing evaluation in TBLTFrom summative to intentional
From assessment-driven to multi-methodologicalFrom external to participatory
From method-testing to program-illuminatingFrom one-shot to longitudinal, cyclical
Learning from TBLT evaluation
From theoretical conclusions to educational decisions
Research, evaluation, and the future of task-based education
TBLL research
Sociocultural, cognitive, and other theories provide useful starting points for thinking about language teaching and learning, and offer principles for building educational programs
Task-based language learning research helps in that it raises our awareness about particular factors that we should pay attention to in the instructed L2 learning process
Task-based language learning research cannot tell us much about how or why language education programs work; findings from TBLL research should not be interpreted as direct implications for TBLT education
TBLT evaluation
Intentional
Evaluative
Inquiry
Answers questions & informs decisions of local interest
Sheds light on how TBLT ideas work in practice
Provides truths situated in rich contexts of programs
Relates outcomes to TBLT delivery and other factors
Focuses on scope that is meaningful to teachers, learners
Tests and informs innovation on the ground, in situ
Empowers participants to learn, and learn to change
Forces an honest accounting of TBLT
TBLT evaluation
Challenges
For TBLT
Evaluation
Resources: It takes time and money to do evaluation well and to sustain it within L2 educational programs.
Training: Effective evaluation calls upon skills that may not be easily available among personnel at hand.
Dissemination: There are few venues for publishing evaluation reports, thereby limiting learning.
Actual uses: There are many possible uses/needs for evaluation that we are not sufficiently attuned to, yet.
Cheers! (Mahalo!)
http://www2.hawaii.edu/~jnorris
References
Beretta, A. (1986). Program-fair language teaching evaluation. TESOL Quarterly, 20, 431-445.Beretta, A. (1990). Implementation of the Bangalore Project. Applied Linguistics, 11(4), 321-340.Beretta, A. (1992). What can be learned from the Bangalore evaluation? In J. C. Alderson and A. Beretta (eds.), Evaluating second language education (pp. 250-273). Cambridge: Cambridge University Press.Beretta, A., & Davies, A. (1985). Evaluation of the Bangalore Project. ELT Journal, 39(2), 121-127.Cronbach, L. J., & Associates. (1980). Toward reform of program evaluation. San Francisco: Jossey-Bass.Cronbach, L., & Suppes, (1969). Research for tomorrow's schools: Disciplined inquiry for education. New York: Macmillan, 1969.Doughty, C., & Long, M. H. (2003). Optimal psycholinguistic environments for distance foreign language learning. Language Learning & Technology, 7, 50-80.Ellis, R. (2003). Task-based language learning and teaching. New York: Oxford University Press.McDonough, K., & Chaikitmongkol, W. (2007). Teachers’ and learners’ reactions to a task-based EFL course in Thailand. TESOL Quarterly, 41(1), 107-132.Norris, J. M. (2006). The why (and how) of student learning outcomes assessment in college FL education. Modern Language Journal, 90(4), 590-597.Norris, J. M. (2009). Task-based teaching and testing. In M. Long and C. Doughty (Eds.), Handbook of language teaching (pp. 578-594). Cambridge: Blackwell.Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage.Prabhu, N. S. (1987). Second language pedagogy. Oxford University Press, Oxford.Towell, R., & Tomlinson, P. (1999). Language curriculum development research at university level. Language Teaching Research, 3(1), 1-32.Van den Branden, K. (2006). Training teachers: Task-based as well? In K. Van den Branden (ed.), Task-based language teaching in practice (pp. 217-273). Cambridge: Cambridge University Press.