Tblt Presentation

John M. Norris

University of Hawai´i

Task-Based Language Teaching 2005

This Powerpoint presentation is copyrighted by John M. Norris

© September 23, 2005

Citation information:Norris, J. M. (2005, September). The essential roles of assessment, measurement, and evaluation in task-based language teaching. Plenary address presented at the First International Conference on Task-Based Language Teaching, University of Leuven, Belgium (September 23, 2005).

0:00

9:00 3:00

6:00

12:00

21:00 15:00

18:00

What time is it in Leuven?

What time is it in Hawaii?

“Here and Now” condition

“There and… ” condition

Now

0:00

9:00 3:00

6:00

12:00

21:00 15:00

18:00

What is the time difference between Leuven & Hawaii?

Leuven

- Hawaii

= 12hrs

Task condition for presentation =

“Here _and There_ and Now”

measurement,

The essential roles of

assessment,

and evaluation

in task-based language teaching

TBLT 2005: What’s at stake?

???LanguageEducation

???

Does it work? for

whom, when, why?

What’s it worth? to whom, when,

why?

How does it work? for whom,

when, why?

Can it be improved? by whom, when,

why?

TBLT 2005: What’s at stake?

…another L2 teaching method bandwagon…

The critics say…

…cognitive-interactionist SLA hegemony…

…won’t work with _____ learners…

…not possible in _____ program settings…

…too much task, not enough language…

…not happening in practice…

TBLT

Setting the record straight:TBLT at work

TARGET LANGUAGE

S

EnglishDutch

French

Czech

Chinese

Arabic

Ukranian

Japanese

Russian

Spanish

Korean

German

Setting the record straight:TBLT at work

PROGRAM TYPES

SecondLanguage

ForeignLanguage

VocationalEducation

GeneralEducation

LocalInnovation

SmallClasses

NationalEd. Policy

SpecialPurposes

Adult

Child

UniversityRequirement

LargeClasses

BeginningLearners

AdvancedLearners

Face-to-Face

Online

Setting the record straight:TBLT foundations

“All genuine education terminates in discipline, but it proceeds by engaging the mind in activities worthwhile for their own sake.”

Dewey (1933), pp. 86-87


“This discussion rejects the doctrine that students should first learn passively, and then, having learned, should apply knowledge. […] In fact, the applications are part of the knowledge. For the very meaning of things known is wrapped up in their relationships beyond themselves. Thus, unapplied knowledge is knowledge shorn of its meaning.”

Whitehead, 1947, pp. 218-

219


“For starters, this means having students do tasks, or at least meaningful simulations, that experts do in the various disciplines. Second, it means teaching them to think in ways that experts do when they perform these tasks.” Sternberg

(2003), p. 5

Setting the record straight:Other task-based domains

Medical school

Nursing

Curriculum by design

Successful intelligence

Environmental studies

Social workTask-Based

Practice

Music education

Setting the record straight:So what is TBLT?

Task-BasedLanguageTeaching

Rationales and Principles

Education

Philosophy

Psychology SLA

Language teaching

Sociopolitical values

L2 Education ProgramsLearner

needs

Assessment

CurriculumInstructio

n

Teacher development

Evaluation

Materials

The potential of task-based practice:One example

William Reid

Reid. W. J. (1992). Task strategies: An empirical approach to social work practice. New York: Columbia University Press.Social work problems: truancy, aging, delinquency, substance abuse, family health

Task-CenteredPractice

Empirical Justification

Empirical Implementation

Theory Learning, behavior,

cognition

ResearchPsychosocial change,

Problem solving

EvaluationCase work practice,

Intervention outcomes

Needs Identify, agree,

prioritize

Action Plan, rehearse,

attempt, feedback

Assessment Task accomplishment,

Problem change

Development

Practitioner-friendly methods

Improvement

Field-tested

findings



“Assessment, process, and outcome data are systematically collected in each case by means of structured recording guidelines. Continuing research is used to improve the model.”

(Reid, 1992, p. 132).

Development

Improvement

Field-testing

“[Reid] was the single most significant figure in social work research”

Shaw (2004) p. 113


Task-Based Language Teaching:

A(nother) method for

language teaching.

The potential of task-based practice:TBLT


A(nother) method for

language education practice.



An empirical approach to

language education practice.


AssessmentMeasurement

Evaluation

The Role of Measurement in TBLT

Interpreting research constructs…

…to understand task-based learning

Measurement in TBLT:Responding to research purposes

What is measurement?

…trustworthy definition, elicitation, observation, analysis, and

interpretation of research constructs…


Measurement should… Illuminate task-based learning…L2 dvlpmt. in task, over tasks

Noticing, integrating, automatizing

Proficiency, motivation, aptitude

Timing, sequencing, embedding TBI

FonF, modeling, feedback

Reliable reporting of TBL variables

Synthesis of how TBL works

•Define clear learning outcomes

•Identify specific learning processes

•Clarify learner individual differences

•Indicate effects of context features

•Articulate instructional processes

•Enable communication of findings

•Foster accumulation of knowledge


“Perhaps the best recommendation would be that, with research in this area still developing techniques and understandings, the role of A/L work will be to offer suggestions for relevant variables and the standardization of operationalizations of variables…”

Tavakoli & Skehan, 2005, p. 273

In short…

TBLT

Theory/research

•Constructs

•Learner variables

•Setting variables

•Learning process

•Instruction process

•Learning outcomes

Practice

Empirical

Measurement


How does TBLT work, for whom, when, why?

Measurement in TBLT:Problems of practice

Interpreting research constructsNo construct definition

Little standardization

Missing measures

No attention to reliability

Impoverished analyses

Insufficient reportingUnderstanding task-based

learning

1. Inaccurate interpretation oftask-related variables

2. Massive over-generalizationof measurement ‘findings’

3. Minimal meaning for scope, scale, setting of

task-based practice

Task characteristic

Accuracy Complexity

Fluency

Familiarity + 0 +

Dialogic v. Monologic

+ + -

Structure + 0 +

Outcomes 0 + 0

Transformations (planning)

0 + 0

Task influence on L2 performance

(adapted from: Skehan, 2001)


Task characteristic

Accuracy Complexity

Fluency

Familiarity + 0 +

Dialogic v. Monologic

+ + -

Structure + 0 +

Outcomes 0 + 0

Transformations (planning)

0 + 0

Task influence on L2 performance

(adapted from: Skehan, 2001)


“…giving learners the opportunity to plan before a task is done consistently produces greater complexity of language…”

(Skehan, 2003, pp. 395-96)


“To sum up, both modality and group condition affected complexity at the level of syntax…”

(Ellis & Yuan, 2005, p. 184)


Task conditionPlanning

L2 performance outcomeComplexification

Modality?writing/speaking

Complexity?Subordination Complexity?

Other measures

Scope of measure?duration/timing

Planning?Focus, structure, pressure

Learner proficiency?“L-H”, “N-I-A”, “450-600”

Task features?

Type, difficulty, form, mono-

dialogic, interlocutors

?


Task condition+ strategic planning


Complexity?Clauses per C-Unit

Learner proficiency?TOEFL: 427-670

Elder & Iwashita (2005)

XX

X


Task condition+ strategic planning


Complexity?Words/T-Unit

Clauses per T-UnitTotal Subordination

Learner proficiency?TOEFL: 420-480TOEFL: 510-580TOEFL: 560-610

Kawauichi (2005)

Measurement in TBLT:An empirical approach to practice

In what ways are what kinds of planning associated with what kinds of L2 performance for what kinds of L2 users under what conditions…and why?


“...descriptive longitudinal studies which establish norms of performance for particular processes and phenomena in L2 acquisition…”

Norris & Ortega, 2003, p. 738

“…attempting to ‘measure’ acquisition without a sound descriptive basis for meaningful differences in particular acquisition-related behaviors would be akin to timing a runner’s performance over a mile without knowing how many times around the track a mile happens to be…”


Georgetown University German Department:

“Syntactic complexity development in task-based writing”

Longitudinal (23) + Cross-sectional (329)

Extended and extemporaneouswriting tasks

Multiple measures, includingMultiple measures of SC

End-of-curricular levelsAcross 4 years

Syntactic complexity in curriculum-embedded writing assessment: LONGITUDINAL

Global compexification

Phrasal elaboration

Subordination

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

18.0

MLTU

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

CTU

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

10.0

MLC

Syntactic complexity in curriculum-embedded writing assessment: CROSS-SECTIONAL

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

18.0

MLTU

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2.0

CTU

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

10.0

MLC

Global compexification

Subordination

Phrasal elaboration


Implications for measuring syntactic

complexity in task-based research

1. Syntactic complexity is multi-faceted; so, measures are multivariate

2. Complexification is a long-term developmental process, best understood in situ

3. At different learner proficiency/curricular levels, complexification occurs in distinct ways

Synthesis of research findings

on syntactic complexity

See: Ortega (2000, 2003)

In short…

TBLTin situ

Theory/research

•Constructs

•Learner variables

•Setting variables

•Learning process

•Instruction process

•Learning outcomes

Practice

Empirical

Measurement


How does TBLT work, for whom, when, why?

The Role of Assessment in TBLT

Informing decisions and actions…

…to enable task-based learning

Assessment in TBLT:Responding to educative purposes

What is assessment?

…the use of tests and other procedures for gathering trustworthy information about learners and making decisions

and taking actions on the basis of that information, in support of teaching and

learning…


•Ensure learner-relevant instruction

•Support classroom teaching

•Motivate and focus learners

•Target clear learning outcomes

•Identify learning gaps

•Transform curriculum

•Counter imposed testing practice

•Foster range of learning outcomes

•Certify valued accomplishments

Educative assessment should… Enable task-based learning…Needs analysis, intake procedures

Feedback on TB pedagogy effects

Interesting, meaningful tests

Genre, task performance expectation

Feedback on task performance

Process-product articulation

Internal TBA v. external mandate

Language + content + task abilities

Profile individual abilities in L2 use


“In other words, by focusing on task-based assessment—here writing—faculty faced their own assumptions about long-term development in instructed L2 learning and appropriate learning outcomes at particular stages in that sequence.”

Byrnes (2002), p. 428


“…the quality of task completion gave teachers a good understanding of what students did and did not know—often, a better understanding than most tests could provide.”

Leaver & Kaplan (2004), p. 52


“Focusing on what one can do instead of the competencies one has yields a more direct link with language performance that can be observed in the real world.”

Gysen & van Avermaet (2005), p. 51


In short…

TBLT

Implementation

•Curriculum

•Instruction

•Materials

•Learners

•Teachers

•Outcomes

•ETC.

Practice

Empirical

Assessment

Does TBLT work, and how can we enable it?

Assessment in TBLT:Problems of practice

Tensions in “how to” assess…L2 competence v. task ability

Extrapolation v. prediction

Construct v. consequence

Control v. understand

Prescribe v. respond

Enabling task-based learning

1. Irrelevant for many educative uses in classrooms/programs

2. Ignorant of interconnectionsbetween language and task/genre/content

3. Irresponsible to alignment ofassessment with C&I

4. Incompatible with values and goals of task-based education

Dichotomy v. multiplicity


“…the inferences we want to make are about underlying ‘language ability’ or ‘capacity for language use’ or ‘ability for use’…”

Bachman (2002), p. 454

“… assessments can have many different functions. What is appropriate for one assessment purpose may be inappropriate for another…”

Council of Europe (2001), p. 180

Advanced intercultural evaluation: An online

French course

Starkey & Osler (2001)

Common European Framework: Language+

Pluriculturalism, Anti-racism

Online advanced French learning objectives:

•Language development•Social/cultural awareness•Intercultural evaluation

Online advanced French teacher assessments (writing feedback):

•Language development•Social/cultural awareness•Intercultural evaluation



“… is not for the test designer to decide or prescribe. Rather, it should be the test users and stakeholders who inform exactly what needs to be done in the name of language assessment, and it is up to us to enable them to do so.”

Norris (2002), p. 344

“1. The intended use(s) for task-based assessment within the language programme must be specified…”

Long & Norris (2000), p. 600

Assessment in TBLT:An empirical approach to practice

Specifying intended assessment use

From: Norris (2000)

Assessment in TBLT:Examples in practice, Specification process

Specifying intended uses for assessment in the Georgetown University German Department

Clarifying the purposes for assessment, from classroom- to program-level, by…

Investigating existing assessment practices (via interviews, surveys, observations)

Establishing consensus on the roles to be played by assessment (via meetings, workshops, negotiation among intended users/stakeholders)

Assessment in TBLT:Examples in practice, Specification products

Intended assessment use specificationAssessment

policiesUse specsfor all tests

Priorities forDevelopment

AssessmentConstraints

Quizzes

Exams

Performance

Internal

External

Time

Money

Ethics

Training

Unknowns

(Urgent!)

Placement

Learning outcomes

External Accountability

Developing

Implementing

Articulating

Communicating

Evaluating

Assessment in TBLT:Examples in practice, Use specification

Intended useWho: Teachers, curriculum

developers, researchersWhat: Learner written

task/genre performance abilities at the end of each curricular level

Why: Understanding student achievement of targeted task abilities for improvement of C&I

Impact: Feasible (revised?) curricular expectations supported by effective (improved?) pedagogy

Constraints Explicitness of curricular

expectations Availability/agreement of

‘prototypical’ performance tasks by curricular level

Competing uses for assessment (feedback)

End-of-semester timing Learner investment,

understanding of task expectations

Task versus content

Task-based writing assessment

Task

Content

L conventions

L focus

Assignment 1Assignment 2

Process writing4-6 tasks persemester

Assignment 3Assignment 4

PrototypicalPerformance

WritingTask:Level ILevel IILevel IIILevel IV

Curricular level/genre learning expectations

Feedback loop

Explicit criteria

Assessment in TBLT:Examples in practice, Instruments

Assessment in TBLT:Examples in practice, Impact

1. Curriculum: much greater specificity in learning objectives and articulation, overall and for each level/semester

2. Courses: revised/improved pedagogic focus, consistency

3. Teachers: greater awareness of learner abilities and learning trajectories, enhanced clarity of grading/feedback purpose

4. Learners: enhanced understanding of learning targets & expectations, supportive role of assessment

5. Assessment: clearer articulation with C&I, development of similar assessment systems (e.g., speaking tasks)

6. Program: explicit statement of valued learning outcomes in the form of genres, tasks, content, and language


In short…

TBLTin situ

Implementation

•Curriculum

•Instruction

•Materials

•Learners

•Teachers

•Outcomes

•ETC.

Practice

Empirical

Assessment

Does TBLT work, and how can we enable it?

The Role of Evaluation in TBLT

Articulating empirical practice…

…to improve task-based education

…with practical value

Evaluation in TBLT:Responding to program purposes

What is evaluation?

…gathering trustworthy information about program elements in order to

understand, demonstrate, and improve program value…

In short…

TBLTin situ

Society

•Values

•Needs

•Resources

•Stakeholders

•Funders

•Educators

•ETC.

Practice

Empirical

Evaluation

What is TBLT worth, and how can we

improve it?



“Given the checkered history of prescriptions for language teaching, the likelihood that all 10 TBLT MPs will turn out to have been well founded is minimal. Thus, designers … who draw upon these or other sources should build careful evaluation components into their delivery systems.”

Doughty & Long (2003), p. 68“In contrast to the extensive research into tasks…there have been no more than a handful of evaluations of task-based teaching”

Ellis (2003), p. 323

Evaluation in TBLT:Problems of practice

Nunan (1989)Crookes & Gass (1993)

Willis (1996)Skehan (1998)

Lee (2000)Bygate, Skehan, & Swain (2001)

Ellis (2003)Leaver & Willis (2004)

Nunan (2005)

Overview books on TBLT…

…with chapters on program

evaluation…

Ellis (2003)

Evaluation in TBLT:Problems of practice

Bangalore Project Evaluation:

•Task-base program•Large-scale effort•External evaluator

(see Beretta, 1990a, 1990b; Beretta & Davies,

1985)

Crisis of conscienceEvaluation practice?Evaluation purpose?Evaluation impact?

Theory v. practice???

Evaluation in TBLT:An empirical approach to practice

Patton (1996)

Utilization-focused

Evaluation

Intended uses

Intended users

Questions + Methods

UnderstandImproveEducate

Demonstrate worthHold accountable

Test theoryEmpower

TeachersAdministrators

Curriculum writersLearners

Parents/publicResearchers

FundersValues clarification

ImplementationProcess-productExperimentation

Synthesis

Evaluation in TBLT:Examples in practice

Norris (2004)

Evaluation of a US college German FL

assessment program

Van den Branden (to appear)

Evaluation of task-basedteacher development

in Flanders

Assessment, measurement, andEvaluation in TBLT: Essential roles

TBLTin situ

MeasurementTheory,

ResearchAssessment

Imple-mentation

Evalu

ati

on

Society, Value

EMPIRICAL PRACTICE

http://www2.hawaii.edu/~jnorris

[email protected]

See link at: www.tblt.org

For more information:Contact me



mailto:[email protected]

See you in Hawaii for TBLT

2007!


[email protected]

See link at: www.tblt.org



mailto:[email protected]

REFERENCES

Please see the following web site for a complete pdf file of references and resources from this presentation:

http://www2.hawaii.edu/~jnorris/TBLT%20references.pdf

http://www2.hawaii.edu/~jnorris/TBLT%20references.pdf

Technology

Tblt Presentation