John M. Norris University of Hawai´i at Mānoa Learning to value evaluation In language teaching

John M. Norris

University of Hawai´i at Mānoa

Learning to value evaluation

In language teaching

Dealing with change

In language education

Language learning in crisis

“We cannot afford to seek out foreign language skills after a terrorist attack occurs. The failures of communication and understanding have already done their damage. We must provide an ongoing commitment to language education and encourage knowledge of foreign languages and cultures.”

Daniel Akaka, U.S. Senator from Hawaii


The present state of language education and proficiency in Australia is seriously inadequate for our current and emerging

needs, and far behind comparable levels in our peers and competitors. The size of this gap and the work and time required

to close it has led some to describe the situation as a crisis.

At present, though, the English-speaking abilities of a large percentage of the population are inadequate, and this imposes restrictions on

exchanges with foreigners and creates occasions when the ideas and opinions of

Japanese people are not appropriately evaluated.



“We are lousy at foreign languages and shouldn’t be”


“Viva la English”

Tribune Media: 10-26-2007

“I think that the fewer languages we have, the better off civilization will

be.”

Andy Rooney

David Harrison (2007) – Living Tongues Institute


More than half of the world's 7,000 languages are expected to die out by the end of the

century

Most of what we know about species and ecosystems is not written down anywhere,

it's only in people's heads

BIOCULTURAL DIVERSITY?

“English language learners (ELLs) are the fastest growing student population in America. Today, one out of every nine students is learning English as a second language…By 2025, English language learners will make up one out of every four students in our classrooms.”

Margaret Spellings (2005) – U.S. Secretary of Education


Number of children studying Chinese in U.S. schools: 24,000 Number of children studying English in Chinese schools: 200,000,000

Brazilians residing in Japan in 1986 = 2,135

Brazilians residing in Japan in 2005 = 302,080(141X)


Foreign students in Japanese universities in 1985 = 19,741

Foreign students in Japanese universities in 2006 = 100,804 (5X)

Number of ‘newcomer’ children who need to study Japanese in schools in 1991 = 4,463

Number of ‘newcomer’ children who need to study Japanese in schools in 2002 = 19,764 (4X)

LanguageTeaching

& Learning

Who are we teaching and what do they

need to learn?

Are we seeking survival skills, intercultural

competence or quality of mind?

How do we develop teachers to meet current

demands?

What’s the relationship

between L1, L2, social, & academic

development?

Why bother with language learning at all; what does it

offer?

What’s the most effective way to teach

language(s)?


MLA White Paper

Foreign Languages and Higher Education: New

Structures for a Changed World

High-quality teachers

NCATE – TESOL/ACTFL Teacher

Development Program Standards

In the U.S. …


MEXT

“The Ministry of Education, Culture, Sports, Science and Technology (MEXT) shall settle on

an action plan to improve English education within the fiscal year 2002 with the aim of

fostering "Japanese with English abilities". As of the fiscal year 2003, the employment of

excellent assistant language teachers of foreign nationality as regular teachers is to be

promoted.”In Japan …


“What we assess is what we value”

--Lauren Resnick“How we choose to assess will determine what gets valued”

--Norris

Why worry about evaluation?


Top 3 sources of pressure for evaluation in U.S. college FL

programs:

1.University administration

2.The dean

3.Accreditation process

“As part of its re-accreditation, the

university has required all undergraduate

programs to create and implement outcomes-oriented assessment

plans.”

(survey respondent)

“Time-consuming. Takes away from the business of teaching. Many aspects of learning can’t be measured.”

(survey respondent)


All universities, junior colleges and colleges of technology are regularly evaluated and

accredited by organizations certified by the Minister of Education, Culture, Sports, Science

and Technology

All universities, junior colleges and colleges of technology conduct self-checks and self-evaluations concerning education and

research


“This interest was fuelled primarily by a reduction in public spending to cope with recession, internal and external pressures for performance improvement, and constant demand to review, renovate, and reform the way the public sector does business in order to achieve social accountability.”

Nagao, Kuji-Shikatani, & Love (2005)

Founded in 2000

ニュース

公立中、英語で授業４％だけ　英語の授業の大半を英語で行っている公立中学校は約４％にとどまり、「授業の大半を英語で行う」とする文部科学省の目標に遠く及ばないことが、同省の調査で分かった。　また、ビジネス界で英語力の指標とされる「ＴＯＥＩＣ」で７３０点以上の英語教員は、中学で１割、高校で２割にも満たず、英語教育のお寒い実態が浮き彫りになった。　調査は今年２月、全国の公立中学校約１万２００校と、公立高校約３８００校を対象に実施した。授業での英語の活用実態を調べたのは初めて。

“Ministry survey finds schools' English teaching falling short”



NOCHILDLEFTBEHIND

COLLEGE STUDENTCollege Leaving Exam

“Does the Spellings Commission think about language education

at all?”

Michael Holquist (ADFL 2007 Summer Seminar West)

RateMyProfessors

Quality Rating Categories

☺How easy?

☺How fair?

☺How good?

HOW HOT???


Drake University language programs…

• Low enrollments, student dissatisfaction, poor external reviews

• Faculty refuse instructional development support

• Faculty refuse to create strategic plan for improvement

No acknowledgement of need to change

No engagement with evaluation findings

NO MORE LANGUAGE PROGRAMS, NO MORE FACULTY!



Despite such problems, U.S. college FL survey respondents desired increased use of evaluation for:

1. Understanding & improving program outcomes

2. Understanding & improving program functions

3. Improving FL education on the whole

4. Understanding & Improving the worth of the program

5. Raising awareness about FL programs


And some expressed a professional ethic to engage in evaluation:

“We have a social and moral responsibility towards our students and towards society at large to state as clearly as we can what it is that we do for them and

why what we do is valuable.”

(Anonymous survey respondent)

Evaluation as an agent of change

Value of

LanguageEducation

KeyChallenge:

DefineDelimitDetractDismiss

Key Opportunity:

EnableEnhanceEngenderEmpower

Within this milieu, what is the role to be played by evaluation?

Traditions, trends, and the status quo

In language program evaluation

Traditions of language program evaluation

Epistemology

Methods

Program focus

Evaluation purpose

Evaluation practice

Measure L2 outcomes, compare programs

Products of program completion

“The experimenting society”

Summative: judge program worth Knowledge: identify best methods

Evaluation use Perpetuate, fund, or shut down

JIJOEShort-term, external,

Jet-In Jet-Out Expert


…but why did the program fail (or

succeed)?

…but what if we don’t have the option of closing down the

program—how do we make improvements and what do we need

to improve?

…but you cannot compare our program with their program—they are so different!

…but that test really doesn’t measure what

we teach!

…but what can you tell me about what language teaching

method really works the best for my

learners in my school?

…but how can you conduct a rigorous

experiment on actual classrooms, teachers,

and learners?

…but you don’t really understand our

program—you were just here for a week!


Proliferation of language program evaluators (and

texts)…

• Alderson & Beretta (1992) Evaluating second language education

• Rea-Dickins & Germaine (1992) Evaluation

• Brown (1994) The elements of language curriculum

• Weir & Roberts (1994) Evaluation in ELT

• Lynch (1996) Language program evaluation

Changing emphases in evaluation practice…

• Focus on formative (improvement) purposes

• Attention to process of language teaching/learning

• Use of multiple methods (qualitative + quantitative +…)

• Pragmatic problem-solving approach

• Integration into curriculum and context

Rea-Dickins (1994):

“If evaluation in English Language Teaching is to be effective, we will see a stronger integration of evaluation within practice, as part of an individual’s professionalism, and an increase in collaborative activity where teachers (and other relevant participants) are actively engaged in the monitoring process.”

Importance of language teacher participation


Swender (2002), on FL teacher professional development:

“After all, if teachers do not know how to measure what students can do with language, how will they be able to determine whether their students are measuring up to the expectations of the 21st century”.

Countervailing trends in evaluation

Technocratic measurement problem

WebCAPE Foreign Language Placement Exam

Standardized assessment problem:

One size fits most purposes & settings


TSE – SPEAK – SLEP – TOEIC

Standardized assessment problem:

One size fits most purposes & settings


IELTS

The ‘measurement mindset’ in teacher practice:

“Although this student still has problems with grammar, the ideas are there. He is working through the choices the community has about their need for a better water supply system. Hmm, this is difficult. I just wish his grammar errors weren’t so bad then I could give him an ‘A’.”

Bernie Mohan example, TBLT 2007 Conference


What is the appropriate proficiency level to adopt as a student learning outcome for the 2-year language requirement?

INTERMEDIATE -

LOW ? ? ?



“We’re going to stand strong on accountability”--Margaret Spellings, U.S. Secretary of Education

Test Misuse Problem

“If you want to hold schools accountable and make sure they are learning, you have to test ”--Robert Black, spokesperson for the Gov. of Texas

Accountability movement: using standardized tests to hold teachers and students to performance expectations


"There has been an explosion of mandates for more and more

standardized tests with very little evidence to support their use"

--Walter Haney of Boston College's Center for the Study of Testing, Evaluation and Educational

Policy.


“Bilingual education ends up being monolingual education in the language of

the high-stakes test, until the test is over."

--Deborah Palmer, University of Texas


“…parents are dealing with children vomiting on the morning of the tests …"

--Gloria Pipkin, Florida Coalition for Assessment Reform

Outcomes embody the essential purpose of an educational program: developments in knowledge, skills, dispositions of learners

Requires rethinking of educational programs as something more than the delivery of experiences or the exposure of learners to information

Calls for articulation of curriculum and instruction in support of targeted outcomes, demands integrated thinking

Provides a clear statement of educational program value; answers the question “How do you know?” with evidence of educational effectiveness

Student Learning Outcomes

Assessment


Large public institution Accreditation

pressures to assess learning

How about an electronic portfolio?

Huge expenditure, $$$, time, effort Thousands of student

portfolios created

BUT…

We have to assess our “liberal studies” core,

ASAP!


1. *!%#$!@*

…faculty didn’t understand it…students thought:

2. waste of time

Electronic Portfolio

…administrators wanted to do something with it but weren’t sure exactly what

NEVER GOT USED


Barrington (2003), on outcomes assessment in the liberal arts:

“To design and administer (intellectually honest) assessment plans that will measure such capabilities with a dozen or more standardized ‘learning objectives’ is next to impossible” leading to “pestilent repercussions” for the truly valued learning objectives that constitute the liberal arts, in that it “discourages teaching such skills because they are difficult to measure”.

Perception problem


Traditions, trends, and the status quo

1. Focus on doing…

2. Based on standardized testing

3. Reactive v. proactive praxis

4. Driven by external impetuses

5. Technocratic measurement emphasis

6. Little scholarly investment

Back where we started1. Not useful—not used!

2. Potential negative washback, reductionism, waste

3. Not relevant to curriculum & instruction, program values

4. Not ours—done to us, not for or with us

5. Not perceived as worth the effort by faculty

6. Minimal professional development

Status quoLanguage program

evaluation

in language education

Re-envisioning evaluation

Resolving terminological confusion

Measurement is the consistent elicitation of quantifiable indicators of well-defined constructs via tests or related observation procedures; it emphasizes efficiency, objectivity, and technical aspects of construct validity.

Norris (2006) MLJ Perspectives


Assessment is the systematic gathering of information about student learning in support of teaching and learning…It may be direct or indirect, objective or subjective, formal or informal, standardized or idiosyncratic…It provides locally useful information on learners and learning to those individuals responsible for doing something about it.



Evaluation is the gathering of information about any of the variety of elements that constitute educational programs, for a variety of purposes that include primarily understanding, demonstrating, improving, and judging program value; evaluation brings evidence to bear on the problems of programs, but the nature of that evidence is not restricted to one particular methodology.


The nature of useful evaluations

Pragmatic: Context relevant

use & focusParticipatory:

Active involvement of key stakeholders

Democratic:Negotiated

decision making

Responsive:Evaluation responds to primary intended users’

purposesClear & understandable:Transparent processes

and outcomes

Educational & Transformative: Users learn by participating

Manageable & feasible: Adapted to availabletime and resources

Action oriented: Actions are taken based on

evaluation findings

Evaluative assessment

Light (2001) on outcomes assessment:

“…a process of evaluating and improving current programs, encouraging innovations, and then evaluating each innovation’s effectiveness. The key step is systematic gathering of information for sustained improvement. And always with an eye toward helping faculty or students work more effectively.”

Richard Light (2001, p. 224)

PURPOSES

Holding accountable

Revisingcurriculum

Developingprograms

Articulating courses

Diagnosingneed

Improving teaching

Raising awareness

Empoweringteachers

Revising materials

Justifying$ requests

Accreditingschools

Demonstratingvalue

Acknowledging multiple legitimate purposes

Resistingchange

METHODS

LanguageTests

LanguageProfiles

Delphi technique

Selfassessment

Observations

Documentanalyses

Interviews

Performanceassessment

Teacher logs

Studentjournals

Meetings

Portfolios

Aligning empirical methods to purposes

Surveys

Focus groups

Approaching useful evaluation

Received view: Begin by asking…

What are the objectives or outcomes targeted by the language program?

How can they be measured?

Evaluative vision: Begin by asking…

Who is in a position to utilize information for the betterment of the language program?

What questions do they have about their learners, teachers, courses? What challenges do they face?

Who is asking forthat information?

Who is doing the measuring and interpreting?

Who is held

responsible?

Proceduralizing useful evaluation

1. Participation – stakeholders, representatives, primary intended users

2. Prioritization – challenges, questions in immediate need of answers

3. Instrumentation – what data will answer the questions?

4. Collection – how can we get data in available time/resources?

5. Interpretation – what do findings mean in context?

6. Utilization – what decisions & actions are taken?

Language educators are ultimately responsible for what happens in

language education.

Participation by relevant language educators is essential throughout all

phases of evaluation if contextual relevance is sought.

A focus on specific intended uses for evaluation findings is essential from the outset, if evaluation is to make

any difference.

Why bother?

Deepen understanding of language programs by participants

Raise awareness & buy-in among teachers, students, others about evaluation Increase the likelihood

that data will be used by increasing investment in it Decrease frequency and

number of (useless) evaluations Decrease the pejorative

effects of evaluation by increasing relevance and meaning

Enhance communication among program stakeholders

Actually do something on the basis of evaluation!

What does it look like?

Value added by evaluation in language education

Example 1: Improving teacher induction practices

Context

•English Language Institute•U.S. University

•Diverse International Students•Required ESL coursework

•M.A. Graduate TA Instructors

Rapid turnover in

GTAsVariable teaching

experience

“Lack of preparednes

s”

Flexible induction practices

Feeling of ‘sink or swim’ teaching

High student & uni expectations

for ELI

Teacher-led internal

evaluation

Teacher pre-service needs?

Admin’s induction goals?

ELI Administration

= Intended users

Improvements in induction practices?

Example 1: Improving teacher induction practices

Methods

Interviews:

•Admin

Focus groups:

•Experienced teachers Surveys:

•Admin

•Former + new teachers

Findings + Uses

Induction partially successful: identified weaknesses

Induction outcomes not explicit: admin created based on their goals + teacher wants

Teachers value more induction: increase required

Teachers value variety of practices: enhance availability (online resources, observation, meeting with other teachers)

New teachers unaware: early hiring, pre-work internships, annotated syllabi

Transformations

Teachers empowered,

support improved

Admin values evaluation

Dedicated teacher

evaluator position created

ELI Administrator: “I can now see how evaluation is connected to the evolution of the program”

Institution doubts FL

contributionThreats of program closure

Instructors prepared to

teach for proficiency?

Monitor teaching and

learningLearners

invested in FL learning?

Director-led evaluation

Students achieving outcomes?Teachers

focusing on proficiency?

Institution, administration,

faculty = Intended users

Program perpetuation, improvement?

Context

•Stanford language center

•44 foreign languages

•1-year undergraduate language requirement

•Internationalization of the curriculum

•Teach for proficiency, ability to use the FL, performance outcomes

SLC

Example 2: Perpetuating language studies

Pressure to assess

outcomes

MethodsStudents:

SOPI tests (simulated proficiency interviews)

Teachers:

Oral proficiency testing certification

+Student proficiency requirements

+Web-based dissemination

+Student course evaluations

Findings + Uses

Uncertain outcomes of FLs: set proficiency standard on national scale (ACTFL)

Students enroll in FLs: enrollments on steady increase; request funding, staffing $$

Assessment used to ensure proficiency: publish outcomes by language; basis to seek support where needed

Instructors teach for proficiency: reward structure, $

Students evaluations very high: perpetuate requirement and proficiency focus

Transformations

Program survival, growth,

support

Teachers associate

professionalism with evaluation

Institutional respect

garnered for language teaching

Example 2: Perpetuating language studies

Program Director: “…focusing on student achievements has reverberative effects on

professional development in its many facets as well as on the perception of the efficacy of language programs throughout

an institution.”

Example 3: Innovating in teacher development

High public demand for

DutchPoor Dutch L2

outcomes

Teacher understanding

?

National curricular change

Teacher development?

Teacher willingness?

Innovator-led evaluation

How do teachers learn/change?

What are the constraints on

change?

Teacher trainers, schools, teachers

Improvements in Dutch SL learning?

Context

Belgium

•Dutch SL education in Flanders

•Large-scale K-12 education

•Ensuring functional L2 abilities

•Providing access to ed/society

•Task-based teaching innovation

•Teacher pre-service training

•Teacher in-service support

How can teacher-dev be optimized?

Methods

Teachers: Interview, Observation, Survey, Log

Trainings: Observation, Interview

Students: Pre-test, post-test Dutch L2 proficiency, Observation

Findings + Uses

Teacher TBLT dev failing: training not task-based, too short; change training

Teachers & students adapt to new training, but variably: individualize training, provide school-based support

Teachers fear loss of control, lack of structure: adopt gradualist approach, lots of guidelines & coaching

Students’ scores increase most in TBLT classes: disseminate to teachers & schools & public

Transformations

Teacher cognition

illuminated

Teacher agency

respected

Task-based learning

happening effectively

Example 3: Innovating in teacher development

10+ years, cycles of use

Innovation doesn’t happen

over night!

Headteacher comment, early stages: “In the beginning, teachers were suspicious”…“Try

that with my students!”

Teacher comment, later stages: “You have a training, a meeting, you can try things out, and then there is another meeting and you

can evaluate. I like that.”

What can we learn from these examples?

Received View

•Generic, one-size-fits all

Useful Evaluations

Contextualized: specific language programs

•Accountability-driven•Measurement-based

•One-shot, absolute

•Problem-identifying

Intentional: multiple purposes and usesDiversified: methods articulated to usesIterative: change takes time

Problem-solving: improve via understanding

•Imposed, required Engaging: stakeholders (especially educators) take interest in and act upon evaluation

Facing change:

Learning to value evaluation

University of Hawaii, National Foreign Language Resource Center

Summer Institute 2007

“enables the field to articulate and demonstrate—internally and externally—the unique contributions of language studies in a pluralist and globalized world.”

What is the value of evaluation in language education?

Provides a framework

for discussion

Encourages heightened commitment

Increases awareness,

communication

Makes student learning

more efficientDemocratize

s, unifies, engages…

Facilitates solving of problems

Sheds light on how

programs function

What can we do?

Clarify roles & make space for evaluation in language programs

Encourage, enable, and engage in professional development

Generate and share examples, participate in the discourse

Hold evaluation accountable to language programs, teachers, learners

USE it or lose it…

FIND OUT MORE!

University of Hawai‘i at Mānoa, Second Language Studies

http://www.nflrc.hawaii.edu/evaluation

http://www.nflrc.hawaii.edu/evaluation

University of Hawaii

Mahalo!

Documents

John M. Norris University of Hawai´i at Mānoa Learning to value evaluation In language teaching