MASARYK UNIVERSITY

FACULTY OF EDUCATION

DEPARTMENT OF ENGLISH LANGUAGE AND

LITERATURE

Some aspects of assessing pronunciation in EFL classes

Diploma thesis

Brno 2007

Supervisor: Written by:

PhDr. Jaroslav Ondráček Simona Šebestová

Declaration

I hereby declare that this thesis is my own and that I used only the resources

stated.

Simona Šebestová

.......................................

2

Acknowledgements

I wish to thank to PhDr. Jaroslav Ondráček, the supervisor of my thesis, for his

valuable advice and guidance on the whole work. Thanks also belong to Petra

Hořejší and Štěpánka Zemanová for their help with the assessment and my

special thank goes to all my students who willingly cooperated.

3

Contents

Introduction...............................................................................................................6

1 Teaching and learning English pronunciation..........................................................7

1. 1 The role of the teacher......................................................................................8

1. 2 The role of the learner......................................................................................8

1. 3 Setting goals in English pronunciation.............................................................9

1. 4 What affects pronunciation learning..............................................................11

1. 4. 1 The native language.................................................................................11

1. 4. 2 The age....................................................................................................12

1. 4. 3 Phonetic ability........................................................................................13

1. 4. 4 Exposure to the language.........................................................................14

1. 4. 5 Attitude and sense of identity..................................................................15

1. 4. 6 Motivation...............................................................................................16

2 Giving feedback on pronunciation.........................................................................16

2. 1 Assessment methods.......................................................................................17

2. 2. 1 Impression-based pronunciation testing..................................................18

2. 2. 2 Atomistic testing......................................................................................19

2. 2. 3 Holistic testing.........................................................................................21

2. 2. 4 Atomistic versus holistic approach..........................................................25

3 Practical part...........................................................................................................26

3. 1 Introduction....................................................................................................26

3. 2 Evocation........................................................................................................27

3. 2. 1 Activity 1.................................................................................................29

3. 2. 2 Activity 2.................................................................................................31

3. 3 Realisation......................................................................................................34

3. 3. 1 Limited-response activities......................................................................35

3. 3. 2 Free-response activities...........................................................................38

3. 4 Reflexion........................................................................................................40

4 Research part.........................................................................................................46

4. 1 Plan and goals.................................................................................................47

4. 2 Methods..........................................................................................................47

4. 3 Evocation and pre-research............................................................................48

4. 4 Realisation......................................................................................................52

4

4. 4. 1 Atomistic testing results..........................................................................54

4. 4. 2 Holistic testing results.............................................................................57

4. 5 Reflexion........................................................................................................63

4. 5. 1 Assessors‘ agreement in atomistic testing...............................................64

4. 5. 2 Assessors‘ agreement in holistic testing..........................................67

Conclusion.......................................................................................................................70

Appendix 1......................................................................................................................73

Appendix 2......................................................................................................................75

Bibliography....................................................................................................................77

5

Introduction

English language plays a dominant role in modern

world. Each of us has ever been encountered with the

necessity to learn the language for many reasons. For many

people the most important function of any language is

communication with other people when travelling, studying

or working in a foreign country. Therefore, the trend of

learning English is increasing and the opportunities for

the actual use of the language are becoming more frequent.

The topic of this work is based largely on the

communicative function of language – speaking and

listening. We intend to deal with an aspect of English

which is, in our opinion, often neglected in English

lessons – pronunciation testing and its assessment.

The idea arose from our belief that teaching

pronunciation should be a part of a syllabus of any English

course, moreover, should be included from the very

beginning. One reason for this statement is that it is

generally believed that the age factor is influencial and

small children can learn pronunciation more easily than the

older or adults. The next benefit of pronunciation learning

is its possible effect on listening skills. In order that

our communication is comfortable for both sides, we need to

understand as well as our speaking should be

understandable. So the practice of our pronunciation helps

other people to understand us better.

Nowadays, almost every coursebook offers pronunciation

practice but any suggestions on giving such a structured

feedback like those on other aspects of language are rare.

We will try to present ways of testing pronounciation and

try to deal with the possible ways of the consequent

assessment.

6

Apart from the objective methods of testing

pronunciation which are not usually available in schools,

there are other – more often used – subjective methods.

These methods are based on the assessor’s impression, so

certain criteria should be followed to reach as reliable

results as possible.

The first part of this work will present the tasks

needed to be considered for efficient learning and teaching

– the role of the teacher and the learner and the factors

which have an effect on the whole pronunciation teaching

and learning process.

Next, we will give an overview of the testing methods

in connection with the two approaches to impression-based

assessment – atomistic and holistic.

The last research part will implement the theory into

practice and find out about what results can be actually

brought by the pronunciation testing and what problems in

assessment may be encountered.

1 Teaching and learning English pronunciationPronunciation seems to be sometimes a neglected part

in our English lessons. Many teachers are not used to

teaching it for many reasons. As most teachers in our

schools are not native speakers, there is no need to set

native-like pronunciation as one of our goals, moreover,

most teachers themselves do not feel perfect in this

language component and thus feel reluctant to show it. But

we do not need to be perfect to enable our pupils to

achieve their best. The next obstruction for teachers is

the lack of pronunciation tests and general unfamiliarity

with their assessment. Nevertheless, giving feedback on

correct speaking to our students should be involved.

Learning pronunciation will elevate their level of speaking

and undoubtedly will improve their listening skills.

7

Before teaching pronunciation, many aspects should be

taken into account. Among the most important ones are the

roles of the teacher and the learner. On the one hand, what

they aim to achieve and, on the other hand, what they are

willing to give to succeed in achieving it. Before setting

up goals and working out a plan, the teacher should know

about their learners’ skills and limitations. Needless to

say, the overview of the various aspects of English

pronunciation – sounds, stress, rhythm and intonation – is

essential.

1. 1 The role of the teacherAs pronunciation is a very complex language component,

there are many tasks for the teacher. Firstly, helping

learners hear and produce sounds from their native language

point of view. Secondly, it is establishing what to

concentrate on. It is advisable to set out priorities

according to the acceptability to the English speaker as

well as to what is beyond good intelligibility and is not

necessary to take one’s stand on. Thirdly, devising

activities, adjusting them for different learning styles,

and according to the aspects which influence pronunciation

learning as it is going to be described in the following

chapters. The last but not least important task is

providing feedback and assessing learners’ performance and

progress. Assessing one’s own production of speech is very

difficult as we tend to hear ourselves in a distorted way

and thus it is complicated to compare with the correct

production. Moreover, the feedback on how the learner is

doing and progressing is an essential motivation factor in

further pronunciation learning.

8

1. 2 The role of the learnerThe learner’s role is the same as in any other subject

which means taking one’s own responsibility and being

willing to learn. Here, the teacher’s possibilities are

limited, but still the choice of appropriate activities,

motivating learners and building the general awareness of

usefulness may be supportive. For example, one of the

methods for increasing motivation can be a class-discussion

based on one’s own experience with foreigners and their

pronunciation, what is acceptable and what is disruptive

while talking to a foreigner in any language. The issue of

motivation within all subjects of learning is definitely

very complex, depending on many inner and outer factors and

should not be neglected.

1. 3 Setting goals in English pronunciationPeople learn languages for many different purposes.

And therefore, the goals for individual learners may vary.

From the teachers’ point of view, the following aspects

should be taken into account: the age, natural ability and

motivation of the learners which is to be the base for

answering the questions about how much time we will devote

to teaching pronunciation and what level is needed for

obtaining efficient communication. This is difficult,

since, in contrast with e. g. grammar or vocabulary plans,

pronunciation does not enable this particular progressive

pace as all phonetic and phonological features occur from

the very beginning. Nevertheless, we can count on the

subconscious acquisition of the sound of English which will

be beneficial for both, teachers and learners, later on.

We can delimit two extreme targets in learning

pronunciation. On the one hand, some learners aim to obtain

native-like pronunciation, on the other hand, many

learners’ purposes are more practical in the way that as

9

long as their speaking is comprehensible, they do not have

the need to improve. Both these opinions have their

advantages and disadvantages. In practice, many learners do

not achieve native-like pronunciation and the question is,

if its obtaining is necessary. At the same time, learning

pronunciation does not only improve speaking, but has a

great influence on our listening skills, so its practice is

useful.

According to Gimson (1994, p. 273), the first extreme

target is achieving just such a level of pronunciation

which enables understanding. Gimson mentions so called

Minimal General Intelligibility as the lowest requirement.

This Minimal General Intelligibility:

possesses a set of distinctive elements which

correspond in some measure to the inventory of

the RP phonemic system and which is capable of

conveying a message efficiently from a native

English listener’s standpoint, given that the

context of the message is known and that the

listener has had time to “tune in” to the

speaker’s pronunciation. (p. 273)

In its opposition, Gimson describes High Acceptability as:

a form of speech which the native listener may

not identify as non-native, which conveys

information as readily as would a native’s and

which arrives at this result through precision in

the phonetic realization of phonemes and by

confident handling of accentual and intonational

patterns. (p. 273)

10

As the previously mentioned statements were the

extremes in attitude to pronunciation learning, our aim

should be somewhere in between, which means that we should

aim to reduce the time to “tune in” for the listener as

well as to put down the strain for the speaker in order to

make the conversation comfortable for both sides.

1. 4 What affects pronunciation learningLike in other subjects, there are many factors which

affect pronunciation learning, but some of them are

specific to this language section as well as to learning

foreign language as a whole. It involves both learning as

an organized and intentional process, and the unintentional

language acquisition, which proceeds subconsciously and

intuitively, however, is not less beneficial. Both,

learning and the language acquisition, are dependable on

the inner and outer factors, which the teachers should be

aware of when setting out goals and preparing methods and

materials.

The factors which cannot be influenced neither by the

learner, nor his or her surroundings are the native

language, the age and to a certain extent the phonetic

ability of the learner. In contrast, the amount of exposure

to English, the attitude and motivation may be supported

and changed a lot either by the teacher, or the learner,

but usually in cooperation and support from both sides.

1. 4. 1 The native languageThe influence of the native language is unquestioned.

It involves the individual sounds as well as combinations

of sounds and features such as rhythm and intonation. The

more differences there are between the native and the

target language, the more difficulties will be encountered

by the learner. The learner’s errors derive from various

11

sources. When a particular sound does not exist in the

mother tongue, the learners tend to substitute it by the

nearest equivalent from their native language. For example,

the sound [ð] will often be substituted by [d] or [z] as

these are the nearest. This first language application

affects the rhythm and the intonation too. Still, the

influence of the native language is only one of the factors

and does not need to be crucial.

1. 4. 2 The ageIt is commonly assumed that the age factor has a great

influence on pronunciation. If someone has a native-like

pronunciation in a second language, they are very likely to

have started learning it as children. And conversely,

people who started to learn a second language in adulthood,

will scarcely achieve a native-like accent, although their

grammar and vocabulary may be perfect. However, the results

of the studies on the age-relation issue which have been

carried out are conflicting. One should take into account

that it is very difficult to control other factors like

motivation, ability, opportunity to hear and use the

language and the learners‘ attitude, therefore the results

are bound to be distorted to a certain extent. Don Snow

(1992) comments on this issue in a contradictory way:

Research comparing children to adults has

consistently demonstrated that adolescents and

adults perform better than young children under

controlled conditions (e.g., Snow & Hoefnagel-

Hoehle, 1978). One exception is pronunciation,

although even here some studies show better

results for older learners. (p. 129)

12

The idea of pronunciation learning being somehow

dependent on the age has been supported by some researchers

who claim that language learning has a sensitive period and

that after a certain age people lose some abilities. This

certain age is said to be between ten and thirteen years.

(Kenworthy, 1987, p. 6) The age-relation is supported also

by others:

Pronunciation is one area where the younger-is-

better assumption may have validity. Research

(e.g., Oyama, 1976) has found that the earlier a

learner begins a second language, the more

native-like the accent he or she develops. (Snow,

1992, p. 129)

From the teacher‘s point of view, we assume, that for

learning pronunciation the age of primary and lower-

secondary level pupils is more convenient, as they are less

shy and often more willing to do activities which the adult

learners may find strange or silly. In children, we can

take the advantage of their spontaneity and willingness to

sing and chant, which is often not appropriate in older

learners. „Research suggests that older students will show

quicker gains, though younger children may have an

advantage in pronunciation.“ (Snow, 1992, p. 129) However,

when the adult learners are motivated and want to approach

native-like pronunciation, they are able to focus better on

their problematic areas and this fixed-target acquisition

may be also very efficient.

1. 4. 3 Phonetic ability

It is generally assumed that some people have a better

ability to hear foreign languages than others. This

“aptitude for oral mimicry”, “phonetic coding ability” or

13

“auditory discriminating ability” proved by researchers who

designed tests which measure this ability. Although the

results proved that some people can better discriminate

between two sounds, still it is not a crucial condition of

obtaining good pronunciation as training itself has a

significant effect. (Kenworthy, 1987, p. 6)

The point is, whether different types of learners

should be taught with different methods. Kenworthy (1987)

divides these learners into poor discriminators and good

discriminators and explains:

One study has indicated that those with good

phonetic abilities benefit from pronunciation

drills, tasks in which particular sounds are

heard and the learner has to imitate again and

again. Their innate abilities enable them to

exploit all the opportunities to compare what

they are doing with the model presented. “Poor

discriminators” do not seem to benefit from

drills very much. In fact, drills seem to cause

their attempts to stabilize before they reach an

accurate production of a sound. Because of the

complexities involved, this seems a factor which

is very much out of the control of the teacher.

We can only operate on the assumption that our

learners have the “basic equipment” and provide a

variety of tasks so that something will suit the

needs and abilities of each learner. (p. 7)

1. 4. 4 Exposure to the language

The exposure to the target language has a wide range

of meanings and intensity. The meanings may vary from

living in the country where the language is spoken to

talking to native-speakers or using authentic materials, e.

14

g. films, literature, music, television, so the term is

quite wide in its meaning. Living in the country of the

target language does not always mean the actual use of the

language. Many people may live in a non-English-speaking

environment, or use their mother tongue with their

families. Conversely, many learners who live in a non-

English speaking country may use English in many daily

situations like school or work, so the amount of exposure

basically depends on its quality and quantity.

Logically, being exposed to English in the English

environment should have a positive effect on the learner‘s

pronunciation and listening skills. “Research, however,

indicates that this increased exposure to English does not

necessarily speed the acquisition of English.“ (Snow, 1992,

p. 129)

It is considered to be a contributory factor, but is

not the most important.

1. 4. 5 Attitude and sense of identityIn the acquisition of pronunciation of a foreign

language the attitude and sense of identity of the learner

plays a major role. It is a personality-determined factor

and we can hardly predict whether people having come to the

environment with different accent will modify their own.

“Some seem to be “impervious” and even after a long time

will absorb only some turns of phrase and the pronunciation

of a few individual words; others seem very receptive and

begin to change their accent almost as soon as they step

off the plane.” (Kenworthy, 1987, p. 7)

Using or not using the accent may show different

attitudes to people. When imitating or adopting the accent

of the host country, we show our positive relation to their

language and culture as well as respect for them as people.

15

It is sign of how much we would like to integrate ourselves

in the new place.

As far as teaching pronunciation is concerned, the

attitude and sense of identity factor is closely connected

with the inner motivation factor which is going to be

explained next.

1. 4. 6 MotivationFor some learners pronunciation is more important than

for others. The learners who consider pronunciation to be

an important part of their English learning are usually

eager to be corrected and concerned about how good or bad

their pronunciation is. This can be also expressed by

unwillingness to talk when being uncertain about one’s

pronunciation: “If we cannot say it perfectly, we won’t say

it at all.” In these learners the teacher can perform

magic, because the learner wants to cooperate and improve.

To maintain this motivation from the learner’s perspective,

the teacher should set out goals and give the learner

feedback, so that both are aware of some kind of progress.

Conversely, the other extreme is, when the learner is not

aware that their way of speaking causes difficulty,

irritation or misunderstanding for the listener. The

teacher’s possibilities in motivating learners are limited,

nevertheless, the factor of motivation should not be

neglected as it is considered to be one of the most

important factors affecting pronunciation learning.

2 Giving feedback on pronunciationTeaching a language is inevitably tied with feedback

which involves either formal or informal testing, assessing

achievements, and offering suggestions for making

improvements.

16

As we have already mentioned, giving feedback on

pronunciation is essential for maintaining motivation and

providing students with information on how they are

progressing and what they should focus on. Compared to

learning grammar or vocabulary, where students are able to

assess themselves objectively having the correct answers at

their disposal, self-evaluation in pronunciation is

complicated since it is very often distorted by the

student’s own ear and phonetic ability. The importance of

teacher’s role in giving feedback on this language

component is thus unassailable and crucial. Yet, also from

the teacher’s point of view, assessing pronunciation, in

comparison with other language components seems to be a

difficult task as, at least, in the literature which we

have had at our disposal not much attention is paid to this

issue. The main reason for this negligence is the fact that

speaking, comprising pronunciation, is a skill which is too

complex to enable any realiable analysis which could be

considered to be objective.

17

2. 1 Assessment methods

Learning pronunciation is based on both production and

reception/ identification, therefore, it is closely

connected with oral exams and listening tests. Both

speaking and listening activities may be used for testing

and assessing our learners’ progress, although they do not

have to provide us with comparable and agreeable results.

Contrary to production, reception can be tested and

quantified by counting the correct answers in a written

test, whereas production is more difficult to assess as it

involves testing speaking or reading which, from the

listener’s point of view, include many other aspects of

communication, not pure sounds. It is always subjective to

a large extent and so the problematic matter of assessing

pronunciation production is its reliability. We are bound

to rely on the impression of the assessor. Therefore, both

the procedure and the assessment should be defined as

concretely as possible. Depending on the aim of our

assessment, two approaches of testing pronunciation are

going to be described – atomistic and holistic. Before

dealing with these, we look at the most common concept of

pronunciation testing.

2. 2. 1 Impression-based pronunciation testing

Taking into account our school environment, teaching

and technical conditions in our classes, testing based on

the teacher’s impression rather than using any objective

scientific methods is inevitable. Nevertheless, we should

be aware of the possible difficulties. According to Szpyra-

Kozlowska et al. (2005), the impression-based pronunciation

testing has many drawbacks. She mentions that different

assessors may have different criteria of evaluation and it

is also more general and therefore can be imprecise. In

opposition to such an opinion, Kenworthy (1987, p. 20)

18

believes that also assessments which are impressionistic

and therefore subjective can be both dependable and

accurate. She claims that the impression-based rankings

tend to agree with other objective ranking techniques.

What speaks in favour of the impression-based

pronunciation testing, is the fact that it is typical of

many international examinations in English, like Cambridge

English Examinations, where the assessors evaluate

"intelligibility and the amount of strain a candidate's

pronunciation puts on the listener.“ (Szpyra-Kozlowska et

al., 2005)

For example, the requirements on pronunciation in

First Certificate in English are defined: "although

pronunciation is easily understood, L1 features may be

intrusive"1, in Certificate in Advanced English, the same

is defined as follows: "L1 accent may be evident but does

not affect the clarity of the message"2 and in Certificate

of Proficiency in English: "pronunciation is easily

understood and prosodic features are used effectively; many

features, including pausing and hesitation, are "native-

like"."3 Obviously, the definitions of the requirements in

Cambridge English Examinations are very general and

impressionistic in nature. The examiners who are very often

non-native speakers of English are instructed as follows:

"when assessing pronunciation, examiners should try to put

themselves in the position of a non-EFL specialist, native

speaker of English and assess the amount of strain on the

listener and the degree of patience an effort required to

understand the candidate.“ (as cited in Szpyra-Kozlowska et

al., 2005)

On the basis of this instruction, the assessment may

have more interperetations depending on the assessor and is 1 Paper 3: Speaking. Cambridge Common Scale for Speaking [online; quoted 25. 1. 2007] http://www.cambridgeesol.org/support/dloads/ket/KET_HB_sampleS.pdf2 ibid3 ibid

19

bound to be always subjective to some extent, but we can

conclude that it is a method which can provide us with

information about the progress and achievement in quite a

plausible way and it enables us to compare the learners‘

performances within time or class.

2. 2. 2 Atomistic testingAtomistic approach requires a detailed marking scheme

in which specific aspects of pronunciation are evaluated

separately. It means reading aloud word lists based on

phonemic oppositions, short sentences containing minimal

pairs or it also enables testing appropriate sentence

accentuation or sentence stress and intonation. This

approach is claimed to be more objective than the holistic

one as it judges only segments of speech – particular

vowels, consonants, stress, rhythm, intonation, etc.

Nevertheless, the drawbacks of this approach are the

demands on the assessor – in our school environment – the

teacher. It requires recording the learners’ speech samples

and repeated listening to them, so it is extremely time-

consuming and thus unsuitable for large classes.

3. 2. 2. 1 Methods of testingAtomistic testing may apply to both production and

identification of sounds, stress patterns and intonation.

It is advisable for the teacher to choose a good material

and set out clear criteria beforehand. Atomistic testing

focuses always on a segment of speech, therefore we often

have to omit those wrongly pronounced segments of language,

which have not been aimed to assess. This is the only way

both to compare our learners’ performances, and to avoid

any confusion from the overload of information.

Repetition

20

On the beginner level, the easiest test to prepare is

repetition exercise. It is useful for learners who cannot

read or who are beginning with English. It is based on

hearing sounds, stress and intonation, and imitation which

gives the teacher the gist of learners’ potential and

phonetic ability. The test may consist of single words or

sentences checking particular items rather than all

pronunciation aspects at the same time in order to be as

much objective as possible.

Hearing identification

Another way of testing beginners as well as more

advanced learners is hearing identification (Madsen, 1983,

p. 61). Good pronunciation is dependent on our ability to

hear the language. It can consist of recognizing sounds in

minimal pairs, the fall or rise in intonation or

indentifying stress in words or sentences.

Reading aloud

Commonly used way of pronunciation assessment is

reading aloud. According to Madsen (1983) three points

should be kept in mind:

(1) When using lists of sentences, evaluate only

one or two points per sentence;

(2) use natural language;

(3) avoid signalling to the student which

pronunciation point you are testing. (p. 66)

Since reading tends to be longer and involve many

points to assess at the same time, it is advisable to

record the learners’ performances in order to listen to

them repeatedly and have the possibility to compare. The

material to read should enable natural sound, e. g. a

21

letter, instructions etc., and students should have time to

read the text silently before reading for assessment to get

the context. The reading aloud testing provides good

control and enables to test almost all aspects of

pronunciation including stress and intonation as well as

vowels and consonants. Nevertheless, we have to count on

the fact that reading and speaking skills are not the same

and, inevitably, the intonation and sentence rhythm in

reading is usually not as natural as in normal

conversation.

2. 2. 3 Holistic testingA higher level of achievement is testing

intelligibility and acceptability of the learner’s

performance. In this holistic approach to pronunciation

testing “examiners are asked not to pay too much attention

to any one aspect of a candidate’s performance, but rather

to judge its overall effectiveness.” (Alderson, 1996, p.

289)

The advantage of this procedure is that it can be

administered to large groups and is not as time-demanding

as the atomistic approach. As it has been already mentioned

this approach is used many international exams in English,

where the pronunciation is involved in so called

intelligibility and acceptability of the candidate’s

speaking performance. The definition of intelligibility is

very general: “Intelligibility is being understood by a

listener at a given time in a given situation. So it’s the

same as “understandability”.” (Kenworthy, 1987, p. 13) To

put intelligibility more concretely, we can say that: “The

more words a listener is able to identify accurately when

22

said by a particular speaker, the more intelligible that

speaker is.” (Kenworthy, 1987, p. 13)

In practice, words are made up of sounds and if the

speaker substitutes one sound for another and the result is

that the listener hears a different word, it means that the

speaker is unintelligible or not intelligible comfortably.

However, when the wrong substitution occurs in a speech,

but the word is anyhow understood, then we can say that the

speech is intelligible. Considering this should be the base

for setting out the aims of our pronunciation lessons,

which does not necessarily need to be native-like speaking

performance.

According to Kenworthy (1987; p. 14), as

intelligibility is very complex, there are many factors

influencing it. One of them is the counts of sameness. This

applies to the Czech practice for example in the sound [ð]

which is often substituted for [d] as it is the nearest

equivalent in the Czech language. But, when the listener is

aware of this and the incorrect substitution does not

produce another word, he or she will not have difficulties

understanding. The other factors affecting learner’s

intelligibility are, for example, the over-use of self-

corrections, hesitations or grammatical restructurings. All

previously mentioned factors may be annoying or disruptive

for understanding as well as for the whole communication.

Another speaker factor which can cause problems is that the

person speaks too fast. According to Kenworthy (1987, p.

14) it is usually not the speed itself which causes

difficulties, but incorrectly applied other features of

speaking, like sentence stress, rhythm, and intonation

which makes it difficult for the listener to pick out the

most important bits of the message.

So far, all the above mentioned factors were speaker-

focused. But we have to be aware of the fact that there are

23

always at least two participants in a communication.

Intelligibility also depends on the listener. Kenworthy

mentions two important listener factors: “first, the

listener’s familiarity with the foreign accent and, second,

the listener’s ability to use contextual clues when

listening.” (Kenworthy, 1987, p. 14)

To conclude, understanding is dependent on the amount

of exposure and being used to the foreign accent. A teacher

should be aware of this fact. Being exposed to the

learners’ pronunciation, the teacher’s assessment is very

likely to be distorted.

The issue of intelligibility is very complex and is a

major part of communication. Therefore, the goal is not

only the correct production of sounds, stress patterns and

intonation, but efficiency of communication without

irritation and difficulties understanding. So the goal of

pronunciation can be defined as comfortable

intelligibility.

2. 2. 3. 1 Methods of testingAs we have already mentioned, the main criterion for

holistic testing is the efficiency of communication between

two people. Therefore, the best method is interactive

testing including more than only one participant. All the

activities should be used in the interaction of the

assessor or another student to involve both sides of the

communication – the speaker and the listener – to function

as an oral interview including natural situations and

asking questions.

Re-telling stories

This kind of test involves first reading a story

silently and then telling the story using one’s own words

24

and sentence structures. The assessor may interefere giving

further questions.

Description of pictures

Pictures may be used for description of objects,

people or scenes, or for comparison of two similar

pictures, in which the learner looks for similarities and

differences.

Sequence of pictures

This test is based on telling a story involving

linking words expressing the cause and the result. It can

be applied to only one student or a pair where each of them

is given one half of the pictures and they should decide on

the correct sequence of the story.

Pictures with speech bubbles

In this test students are required to guess what the

people in the pictures are saying. It may be used

individually or in pairs.

Using maps

Many student’s books involve a unit dealing with

giving directions. This activity is to be done in pairs,

where one gives the directions and the other one follows

them.

2. 2. 4 Atomistic versus holistic approachWhen deciding between atomistic and holistic testing,

the purpose of testing should be considered. As far as

reliability is concerned, in Szpyra-Kozlowska (2005) Hughes

(1991) says that atomistic tests are more reliable for

diagnostic purposes in the language classroom and in cases

in which scoring is carried out by different assessors,

25

whereas holistic approach is faster and more appropriate

for experienced assessors.

26

3 Practical part

3. 1 Introduction

In this practical part we are going to deal with the

practical aspects of pronunciation testing and assessment

in class. On the basis of the theoretical part, a

qualitative research has been carried out to test students‘

pronunciation and their progress within a period of time.

The idea of assessing students‘ pronunciation arose

out of its teaching. In my English lessons we did

pronunciation exercises but the process was somehow not

complete. The feedback for the learners as well as for me

as a teacher was missing apart from me informally

responding to the individual performances. The conception

of pronunciation in our coursebooks did not enable me more

than assessing the learner‘s immediate performance. Before

starting our research, the students had been taught

pronunciation but had not been provided with any statement

on if they were progressing or not, or just a list of

points what they should focus on. So the need of a

structured feedback and the lack of instructions in our

coursebooks led us to try to design a complete

pronunciation course involving all the necessary steps.

Having studied literature dealing with the issue of

pronunciation tests, I found the instructions scattered in

many books, always being presented just in bits and pieces.

Therefore, I tried to put them together to be used in my

lessons in order to see to what extent they are beneficial

for the learners and achievable as far as their

reliability, time and demands on the assessors are

concerned.

The research itself and its results will be presented

as the last item of this part and will be preceded by three

27

parts generally relating to the three stages used in the

research.

The first part is the „evocation stage“. It deals with

the factors affecting pronunciation learning which have

already been described in the theoretical part, to be

precise, it deals only with such factors which the teacher

or the learner have some influence on. Further, we present

activities and recommendations which may be used in the

classroom.

The second part is called the „realisation stage“.

This stage involves the realisation of teaching

pronunciation and its testing. We omit the teaching

pronunciation phase and focus on the ways of testing

proceeding from the level of sounds production towards

testing intelligibility and considering their weaknesses

and suitability.

The third part is the „reflexion stage“. It presents

the description of criterial levels for atomistic and

holistic assessing and the requirements on the choice of

the assessors.

3. 2 Evocation

Setting out goals and criteria of assessment as well

as teaching pronunciation itself requires considering all

aspects affecting pronunciation and its learning. Having

been generally explained in the teoretical part, some of

these aspects are given and cannot be changed by any

participant of the learning process. Among these the

following are listed: the age, the native language and the

phonetic ability of the learner. Since we cannot influence

them, it is essential that we take them into account when

planning a course. The factors which the teachers or the

learners have some, however, limited control of, are the

exposure to English, the attitude and the motivation.

28

Nevertheless, the exposure to English during the course of

learning it, is somehow given depending on the English or

non-English environment, but we can still advise our

students on other possibilities which can improve their

English outside the class, e. g. help them use authentic

materials like film, newspaper or find a native speaker to

communicate with. The teacher’s control is quite limited,

because we cannot influence whether our learners really do

so. But we can contribute by using the right motivation

methods, either self-made, or inspired by those presented

in literature.

In the theoretical part an opinion was expressed that

motivation belongs to the crucial factors affecting

pronunciation. Kenworthy (1987, p. 57) claims that "a

higher level of awareness and concern usually has a

positive effect on motivation and therefore achievement."

For this reason the stage of „building awareness“ should

not be neglected.

What may be particularly helpful are the „General

awareness-building activities“ (Kenworthy, 1987, p. 54).

These should be used in such situations in which the

teacher encounters any negative attitute from the learners‘

side towards the pronunciation learning. Kenworthy mentions

two of such activities: „Questionnaire based discussion“

and „A tape-based activity“. „The first is based on a

questionnaire, the second centres on samples of the native

languages of the learners spoken with a heavy foreign

accent. Both aim to help learners develop a concern for the

pronunciation of English through their personal experiences

of language in use.“ (ibid)

The descriptions of the activities have been inspired

by various resources and the instructions are only

suggested and open to any adjustments.

29

3. 2. 1 Activity 1Aim: Increasing interest and motivation, discussion on

pronunciation

Resource: Questionnaire, questions and instructions in

Kenworthy’s Teaching English Pronunciation, Dalton, Ch.,

Seidlhofer, B. Pronunciation (1994, p. 155) and Ur, P. A

Course in Language Teaching (1997, p. 51)

Time: at least 30 minutes

Language used: mother tongue is acceptable for meaningful

discussion

Procedure:

1 Filling in the questionnaires individually;

2 Comparing the answers in pairs or groups;

3 Creating statements and questions based on the

pair/group discussion;

4 Whole-class discussion of the prepared statements and

questions.

Output: Teacher's evaluation:

- How involved were the students in the discussion?

- Did agreement or disagreement predominate, and what do

you think are the reasons?

- Were there questions which did not lead to an interesting

discussion? If so, why?

- Are there questions which you would like to add?

30

Activity 1 - Questionnaire4

1 Imagine you are talking in your own language with a foreigner. The person doesn’t speak

your language very well and is very difficult to understand. What do you do? Do you:

(a) pretend you understand even when you don’t?

(b) ask him or her to repeat everything slowly and carefully?

(c) try to get away?

2 What do you say when the foreign speaker apologizes for his poor accent? Do you:

(a) tell him his accent is very good even when it isn’t?

(b) tell him that his poor accent doesn’t matter?

(c) tell him that his accent is very bad and that he must work hard to improve it?

3 How do you feel when a foreigner pronounces your name wrong?

(a) very angry

(b) it bothers me a little

(c) it bothers me a lot

(d) it doesn’t bother me at all

4 How do you feel when you meet a foreigner who speaks your language with a very good

accent?

(a) surprised

(b) pleased

(c) not surprised

(d) full of admiration

(e) don’t care or think about it

5 In the future, who will you speak English to?

(a) mostly English5 people visiting my country who don’t know my language

(b) mostly English-speaking people in this country (Britain, USA, etc.)

(c) mostly non-English6 people who don’t know my language and whose language I don’t know,

so that we speak English together

(d) don’t know

6 Do you think it is more important to have good pronunciation when:

(a) you are speaking English to English people?

4 presented in Kenworthy (1987, p. 54)5 We would suggest using „English-speaking“ people for all English native-speakers rather than „English people“ which we consider to stand only for the people from England.6 Similarly, we would suggest using „non-English speaking“ people for the same reason as in the previous footnote..

31

(b) you are speaking English to non-English people?

7 Below are some situations. When is it most important to pronounce well? Put them in order of

importance with a number if you want.

(a) speaking on the telephone

(b) meeting someone for the first time

(c) talking to someone you know very well (a good friend) in an informal situation (e.g. at a

party)

(d) doing business in English (e.g. at the bank, post office, bus station, railway station, in

shops, etc.)

(e) talking to strangers (e.g. asking the way)

(f) chatting to a fellow student (e.g. during break time)

To sum up, on the basis of this activity, the teacher

should receive information on the students‘ attitude

towards learning pronunciation which is helpful to design

the course according their own needs and goals.

3. 2. 2 Activity 2Another awareness building activity is "a tape-based

activity". It consists of a recording of a strongly

foreign-accented speech and following activities based on

either discussion of given questions, or also a table to be

filled in preceding further discussion. As an example,

Penny Ur (1991, p. 51) presents a box for the students to

work with:

WORKSHEET: RECORDINGS OF FOREIGN PRONUNCIATION

32

Speaker’s mother tongue:

Words/phrases mispronounced Define or describe the mistake

Aim: Increasing interest and motivation, stimulate

discussion of the implications for communication and social

interaction of heavily foreign-accented speech.

Resource: Kenworthy (1987, p. 55), Ur, P. (1991, p. 51)

Time: at least 45 minutes

Language used: Mother tongue is acceptable for meaningful

discussion

Procedure:

1 Introduce the task asking: "I'd like to know what you

think about this speaker?"

2 Play the tape (2 - 3 minutes) of foreign accents (people

who are not very proficient in the target language)

and let students fill in the box or write down notes

concerning the aspects which make the people's

performances sound foreign;

3 Play the tape as many times as needed;

4 Let the students compare their notes in pairs or groups;

5 Let them discuss the answers to the following questions:

1. What seem to be the most common errors?

2. How would you describe this person's accent?

3. Would you like to have a conversation with

this speaker?

33

4. (If we have a tape of a foreigner speaking

Czech) Do you dislike hearing your language

spoken in this way?

5. Why do you think the person pronounces so

badly/in this way?

(a) The person doesn't really care about

pronunciation.

(b) The person hasn't been told how bad

their accent is.

(c) It's very difficult to pronounce this

language well.

(d) The person hasn't been told about

pronunciation.

6 Whole-class discussion

Output: Identifying words which are very important and need

to be pronounced well. The teacher gets a general idea of

how responsive the learners are likely to be to

pronunciation work, both as a group and as individuals.

An equally useful component of the "evocation stage"

is eliciting information from the learners about their

previous pronunciation and English learning experience.

Which questions we prepare depends on our needs, but we

should know if they want to improve their pronunciation,

what they think their weak and strong points are and how

much they want to improve.

Getting this kind of information may be either

included in one of the "building-awareness techniques"

mentioned above, or done separately in the form of a

questionnaire or an informal discussion. It is beneficial

for the learners themselves as well as for the teachers.

The task makes the learners consider and define their

individual goals which itself has a motivating effect.

34

The „evocation stage“ should provide us with the input

information for the realization stage including setting out

goals and criteria of further testing and assessing,

deciding on the amount of time to be devoted to

pronunciation and the form of a test.

3. 3 Realisation

The main part of the realization stage may involve

teaching pronunciation and carrying out a particular test.

As this work does not focus on teaching pronunciation

itself, we are going to present various techniques of

testing. All we can find in literature are just recommended

activities which may be used for testing rather than any

strict rules and exact instructions, so the teacher is free

in their choice and may adjust any material from literature

according to their own individual needs or create some

themselves.

Before designing a test it is important to consider

the aim. Like in any other language component, there are

two kinds of tests. The first one is a progress test, which

monitors the students‘ progress. The second one – a

proficiency test - deals with the learners‘ general level

of a language skill without reference to any course, which

is presented in the international exams.

In the practice of pronunciation testing this division

between the two kinds of tests infers the division between

atomistic and holistic testing although it does not have to

obtain in all cases. But the progress test is usually based

on what has been taught during the course and in most

coursebooks the pronunciation teaching course design has

some structure and deals with the aspects of pronunciation

in successive steps. So it is obvious that the test will

only include the points which have been covered. The

proficiency test does not take into account what has been

35

taught and is designed only according to the general level

of the learner’s English. It examines intelligibility

including all aspects of pronunciation where the main

criterion is comfortable communication between, the speaker

and the listener.

Another way of dividing tests is whether they test

reception or production. Reception deals with hearing -

identification of sounds, stress and intonation in the

atomistic testing and listening comprehension in the

holistic one. The advantage of the reception testing is

that the result can be quantified simply by counting the

correct answers. That is also the reason for their

sufficient amount in course books. More difficult is

examining production since it cannot be measured and

quantified but is entirely dependent on the assessor’s

impression. We will deal with the tests on production and

point out the rules which should be followed in order to

receive as reliable results as possible.

We will now look at the activities which may be used

for testing. They have already been briefly presented in

the theoretical part from the point of view of the holistic

and atomistic way of assessing. In this part we are going

to introduce them more thoroughly and focus more on their

advantages and disadvantages in connection with their

particular purposes.

3. 3. 1 Limited-response activities

In limited-response activities the teacher has total

control of the learner’s utterance. These activities are

usually easier to prepare as well as assess and their

results are comparable with other learners.

Test 1 - Words in isolation

This test is aimed at examine the learners’ ability to

distinguish between sounds. It is advisable to start

36

pronunciation learning based on minimal pairs and correct

production of sounds.

This is just an example which may focus on a limited

number of sounds. Students read a list of words containing

minimal pairs. The assessment form shows which sound is to

be assessed and the assessor puts a tick or a cross next to

each word. The test can be done in one lesson in class, or,

what is more reliable, we can record the learners‘

performances and let them be assessed by more people or by

someone else, not the teacher of the class who has become

familiar with the students‘ accents.

Assessment form7

think thingpen pan

sick thick

ran rang

vest west

Test 2 - Pictures

For pupils who cannot read, we can use pictures for

identifying objects, where there in each picture there is a

possible source of confusion. For example, a picture of a

pen can cause a confusion with the word pan. Which pictures

we choose is dependent on the words which our learners

know. Since reading is not involved, the advantageous

feature of the activity is avoiding any “spelling

pronunciations” and thus its suitability for also dyslexic

students.

Test 3 - Words in sentences 8

7 Inspired by UR’s publication A Language Teaching (1997). 8 Inspired by HEATON’s Classroom Testing (1990).

37

Reading aloud sentences containing the problematic

sounds is preferable because they provide context and

reading will be more natural.

There were several people standing in the

hole/hall.Are you going to sail/sell your boat today?Do you like this sport/spot?

Test 4 - Reading Aloud

Reading aloud is one of the most common ways of

examining pronunciation. On the one hand, it provides

control, on the other hand there is still a big difference

between reading and normal conversation, so we have to

count on the fact that it is not spontaneous. Learners are

often confused by the spelling and as they see pauses

between words they find it difficult to link, so we should

be aware that in real conversation the performance would be

better. Nevertheless this kind of test may give us an

approximate level of the learners‘ pronunciation. It is

important to choose the right material to be read. The best

is to prepare something that may actually be read aloud. It

can be a letter, instructions or a dialogue.

The following example is a short letter or a message.

The student will read only the left side, but the assessor

may get also the points which should be assessed. It is

better to divide the text into more lines and assess only

one or two points per line, because it is impossible to

notice everything and give such a thorough feedback when

there are many students in the class. The result of such a

kind of assessment would be too chaotic, because in case of

more assessors, each of them will point out and omit

something else and the results received by this kind of

38

assessment do not enable us to compare students‘

performances.

TextIt’s my girlfriend’s birthday

next week and I want to give her

a surprise with a special present

or a special evening somewhere.

What should I do?

Examiner’s notes´gз:l,frend; bз:θdeı

wi:k;

wið ə;

or ə; i:vnıŋ;

falling intonation

The above mentioned activities were based on a limited

response. Considering their advantages, they are easy to

prepare and their assessment enables comparison between

students, as the text is the same for everyone.

Further, we are going to present free-response

activities in which there is not one correct answer but the

student is free in the choice of his or her answer.

3. 3. 2 Free-response activitiesFree-response activities do not test pronunciation

during reading, but the student’s own utterance. Student is

free to choose a formulation for his or her answer. On the

one hand, the advantage of this method compared to limited

response is avoiding “spelling pronunciations” and the

student’s speech generally sounds more natural. On the

other hand, we should take into account the fact that

nervousness of a learner may affect his or her performance

and for this reason a lot of practice is required

beforehand.

Test 5 - Re-telling stories

As the title tells us, this test involves re-telling a

story or an incident. A student reads the text silently and

then is supposed to re-tell it in their own words. If the

text is chosen appropriately, the student should use such

39

elements of pronunciation which we want to examine. The

reading comprehension should not affect the result in case

of inaccuracy on the summary.

Test 6 - Using pictures

Pictures are useful for testing speaking skills

because their description does not involve any reading,

which could affect the performance. There are various ways

how we can use them.

For a description students should have time to see

them beforehand for a few minutes to prepare and order

their thoughts.

The international Cambridge examinations include

pictures for comparison in their speaking activity. The

purpose of this activity is to find out the differences or

similarities and their common topic. This activity can be

done comfortably as a part of a normal lesson better than a

part of a test and it also enables more than one

participants to work together.

Another variation of using pictures is the sequences

of pictures which is similar to telling a story. Students

are to look for the links between the pictures rather than

describe pictures in isolation.

For either individual or pair work there are pictures

with speech bubbles. In this activity students are supposed

to guess what the people are saying.

Almost every coursebook includes a „giving directions“

task and maps can be used for this purpose as well.

3. 3. 2. 3 Testing intelligibilitySo far, we have mentioned two levels of pronunciation

testing. The first dealt with limited-response activities

which are suitable for testing all aspects of

pronunciation, but the use of stress and intonation is not

40

as natural as in normal conversation. The second type

provided us with activities more suitable for testing

stress and intonation. The third type is testing

pronunciation in interaction. This involves production as

well as comprehension and the main criterion is the

intelligibility of the learner’s speech. It is not so

detail-focused as the previous two but involves all aspects

of pronunciation – segmental and suprasegmental.

The activities are mostly interview-like, either it is

the interview with the examiner, or with the peers. For

this technique, pictures, role-plays or discussion may be

used, but so that the speaker’s performance is as

spontaneous as possible, it is importnat to choose a topic

which the learner is familiar with.

In testing speaking, pronunciation is just one of

several aspects which are evaluated and which make the

speech comprehensible. So, apart from accent, there is

grammar, vocabulary, fluency and comprehension which make

the picture of the student‘s performance complete.

This kind of testing is demanding as far as the

assessment is concerned. As there are many aspects

involved, setting out clearly defined criteria is very

important, moreover, one assessor only is not enough for

achieving relatively reliable results. But this technique

enables the learner to „show their best“ in pronunciation

and so the results could be considered to be the most valid

and reliable.

3. 4 Reflexion

Having presented variations of pronunciation tests, we

are coming to their final part which is the feedback. This

part will deal with assessment and the task of the

assessor. As has already been mentioned, giving feedback on

pronunciation is a very difficult task because it is

41

impression-based. Another stumbling block is the complexity

of this language component, so what should be taken in mind

first is the aim of a particular test and on its basis

setting out concrete criteria, otherwise the assessor will

get overloaded with much information at the same time.

Assessing the limited-response tests is probably the

easiest. As it focuses on a limited number of items, we may

just tick or cross the correct or wrong production of

sounds.

Free-response tests are more difficult to assess and

so it is useful to record the learners’ performances and

listen to them more times. In the atomistic assessment

there will be only a few criteria, because it is very time-

consuming to cover everything. In the holistic assessment

the criteria are differently defined as they include both

speaking and listening.

Description of criterial levels

Setting out criteria of assessment should refer to the

design of the test. Learners are likely to make more

pronunciation errors when reading than when speaking

spontaneously, but there are many students who would become

self-conscious and shy to speak on a topic, so it is

recommended to use both methods.

Hughes (1989) describes the criterial levels of

obtaining valid and reliable scoring as follows:

Scoring will be valid and reliable only if:

Clearly recognisable and appropriate descriptions

of criterial levels are written and scorers are

trained to use them.

Irrelevant features of performance are ignored.

There is more than one scorer for each

42

performance. (p. 110)

The first point to consider before setting out other

criteria is the standard according to which we are going to

judge the learners’ performances. Is it the native-speaker

standard? It could be, but we consider it to be a

discouraging goal for our learners as this is almost

impossible for them to achieve. Moreover, what we intend to

suggest, are the possible ways of testing and assessing

pronunciation in our environment which entails the

prevalence of Czech teachers of English over native-

speakers, so it might seem quite daring of the Czech

teachers to give our learners feedback in the way of

comparison with native-speaker. This implies that the

criteria of marking should be set out so that it is

possible for the learners to obtain the highest score which

can correspond with comfortable intelligibility.

From the didactic point of view, the progress should

be appreciated more than the achievement itself, as hard

work and willingness to improve is more important in life

than having just a good starting level – i. e. good

phonetic ability.

As far as the approach is concerned, the descriptions

may be atomistic or holistic. In order to reach reliable

scoring we may use both methods as a check. The atomistic

approach usually checks only a limited number of items,

e.g. production of particular sounds, stress, linking and

the result is either „correct, or „wrong“, whereas the

holistic focuses on the quality rather than the total of

correct answers. In the theoretical part, we have already

mentioned the scale in Cambridge examinations and now we

will look at another example – the American Foreign Service

Institute interview procedure. Two testers are required to

assess the candidates holistically and rate them on a six-

43

point scale for each of the following: accent, grammar,

vocabulary, fluency and comprehension. Both results then

should agree and as Hughes (1989) mentions there is a high

level of agreement between holistic and atomistic scoring.

As an example, we present the six-point scale for accent:

1. Pronunciation frequently unintelligible.

2. Frequent gross errors and a very heavy accent

make understanding difficult, require frequent

repetition.

3. „Foreign accent“ requires concentrated listening,

and mispronunciations lead to occasional

misunderstanding and apparent errors in grammar

and vocabulary.

4. Marked „foreign accent“ and occasional

mispronunciations which do not interfere with

understanding.

5. No conspicuous mispronunciations, but would not be

taken for a native speaker.

6. Native pronunciation, with no trace of „foreign

accent“. (Hughes, 1989, p. 111)

We may notice that the criteria above refer mainly to

how difficult the speaker is not only to be understood, but

also to listen to. It is not an easy task to classify a

performance into one of the groups and what makes this kind

of assessment even more demanding is the number of

assessors needed. But the issue of the scorers is going to

be tackled later on.

Irrelevant features of performance

In assessing pronunciation it is important to focus

only on such aspects or items of pronunciation which have

been decided to be tested, otherwise the results will not

44

be reliable, but chaotic. This is important to be stressed

particularly in pronunciation testing, as it is difficult

to ignore other errors in a performance. An example has

already been mentioned in Test 5 – Re-telling stories where

the attention should be paid to pronunciation, not to wrong

comprehension. And the same applies to the test of

distinction between the sounds: if [θ] and [ð] is tested,

the result should not be affected by wrong stress, for

example.

The scorer

Unlike assessing other language components, assessing

pronunciation has its unique requirements and demands on

the scorers.

The first problem is, whether the class teacher is the

best assessor. No publication or article being referred to

in this work took the view that the class teacher is

suitable for assessing their learners. It is difficult for

the teacher to fulfil this requirement, but still, there

are some possibilities.

We believe that, if necessary, the class teacher is

able to assess according to the atomistic criteria, meaning

that they are able to distinguish betweent the correct and

wrong production of sounds, stress and intonation. For that

reason, if necessary, the class teacher may be helpful. In

order to get more reliable results it is better to ask a

colleague from the school to assess for checking.

The holistic assessment is more complicated. Here, one

teacher is never reliable, if two have different opinions,

the result is also pointless, because we can count only the

points of agreement between both assessors and the rest

does not give us any information which implies that three

assessors are needed to provide some level of reliability.

45

To make the method even more difficult, Kenworthy

(1987, p. 20) claims that all English teachers themselves

are not suitable judges, as they have a great exposure to

non-native accents and so they have developed special

skills as listeners which make them atypical listeners and

so unsuitable. „The ideal judges are listeners who have not

had an abnormal amount of exposure to non-native speech nor

any previous contact with the speakers being assessed. […]

Judgements by teachers of English are of limited value.“

(Kenworthy, 1987, p. 20). At least the good news is

according to Kenworthy that non-native listeners can be

used as judges, e. g. other learners of English in the

class. We think that this requirement is very difficult to

fulfil in our school environment, so we should at least

stick to the rule of three scorers.

46

4 Research part

On the basis of the theoretical part a qualitative

research has been carried out in order to consider the

importance of pronunciation testing in EFL classes.

Our research tried to implement the three previously

mentioned stages: evocation, realisation and reflexion into

practice.

As theory often differs from practice, we were not

able to fulfil all the requirements, rules and

recommendations, but followed the opinion expressed at the

beginning of the practical part that any methods and

techniques are just recommended and are open to any

necessary adjustments in order to be administered according

to our individual needs and conditions which were limited.

Our limitations lay in the time possibilities and the

number of students at our disposal. The whole procedure was

carried out over a span of four months including all the

mentioned stages.

The whole procedure consisted of several parts. The

first part dealt with receiving and analysing the input

information using the „building-awareness activity“,

questionnaires and pre-research. The second part involved

the research itself including the first and the second

recording, pronunciation practice, and subsequent analysis

of the results. The third part dealt with assessment and

analysing the results of the three assessors.

There were sixteen students involved in the research

so we are aware of the fact that such a limited number does

not allow us to fully categorize our findings as far as the

progress is concerned.

For the assessment there were three assessors – non-

native teachers of English. One can argue that the choice

of non-native English teachers, who may be familiar with

47

foreign accent as they have a large exposure to it in their

own classes, will lead to unreliable results. There is a

point in this argument, however, for assessing in our

English classes our conditions will hardly be much

different, so our way of testing pronunciation should

correspond to the possibilities in our classes.

Our research will analyse two aspects of our testing.

The first is the feedback on the two performances of each

student including the results of the atomistic and holistic

testing. The second aspect to be analysed are the results

of the three assessors who were involved. On the basis of

our results we will infer the rate of agreement separately

in atomistic and holistic part.

4. 1 Plan and goals

Firstly, we will deal with the methods and results of

our research. In my classes, having introduced the new item

in our lessons – focused pronunciation learning and

consequent recording - I found out that most students

welcomed it as they found it challenging and only few

remained indifferent. Consequently, their performances

obviously reflected their attitude and motivation.

The second point to be discussed in this work deals

with the issue of the assessor. We will try to point out

what the concrete problems in assessing are, based on our

experience, and how much our assessors‘ results differ from

one another.

4. 2 Methods

48

At the beginning, it is important to mention that we

have been restricted only to what we had at our disposal.

The first item to be mentioned is the time which limited us

as far as both teaching, and recording are concerned. The

next important item is the equipment and space. We did not

have any recording studio, so the sound was not always

perfect. Nevertheless, we dare say that we did not have any

better or worse conditions than in most schools, so in

practice we can hardly count on having much different

conditions.

4. 3 Evocation and pre-research

49

The evocation stage consisted of several steps. Most

of the students had not had any experience with being

taught pronunciation and they seemed quite unfamiliar with

this language component. So the first task was to make an

attempt to motivate them to learn.

In my classes we used the building awareness activity

– questionnaire - presented in one of the previous

chapters. It can be discussed in class informally, learners

can fill it in and compare in pairs or groups before the

whole-class discussion. It is not necessary to carry out

the discussion in English, since learners should not be

restricted only to their own level of English, but the aim

for the learners is to become aware of the meaningfulness

of pronunciation learning. The questions in the activity

fall into two groups: the first part is aimed to find out

about the learner‘s own experience and the following set of

questions is aimed to stress the importance or need of

pronunciation. It is difficult to make a concrete

conclusion out of an informal discussion but I found it

contributing to show the pronunciation issue also from the

point of view of the listener. When imagining or evoking a

situation with a foreigner speaking Czech with a very

strong foreign accent, students agreed on the fact that it

may make the conversation difficult and very demanding. To

conclude our discussion, we reached the view that learning

pronunciation is as important for speaking as for listening

and that at least, basic knowledge of the English

pronunciation principles may improve also our listening

skills.

Having introduced the learners to our general goals

and made them think of the possible benefits of

pronunciation learning for themselves, we moved on to

gather information about our students concerning the

factors affecting pronunciation. We have decided again on

50

the questionnaire form eliciting information about the age

and the length of learning English, the current or any

previous exposure to English, the actual experience with

learning pronunciation and their personal goals concerning

personal opinions of their strong points and weak points

which they would like to practise. We are going to describe

the input information which we consider to be relevant to

get the overview of the age and the possible influence of

other factors.

Level and age

The students differed in the level of English as well

as in the age at which they began to learn English.

As far as the level of English is concerned, students

belonged to three groups: A2, B1 and B2. The age at which

they began learning English ranged from 6 to 50, so the

whole group was very heterogeneous. Seven out of sixteen

students began learning English in adulthood and the rest –

nine students – before.

Exposure to English

None of the students reported using English out of the

classroom or having been to an English-speaking country for

more than a few weeks, so this factor can be considered as

insignificant.

Native language

The native-language factor will be mentioned later on

in connection with the problematic sounds. All students

were Czech and therefore in most cases the problems with

production of sounds, stress and intonation were similar

within the whole tested group.

51

Information on the learners‘ phonetic ability,

attitude and motivation is difficult to describe. But we

are aware of the fact that these factors are very

imprortant and that is also why we used the evocation phase

before the research itself.

Pre-research

Prior to starting the main research part, a pre-

research had been carried out in order to gather

information of each learners’ pronunciation and on the

basis of this a decision was made on setting out the goal

and the criteria for our further steps.

The pre-research part was a reading-aloud task. The

learners were given a text to read. This was a short

dialogue. They were given time to read the text silently

before the recording.

A Are you working at the moment?B Yes, I’ve got a job with a record company called the Mad Cat.A Really? Is that normal working hours?B Yeah, nine to five.A Do you share an office with other people?B Uhuh, there are three of us.A And do you get on well?B Yes, we have a good working relationship.(Natural English. Pre-intermediate. OUP9)

The recordings were given to three assessors, who were

asked to write down their first impression on the

performances in an unstructured way. In the pre-research no

strict criteria were given as their setting out was

supposed to be the result of the procedure.

Here is the example of one assessment given by one

assessor:

Student 14- links quite well (apart from ‘three of us’)- no weak forms- recognizable w, velar n; recognizable difference in there vs three

9 http://fds.oup.com/www.oup.com/word/elt/products/nepre_ts11.doc?cc=cz [online; quoted 16. 2. 2007]

52

- schwa – short, long – not clear- mad – æ is not open enough- no aspiration

To analyse the data received by the three assessors, I

used a table in which all the mentioned strong and weak

points were written down. The pluses stand for the aspects

which were considered to be pronounced correctly and the

minuses for those considered wrong. Further on, those

points on which at least two assessors agreed were counted

and the results were analysed.

Pre-research: results

S1

S2

S3

S4

S5

S6

S7

S8

S9

S10

S11

S12

S13

S14

S15

Linking + - -

- - -

+ + +

+ - +

+ + +

+ + +

- + -

+ - -

- - -

- + +

- - -

- - -

+ - -

- - -

- - -

[w], [v] + + - - + + + + + + - - + + + + + + - + + - - - -

- - - - - - -

[ł] - - + - - - - - + - - - + - -

[ŋ] + - - - + + - + + - + + + -

+ - - + - + + - - - - + -

++ - + - -

[θ] + - + - - + + + -

+ + - - + - - + - - - - - - - - -

[ð] + - + - - + + -

++ - + + - - - - + -

+ - - - - - - -

[ә] + - + + + - - - - - - - -

[З:] + - - + - + + + - + - - - - - -

[æ] - - - - - - - - - - - + - - - + - - - - - - - - -

Aspiration - - - - + - + - - - +

Aspect of pronunciation

Number of students who pronounced wrongly

Number of students who pronounced correctly

The order of the most problematic areas

Linking - 10 + 5 1[w], [v] - 7 + 8 3[ł] - 0 + 0 9[ŋ] - 5 + 6 5

53

[θ] - 6 + 2 4[ð] - 4 + 4 6[ә] - 1 + 0 8[З:] - 2 + 1 7[æ] - 9 + 0 2Aspiration - 0 + 0 9

According to the table the most problematic area for

the students was considered to be linking, production of

[æ] and [w], [v]. But the number of other wrongly

pronounced sounds was also not insignificant. Therefore,

the conclusion and the goal of our further steps was

decided to be linking on the first place and practising

some of the problematic sounds using exercises on minimal

pairs.

4. 4 RealisationAccording to the results of the pre-research, we have

designed a test consisting of several parts. The first two

parts applied the atomistic approach, the second three

parts holistic. The atomistic method of testing examined

the production of problematic sounds in minimal pairs and

linking in short phrases. All the students‘ performances

were recorded and given to three assessors to be evaluated.

In the holistic testing we used a reading-aloud

activity. Students were given time to read the test

silently before the recording.

Test on Pronunciation 1

sick thickfree threeden thenran rangsink singvest westvet wetpen panmen man

54

drink a cup of teaput it onAre you in the same place?Don’t crash into anything!read a booka big umbrellagoing outThere isn’t any doctor.I couldn’t do anything about that.The story is very interesting.

I am writing to ask for information about your language courses. I am especially interested in an intensive course of two or three weeks. I am thirty-one and I work in the library at Milan University. I can read English quite well but I need to improve my listening and speaking.

I have looked at your website, but there is no information about intensive courses next summer. Could you please send me information about dates and prices? I would also like some information about accommodation. If possible, I would like to stay with a family. My wife is going to visit me for a weekend when I am at the school. Could she stay with me in the same family?

Consequently, each of them was recorded reading aloud

all the three parts. The recordings were given to three

assessors to be evaluated and the results from the first

recording were given to students with comments on their

strong and weak points and suggestions for their further

pronunciation practice.

After three months of practice, students were

recorded again reading the same format of the test which

examined the same pronunciation points, just different

words for minimal pairs, phrases for linking and text were

chosen. The whole procedure was aimed to be the same and so

were the criteria of assessing. The test was supposed to

find out about the students‘ progress.

Test on pronunciation 2

veil whalevent wentthink sinkday theybeg bagsend sandsun sung

55

sin singclosing clothing

never againI saw it.Why am I leaving?may askHe’s quite old.An American car.Good evening!for everAre you enjoying it here?The shop isn’t open yet.too expensiveMind your own business.Thursday evening

For me the first good thing about the weekend is that I don’t have to go to work. I like my job, but I have to spend all day inside, in an office, and I’m a person who loves being outside. Another good thing is that I don’t have to get up early. During the week I have to get up at half past six every day. It’s not too bad in the summer but I hate it in the winter when it’s dark in the morning. But above all, I like the weekend because I have time to do all the things I really enjoy doing, like listening to music, reading, or going out with friends.

4. 4. 1 Atomistic testing resultsThe atomistic testing results consist of the

minimal-pair part and the part of linking. Each examined

point was evaluated by the three assessors who either put a

tick or a cross next to each point, according to whether

they considered the points to be rather correct or wrong.

We use the word „rather“ because in many cases it was

difficult to decide on the correctness as not all the

sounds were produced absolutely right or wrong. Therefore,

the assessors‘ evaluation reflected the quality of the

examined point which in many cases was not perfectly clear.

Minimal pairs

In testing minimal pairs, students were given

eighteen words – nine minimal pairs, examining the

students‘ problematic sounds. Such sounds were those, which

had been most often mispronounced in the pre-research: [v],

[w]; [s], [θ]; [d], [ð]; [e], [æ]; [n], [ŋ]. The table

below presents the results of the two recordings. In the

second and the fourth column, there is the total of correct

56

answers, i. e. answers which were ticked (considered

correct) by at least two assessors. The next columns show

each student’s percentage obtained in the test and the last

two columns refer to the differences between the first and

the second recording.

Student 1st recordingTotal: 18

% 2nd recordingTotal: 18

% Difference in correct answers

Difference (%)

A2/1 17 94.4 18 100 +1 +5,6A2/2 12 66.7 13 72,2 +1 +5,5A2/3 14 77.8 15 83,3 +1 +5,5

A2/16 11 61.1 15 83,3 +4 +22,2A2/9 17 94.4 15 83,3 -2 -11,1

B1/10 10 55.6 15 83,3 +5 +27,7B1/11 16 88.9 16 88,9 0 0B1/13 13 72.2 18 100 +5 +27,7B1/14 12 66.7 14 77,8 +2 +11,1B1/15 14 77.8 12 66,7 -2 -11,1B1/18 8 44.4 12 66,7 +4 +22,3B2/6 17 94.4 18 100 +1 +5,6B2/8 12 66.7 16 88,9 +4 +22,2

B2/17 13 72.2 17 94,4 +4 +22,2

Over the three months of practice, all students

apart from three improved by 16% on average. Nevertheless,

the assessors reported that in spite of such improvement in

this part, they had not noticed such significant progress

in the production of sounds in holistic part.

Linking

The test on linking was evaluated in the same way as

the test on minimal pairs. Students were given about

thirteen phrases in which fifteen points on linking were

tested. The assessors ticked or crossed all the points

according to their impression.

57

Student 1st recordingTotal: 15

% 2nd recordingTotal: 15

% Difference in correct answers

Difference (%)

A2/1 5 33.3 12 80 +7 +46.7A2/2 2 13.3 10 66.7 +8 +53.4A2/3 10 66.7 13 86.7 +3 +20

A2/16 4 26.7 6 40 +2 +13.3A2/9 0 0 6 40 +6 +40

B1/10 9 60 14 93.3 +5 +33.3B1/11 7 46.7 10 66.7 +3 +20B1/13 7 46.7 13 86.7 +6 +40B1/14 3 20 10 66.7 +7 +46.7B1/15 3 20 10 66.7 +7 +46.7B1/18 2 13.3 5 33.3 +3 +20B2/6 15 100 14 93.3 -1 -6.7B2/8 4 26.7 7 46.7 +3 +20

B2/17 4 26.7 12 80 +8 +53.3

As the result chart shows us, all students apart

from one improved significantly. The progress was by about

35% on average.

Conclusion on atomistic testing

The results obtained show improvement in both parts

of atomistic testing in most students. The improvement in

linking was more significant than in minimal pairs. This

may be caused by the fact that minimal pairs containing

problematic sounds are more difficult for students to learn

as they are often completely new to students, whereas in

linking students are supposed to use what they already can

just in a different way. The rule for linking is quite

simple for them and all they need is just concentrate on

its application.

4. 4. 2 Holistic testing resultsThe task for students in the holistic test was

reading aloud a short text.

58

The holistic assessment was divided into five parts:

overall impression, comprehensibility, production of

sounds, stress and intonation. All of these five parts were

evaluated according to a five-point scale in which each

point was defined in the assessor‘ sheet. The assessors

were supposed to use the five-point scale for each part

separately in accordance with their impression of the whole

performance.

Overall impression

The overall impression examined how natural or

difficult to understand the student’s performance was. The

scale for the assessors was the same as in the oral part10

of the Cambridge examination:

How does the performance sound to you?

1) pronunciation is heavily influenced by L1 features and

at times be difficult to understand

2) pronunciation is generally intelligible, but L1 features

may put a strain on the listener

3) although pronunciation is easily understood, L1 features

may be intrusive

4) L1 accent may be evident but does not affect the clarity

of the message

5) pronunciation is easily understood and prosodic features

are used effectively; many features, including pausing and

hesitation, are „native-like“

The assessors‘ task was to evaluate this part

without looking at the text which the students were

reading. In this part the marks of all assessors were

included in the results regardless of whether they differed

10 Paper 3: Speaking. Cambridge Common Scale for Speaking [online; quoted 25. 1. 2007] http://www.cambridgeesol.org/support/dloads/ket/KET_HB_sampleS.pdf

59

or not. The average mark was counted out of the three. The

table below shows the average mark of each performance from

the first and the second recording as well as the

difference between them. „Plus“ stands for improvement,

„minus“ for worse performance.

StudentOverall

impression 1

Overall impression

2Difference

A2/1 2.7 3.3 +0.6A2/2 2 3 +1A2/3 4 3.7 -0.3A2/4 3.3 - -

A2/16 2.3 2.7 +0.4A2/9 2.3 2 -0.3

B1/10 2.7 2.7 0B1/11 2.3 2.3 0B1/13 2 2.7 +0.7B1/14 2 2.7 +0.7B1/15 2.3 2.7 +0.4B1/18 1.7 2.3 +0.6B2/6 4.7 5 +0.3B2/7 3.3 - -B2/8 3.3 4 +0.7

B2/17 3.3 3.7 +0.4

The results showed slight improvement within one

level in most students and one student obtained the highest

mark – native-like performance.

Comprehensibility

The comprehensibility part tested how much of the

performance was understandable for the assessors. In this

part the assessment was based again only on hearing without

reading the text. The scale for assessment was given as

follows:

How much do you understand?

1) not most of the performance

2) not quite many words/phrases

3) not a few words/phrases

4) everything with difficulties

60

5) everything

Student Comprehensibility 1

Comprehensibility 2 Difference

A2/1 4.3 4.7 +0.4A2/2 3.7 4.3 +0.6A2/3 5 5 0A2/4 3.7 - -

A2/16 4.3 4.7 +0.4A2/9 3.3 4 +0.7

B1/10 3.7 4.3 +0.6B1/11 3.7 4 +0.3B1/13 4.3 4 -0.3B1/14 3.7 3.7 0B1/15 4 4.3 +0.3B1/18 2.7 4.3 +1.6B2/6 5 5 0B2/7 5 - -B2/8 4.7 5 +0.3

B2/17 5 5 0

Also in this part there was a slight improvement in

most students and the comprehensibility compared to the

overall impression has brought much higher scores for more

students which infers that despite a lower score in the

overall impression, the comprehensibility can still be

considered perfect.

Production of sounds, stress and intonation

The last three parts were assessed at the same time

and the assessors were asked to mark their overall

impression of the production of sounds, stress and

intonation according to the following scale:

1) Bad

2) Quite good

3) Good

4) Very good

5) Excellent

61

StudentProduction of sounds

1

Production of sounds

2Difference

A2/1 2.7 3.7 +1A2/2 2.7 3 +0.3A2/3 4.3 4 -0.3A2/4 3 - -

A2/16 2 3 +1A2/9 2.7 2.7 0

B1/10 2.7 3 +0.3B1/11 2.7 2.3 -0.4B1/13 3.3 3 -0.3B1/14 2 2.7 +0.7B1/15 2.3 3 +0.7B1/18 2 3 1B2/6 5 4.7 -0.3A2/7 2.3 - -B2/8 3.3 3.7 +0.4

B2/17 3.7 4.3 +0.6

As the production of sounds was more focused than

the overall impression and the comprehensibility, the marks

for this aspect were usually lower. But again, the results

showed slight improvement in most cases.

Student Stress 1 Stress 2 Difference

A2/1 2.7 3 +0.3A2/2 2.3 3 +0.7A2/3 3.3 3 -0.3A2/4 3 - -

A2/16 1.6 2.3 +0.7A2/9 2 1.7 -0.3

B1/10 2 2.3 +0.3B1/11 1.7 2.3 +0.6B1/13 2.3 2.7 +0.4B1/14 1.7 2.3 +0.5B1/15 1.7 2.7 +1B1/18 1.3 2.3 +1B2/6 4 4.7 +0.7A2/7 3 - -B2/8 2.7 3 +0.3

B2/17 2.3 3.7 +1.4

62

The assessment of stress showed improvement in all

students apart from one. Stress compare to the production

of sounds showed worse results in most cases than in the

test on production of sounds. We suppose that the reason

may be too much concentrating on fluent reading and correct

production of sounds and linking, as these aspects had been

involved predominantly in the previous practice.

Student Intonation 1

Intonation 2 Difference

A2/1 2 2.3 +0.3A2/2 2 2 0A2/3 2.7 3.3 +0.6A2/4 3.7 - -

A2/16 1.6 2 +0.4A2/9 2 1.7 +0.3

B1/10 2 2 0B1/11 1.7 2 +0.3B1/13 1.7 2 +0.3B1/14 1.7 2 +0.3B1/15 1.7 2.7 +1B1/18 1.3 1.7 +0.4B2/6 3.3 4 +0.7A2/7 3.3 - -B2/8 2.7 2.7 0

B2/17 3 4 +1

Out of all the previous parts, the test on

intonation proved the lowest scores. The reason may be the

same as in the test on stress. Moreover, the assessors

reported that they had found this part most difficult to

assess, as it is almost impossible to concentrate only on

intonation and they usually noted down their first

impression although there may have been other factors which

have distorted the evaluation.

Student Overall impression Comprehensibility Sounds Stress Intonation Total

A2/1 +0.6 +0.4 +1 +0.3 +0.3 +2.6A2/2 +1 +0.6 +0.3 +0.7 0 +2.6A2/3 -0.3 0 -0.3 -0.3 +0.6 -0.3

63

A2/16 +0.4 +0.4 +1 +0.7 +0.4 +2.9A2/9 -0.3 +0.7 0 -0.3 +0.3 +0.4

B1/10 0 +0.6 +0.3 +0.3 0 +1.2B1/11 0 +0.3 -0.4 +0.6 +0.3 +0.8B1/13 +0.7 -0.3 -0.3 +0.4 +0.3 +0.8B1/14 +0.7 0 +0.7 +0.5 +0.3 +2.2B1/15 +0.4 +0.3 +0.7 +1 +1 +3.4B1/18 +0.6 +1.6 +1 +1 +0.4 +4.6B2/6 +0.3 0 -0.3 +0.7 +0.7 +1.8B2/8 +0.7 +0.3 +0.4 +0.3 0 +1.7

B2/17 +0.4 0 +0.6 +1.4 +1 +3.4

Conclusion on holistic testing

According to the results in this part, most students

progressed in all five parts. The average highest score was

reached in the comprehensibility part which can be

considered good news as, for communication, it is more

important than the way someone’s speech sounds.

Conclusion on both – atomistic and holistic - parts

The results of both parts did not prove any

significant link between the atomistic and the holistic

part. This may be caused by a small number of students

tested or the fact that many students were able to learn to

produce the problematic sounds and linking in words or

short phrases. Accordingly, the progress in the atomistic

test was significant. But the holistic test revealed that

many students were not able to apply the items from the

atomistic test. Some may have achieved a very high score in

the minimal pairs and linking, but still the holistic test

showed wrong pronunciation of these items.

To sum up, according to our results, the

improvements in atomistic test may be a question of short-

term practice, whereas we consider any progress in holistic

testing a long-term matter as more factors are involved

like subconscious acquisition within some time and the

attitude and motivation of the learner.

64

4. 5 ReflexionThe last stage of our research dealt with the issue of

assessing. The feedback reflecting the students‘ results

has already been mentioned in the previous chapter.

In this chapter we will explain the reasons for our

choice of the assessors in reference to the criterial

suggestions presented formerly and analyse the results of

the assessors in the atomistic and holistic parts of the

test. The question that we would like to find the answer to

is whether according to our results our choice of assessing

methods can provide us with somehow reliable, nevertheless,

impression-based results which may help us in further

teaching English pronunciation.

Firstly, we will deal with the choice of assessors.

Having followed the rule of more than one assessor for each

performance we have opted for the number of three assessors

in case of different results of the two assessors. For

practical reasons three non-native teachers of English were

chosen. According to some authors this decision may not be

the ideal one, but not everybody opposes it. We think that

it was the best option which we had at our disposal and

also the easiest option when testing pronunciation in

school. Hardly ever can we count on a sufficient number of

native speakers in our schools, therefore, in our opinion,

we have administered an assessing method which we consider

to be the most likely applied in school practice in

testing pronunciation.

The most difficult task for us to tackle was setting

out concrete criteria for the assessment and following

them. We will see later how difficult it was for the

assessors to decide whether, e. g. the particular sound was

produced correctly or wrongly and decide on the level in

holistic assessment.

65

The last point supporting reliability is closely

connected to the previous one and means that any irrelevant

features of a learner’s performance which have not been set

as criteria should be ignored.

The assessors have been instructed according to the

suggestions mentioned above and in the next chapter we will

look at their agreeable or different results of assessment.

4. 5. 1 Assessors‘ agreement in atomistic testingThe test intended for the atomistic assessment

consisted of two parts. The first was aimed to examine the

students‘ recognition between problematic sounds and their

native-language nearest equivalents and the second part

focused on linking. The assessors were instructed to put a

tick or a cross next to each examined word or phrase and to

evaluate only the relevant features. This at first

obviously concrete criterion turned out to be difficult to

fulfil. The table below presents the results of one hundred

percent agreement of the three scorers.

Atomistic TestingPart 1 - Minimal pairs

Student

Assessors‘ 100% agreement

(number of the same answers out

of 18)

Percentage of assessors‘ 100%

agreement(%)

Assessors‘ 100% agreement

(number of the same answers out

of 18)

Percentage of assessors‘ 100%

agreement(%)

1st recording 2nd recording1 15 83,3 15 83.32 12 66,7 15 83.33 15 83,3 14 77.84 14 77,7 - -

16 16 88,9 13 72.29 13 72,2 16 88.96 18 100 18 1007 14 77,7 - -8 10 55,6 17 94.4

17 11 61,1 15 83.310 9 50 14 77.811 14 77,7 14 77.8

66

13 11 61,1 14 77.814 12 66,7 12 66.715 16 88,9 15 83.318 12 66,7 14 77.8

Assessors‘ agreement in total74%

Assessors‘ agreement in total82%

As we can see the total of assessors‘ 100% agreement

is 74 % in the first recording and 82% in the second one.

Their agreement varied on the scale between 50% and 100%

which is quite a significant difference. Even though the

criteria for assessment had been explained and all the

assessors had discussed them beforehand, they reported that

what they had found the most difficult was such a

production of a sound which they felt was not completely

correct but at the same time not completely wrong and

therefore some inclined to consider it correct and some

wrong. The next problem was the correct production of the

particular sound on the one hand, but mispronunciation of a

different part of the word or phrase which sometimes led to

the impression that the word had been pronounced wrongly.

The second recording compared to the first one showed

better results in the assessors‘ agreement. According to

the assessors, this is because as most students‘

pronunciation got better, more sounds were clearer and thus

easier to assess.

Atomistic TestingPart 2 - Phrases on Linking

 

Student

Assessors‘ 100% agreement

(number of answers out of

15 )

Percentage of assessors‘ 100%

agreement(%)

Assessors‘ 100% agreement

(number of answers out of

15)

Percentage of assessors‘ 100%

agreement(%)

1st recording 2nd recording1 8 53,3 12 802 13 86,7 10 66.7

67

3 7 46,7 9 604 8 53,3 - -16 14 93,3 5 33.39 15 100 10 66.76 11 73,3 14 93.37 10 66,7 - -8 9 60 7 46.717 7 46,7 12 8010 4 27 11 73.311 10 66,7 14 93.313 8 53,3 12 8014 13 86,7 10 66.715 10 66,7 10 66.718 15 100 9 60

Assessors‘ agreement in total67.5%

Assessors‘ agreement in total69%

The assessors reported the same reasons for different

evaluation in this test as in the first one. The results

differed mainly because of omitted and changed words in the

phrases so that the phrase lost its sense, however, the

student used linking correctly.

When comparing the agreement in assessment between the

first and the second recording, we come to very similar

numbers,

The impression-based assessment is highly subjective

and the „either-or“ evaluation can never be completely

reliable. The „tick“ or „cross“ assessment is too

restricted to such a subjective method of assessment. In

our opinion, the impression-based assessment should not be

restricted only to tick and cross answers as it has much

wider scale.

Nevertheless, according to the results in the table,

we are convinced that at least three assessors are

necessary to balance the impression of only one.

68

4. 5. 2 Assessors‘ agreement in holistic testing

In the holistic assessment the three scorers were

supposed to evaluate the reading-aloud task according to

the five-point scale in which the mark 5 represented the

highest score. As we have already stated that the standard

for the holistic assessment should not be native-like

performance, but comfortable intelligibility, the results

varied on the scale from the lowest to the highest scores.

The assessors‘ task was to choose such a definition of the

recorded performance which represented their impression

best. The table below shows the rate of agreement of the

three assessors in the five parts of holistic testing.

Although the categories Production of sounds, Stress and

Intonation tend to belong to the atomistic part, we decided

to put them at the end of the holistic test as the task for

the assessors was to give a mark for the overall impression

of these three categories.

Holistic Testing (the figures stand for the number of students out of 16)

Overallimpression Understandability Production

of soundsStress Intonation

100% agreement 4 4 4 1 0

2/3 agreement 12 10 11 10 10

Different results 0 2 1 5 6

Apart from the reasons for different assessments of

the scorers, we have come to another factor which

influenced the holistic assessment. The assessors reported

that in spite of good production of sounds, stress and

69

intonation it was often the fluency and the speed of one’s

reading which had a disruptive effect. A lot of pausing and

hesitating in the performance often made the impression and

thus the final mark worse.

Conclusion on the agreement in assessment

As we have seen, the agreement of the assessors was

not absolute in any of the parts. This comes out of the

impression-based assessment. Except for the criteria given

to the assessors, we have to count on each assessor having

his or her subjective criteria which are included in their

assessing. We have mentioned that in the atomistic test the

choice between „tick“ or „cross“ is too limited as we found

out that it was the quality of the tested item which was

considered rather correct or wrong. If the item was

produced rather well for at least two of the three

assessors it was considered correct. Due to the differences

in the three scorers‘ assessments, we assume that three

scorers are more reliable than only one as more scorers

offer more opinions and thus balance the subjective view of

only one impression. This conclusion has its relevance to

the holistic assessment as well.

70

Conclusion

The aim of this work was to present various testing

activities and ways of their assessing.

We believe that pronunciation as one of language

components should be included in any course design.

Moreover, structured pronunciation teaching should involve

giving structured feedback to learners as it simply makes

the whole process complete for both sides.

Pronunciation testing as one of the ways of feedback

presupposes setting out goals which help the learners be

aware of what is aimed to be achieved, therefore, the

practice itself becomes more meaningful for them. In

addition, structured pronunciation teaching is beneficial

in the way that it helps the learners build the general

awareness of the system of pronunciation as an important

language and communication component.

In our opinion, setting out goals and subsequent

working on their achieving, comparing results within time

and class is a significant motivation factor itself which

may enhance the whole process.

Arranging the right conditions will enable our

students to achieve their best. We admit that many of the

testing methods which have been presented are too time

demanding to be administered in large classes, however,

mostly not more than, for example, a correction of a

written assignment. Another problem could arise with the

number of assessors. On the one hand, it is assumed that

the number of three is supporting the reliability of the

assessment, on the other hand, this is just a recommended

procedure. The only recommendation is to try to follow the

advice given and adjust the activities to the possibilities

we have at our disposal. Any feedback which is aimed to

help our learners and which is accomplished thoughtfully

71

and responsibly is better than none. To compare, we often

assess spoken or written work of our learners, which is

also subjective and impression-based, therefore,

pronunciation is not as exceptional as it might seem.

In my own experience, I have found pronunciation

activities and the whole research procedure particularly

contributing as far as not only speaking and listening

skills of my students are concerned, but also their

attitude to many other activities involving reading,

speaking and listening. At the beginning of the testing

process, the students did not seem particularly

enthusiastic, but having received their first assessment

with their strong and weak points, information on the

overall impression and comprehensibility, their further

practice began to be more focused and motivated to achieve

higher scores in the following recording. This was a

turning point in our pronunciation practice, because it is

natural that people are eager to know about how they are

proceeding and consequently, willing to improve. I dare say

that they have become slightly less „scared“ of the

listening tasks as they are aware of the existence of weak

forms, linking and other aspects of pronunciation which

used to make them confused and discouraged from

concentrating before the actual listening.

In my opinion, the good point about pronunciation

teaching in general is that it may be a great time-filler

providing enjoyable activities, great grammar and

vocabulary drills and on the top of it, it can be used as a

meaningful leisure after any concentration-demanding

activity since it does not necessarily require too much

strain and attention.

The suggestion which should be followed in all

circumstances so as to motivate the learners and progress,

is maintaining nice atmosphere, encouragment and giving a

72

chance to everyone to experience their own success and

achieve their best.

73

Appendix 1

Assessor:

Instruction on assessment Student: ______________

Atomistic Testing

Part 1 – Minimal PairsPut a tick or a cross according to the correctness in pronunciation.

sick thickfree threeden thenran rangsink singvest westvet wetpen panmen man

Part 2 – Phrases on LinkingPut a tick or a cross according to good or bad linking.

drink a cup of tea k pput it on t tAre you in the same place? wDon’t crash into anything! ʃ wread a book da big umbrella ggoing out ŋThere isn’t any doctor. tI couldn’t do anything about that. w ŋThe story is very interesting. j j

Holistic Testing

1st listening: Do not read the text.How does the performance sound to you?Scale: natural -------------------------------------------------- very difficult to understand

5 4 3 2 11) pronunciation is heavily influenced by L1 features and at times be difficult to understand2) pronunciation is generally intelligible, but L1 features may put a strain on the listener3) although pronunciation is easily understood, L1 features may be intrusive4) L1 accent may be evident but does not affect the clarity of the message5) pronunciation is easily understood and prosodic features are used effectively; many features, including pausing and hesitation, are „native-like“

2nd listening: Do not read the text.How much do you understand?1) not most of the performance2) not quite many words/phrases3) not a few words/phrases4) everything with difficulties5) everything

74

Part 3 – Reading Aloud

I am writing to ask for information about your language courses. I am especially interested in an intensive course of two or three weeks. I am thirty-one and I work in the library at Milan University. I can read English quite well but I need to improve my listening and speaking.

I have looked at your website, but there is no information about intensive courses next summer. Could you please send me information about dates and prices? I would also like some information about accommodation. If possible, I would like to stay with a family. My wife is going to visit me for a weekend when I am at the school. Could she stay with me in the same family?

3 rd listening :Write down words/phrases you found difficult to understand.……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………

4th listening:Scale: (1) Bad – (2) Quite Good – (3) Good – (4) Very Good – (5) ExcellentEvaluate the student’s:

production of sounds 1 2 3 4 5stress 1 2 3 4 5intonation 1 2 3 4 5

Further Notes:

75

Appendix 2

Assessor:

Instruction on assessment Student: __________

Atomistic Testing

Part 1 – Minimal PairsPut a tick or a cross according to the correctness in pronunciation.

veil whalevent wentthink sinkday theybeg bagsend sandsun sungsin singclosing clothing

Part 2 – Phrases on LinkingPut a tick or a cross according to good or bad linking.

never again rI saw it. wWhy am I leaving? j mmay ask jHe’s quite old. tAn American car. nGood evening! dfor ever rAre you enjoying it here? w ŋThe shop isn’t open yet. p ttoo expensive wMind your own business. rThursday evening j

Holistic Testing

1st listening: Do not read the text.How does the performance sound to you?Scale: natural -------------------------------------------------- very difficult to understand

5 4 3 2 11) pronunciation is heavily influenced by L1 features and at times be difficult to understand2) pronunciation is generally intelligible, but L1 features may put a strain on the listener3) although pronunciation is easily understood, L1 features may be intrusive4) L1 accent may be evident but does not affect the clarity of the message5) pronunciation is easily understood and prosodic features are used effectively; many features, including pausing and hesitation, are „native-like“

2nd listening: Do not read the text.How much do you understand?a) everythingb) everything with difficultiesc) not a few words/phrasesd) not quite many words/phrasese) not most of the performance

76

Part 3 – Reading Aloud

For me the first good thing about the weekend is that I don’t have to go to work. I like my job, but I have to spend all day inside, in an office, and I’m a person who loves being outside. Another good thing is that I don’t have to get up early. During the week I have to get up at half past six every day. It’s not too bad in the summer but I hate it in the winter when it’s dark in the morning. But above all, I like the weekend because I have time to do all the things I really enjoy doing, like listening to music, reading, or going out with friends.

3 rd listening :Write down words/phrases you found difficult to understand.……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………

4th listening:Scale: (1) Excellent – (2) Very Good – (3) Good – (4) Quite Good – (5) BadEvaluate the student’s:

production of sounds 1 2 3 4 5stress 1 2 3 4 5intonation 1 2 3 4 5

Further Notes:

77

Bibliography

ALDERSON, C. J., WALL, D. & CLAPHAIM, C. Language Test Construction and Evaluation. Cambridge: Cabridge University Press, 1996. 310 p.

BACHMAN, L. F. Fundamental considerations in language testing. Oxford: Oxford University Press, 1990. 408 p.

BACHMAN, L. F., PALMER, A. S. Language testing in practice. Oxford: Oxford University Press, 1996. 377 p.

BAKER, A. Ship or sheep. 2nd edition. Cambridge: Cambridge University Press, 1981. 168 p.

BAKER, D. Language testing. London: Edward Arnold, 1989. 110 p.

DALTON, C., SEIDLHOFER, B. Pronunciation. Oxford: Oxford University Press, 1994. 191 p.

GAIRNS, R., REDMAN, S. Natural English. Pre-intermediate. Student’s book. 1st edition. Oxford: Oxford University Press, 2005. 176 p.

GILBERT, JUDY B. Speaking Clearly. Student’s Book. 1st edition. Cambridge: Cambridge University Press, 1990. 136 p.

GIMSON, A. C., CRUTTENDEN, A. Gimson’s Pronunciation of English. Sevenoaks: Arnold, 1996. 304 p.

HANCOCK, M. English Pronunciation in Use. 1st edition. Cambridge: Cambridge University Press, 2003. 200 p.

HEATON, J. B. Classroom Testing. London: Longman, 1990. 127 p.

HEATON, J. B. Writing English Language Tests. 1st edition. London: Longman, 1988. 160 p.

HUGHES, Arthur. Testing for Language Teachers. 1st edition. Cambridge: Cambridge University Press, 1989. 172 p.

JENKINS, J. The Phonology of English as an International Language: new models, new norms, new goals. 1st publ. Oxford: Oxford University Press, 2000. 258 p.

KELLY, G. How to teach pronunciation. 1st edition. Harlow: Longman, 2000. 154 p.

KENWORTHY, J. Teaching English Pronunciation. Harlow: Longman, 1987. 164 p.

McNAMARA, T. Language testing. 1st edition. Oxford: Oxford University Press, 2000. 140. p.

78

MADSEN, S. H. Techniques in Testing. New York: Oxford University Press, 1983. 212 p.

OXENDEN, C., LATHAM-KOENIG, C., SELIGSON, P. New English File. Pre-intermediate. Student’s book. 1st edition. Oxford: Oxford University Press, 2005. 159 p.

ROACH, P. English Phonetics and Phonology: a practical course. 3rd edition.

Cambridge: Cambridge University Press, 2000. 283 p.

SCRIVENER, J. Learning Teaching. 2nd edition. Oxford: Macmillan, 2005. 431 p.

SNOW, Don. Myths and Misconceptions About Second Language Learning [online]. Washington: Center for Applied Linguistics, 1992 [quoted 5 February 2007]. Available from: <http://www.cal.org/recources/digest/myths.html>.

SOARS, J. et al. Headway Intermediate Pronunciation. Oxford: Oxford University Press, 1990. 112 p.

SZPYRA-KOZLOVSKA, J., FRANKIEWICZ, J., NOWACKA, M., STADNICKA, L. Assessing assessment methods – on the reliability of pronunciation tests in EFL [online] . Lublin: Maria Curie-Sklodowska University, Lublin, Poland: 2005. [quoted 14th February 2007]. Available from: <http://www.phon.ucl.ac.uk/home/johnm/ptlc2005/pdf/ptlcp37.pdf>

UNDERHILL, A. Sound Foundations. Oxford: Macmillan, 2005. 210 p.

UNDERHILL, N. Testing spoken language. Cambridge: Cambridge University Press, 1987. 117 p.

University of Cambridge. ESOL Examinations. Paper 3: Speaking. Cambridge Common Scale for Speaking [online, quoted 14th February 2007] Available from: http://www.cambridgeesol.org/support/dloads/ket/KET_HB_sampleS.pdf

UR, Penny. A Course in Language Teaching. Cambridge: Cambridge University Press, 1997. 375 p.

79