51
‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI Anna Shestakova Academic Dissertation to be publicly discussed, by due permission of the Faculty of Behavioural Sciences At the University of Helsinki, Department of Psychology, Lecture Hall I On the 3 rd of November, at 12 o’clock Cognitive Brain Research Unit at the Department of Psychology, University of Helsinki, Finland and BioMag Laboratory Helsinki University Central Hospital, Finland Helsinki 2004

‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

  • Upload
    others

  • View
    10

  • Download
    0

Embed Size (px)

Citation preview

Page 1: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

Anna Shestakova

Academic Dissertation to be publicly discussed, by due permission of the Faculty of Behavioural Sciences

At the University of Helsinki, Department of Psychology, Lecture Hall I On the 3rd of November, at 12 o’clock

Cognitive Brain Research Unit at the Department of Psychology, University of Helsinki, Finland

and

BioMag Laboratory Helsinki University Central Hospital, Finland

Helsinki 2004

Page 2: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

Supervised by

Academy Professor Risto Näätänen Cognitive Brain Research Unit Department of Psychology University of Helsinki, Finland

and

Dr. Minna Huotilainen Cognitive Brain Research Unit Department of Psychology Collegium of Advanced Studies University of Helsinki, Finland

Reviewed by

Dr. Elina Pihko BioMag Laboratory Helsinki University Central Hospital, Finland

and

Dr. Jyrki Tuomainen Phonetics Laboratory University of Turku, Finland ISBN 952-91-7844-1 (pbk.) ISBN 952-10-2135-7 (PDF) Yliopistopaino Helsinki 2004

Page 3: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

“Basic scientific ideas, theories, and explanations are always provisional and always subject to change.”

Sir J.C. Eccles

Page 4: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

To my parents, Nikolaj S. and Ljudmila N.

Page 5: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

ACKNOWLEDGEMENTS

I would like to express my warmest thanks to my supervisors, Dr. Minna Huotilainen and Academy Professor Risto Näätänen. I am deeply indebted to the following people without whom this thesis would not exist:

Alexander Batuev, Alexej Solovjev, Anu Kujala, Darja Osipova Eila Hagfors, Elvira Brattico, Gennadyj Kulikov, Hannele Ahti, Helen Kushnerenko, Irina Anourova, Johanna Meskanen, Jorma Arkkila Kalevi Reinikainen, Kimmo Alho,

Kiyoshi Yiaguchi Leena Wallendahr, Mari Tervaniemi, Marie Cheour, Marja Riistama, Markus Kalske, Mikko Sams, Olga Martynova, Pasi Piiparinen, Paavo Alku, Piiu Lehmus, Polina Balan, Risto J. Ilmoniemi, Rita Ceponiene,

Ritva Siukkola Sampo Antila, Sari Nenonen, Suvi Heikkilä, Teija Kujala, Teemu Peltonen, Vadim Nikulin, Valery Galoonov, Vassili Klucharev, Viktor Vorobjev Yury Shtyrov

I would also like to thank my dear family and friends who have supported and encouraged me through all these years that I have been working on my dissertation. I am grateful to you all! The studies that comprise this thesis were financially supported by the Center for International Mobility (CIMO), Finland, the Jenny and Antti Wihuri Foundation, the European Science Foundation (project number 80572) and the Academy of Finland (project number 79406) and the University of Helsinki. The experimental work was carried out in the Cognitive Brain Research Unit at the Department of Psychology, University of Helsinki, and at the BioMag Laboratory of Helsinki University Central Hospital, Finland I wish to thank Professor Erich Schröger for agreeing to act as my opponent at the public defence of this dissertation.

Page 6: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

6

ABSTRACT

Mismatch negativity, MMN, an auditory event-related response to any discriminable change in an ongoing sound stream, as well as its magnetic counterpart MMNm, and the MMN paradigm as such, have been shown to be extremely useful in studies of speech perception in humans. However, certain aspects related to brain plasticity, language specificity, and robustness of brain responses evoked by changes in speech stimuli remained uncovered so far. The aim of the research reported here was to study MMN and other components of event-related potential (ERP) (or evoked magnetic field (EMF)) that could be recorded in the same paradigm, and associated with the preattentive auditory discrimination of native and non-native phonemes. Two different methodologies were used: electroencephalography (EEG), and magnetoencephalography (MEG). First, EEG recordings were made in order to investigate children’s brain responses to phonemes of a second language as a function of their exposure to it in terms of time. The MMN results showed that the memory representations of the foreign phoneme system developed remarkably quickly. Finnish children (aged 3-6 years) who were exposed to French developed the ability to discriminate French phonemes in just a couple of months as indexed by an amplitude increase of the MMN concurrently with a latency decrease. The learning process was also reflected by changes in the P3a and in the late difference negativity (LDN) response. The functional behaviour of the MMN-P3a-LDN complex during learning was further assessed in detail. It was hypothesized that the functional role of LDN could be similar to that of reorienting negativity (RON), an adult’s brain response indicating the reorienting of attention back to the main task following distraction. The correlation analysis performed on the amplitudes of the MMN, P3a, and LDN components provided indirect evidence in favour of this view. Second, MEG was used to study MMNm in response to abstract phoneme representations. MMN and MMNm index the ability of the central auditory system to automatically extract the invariant, i.e. phonetically relevant, information for vowel recognition from acoustically continuously changing speech sounds uttered by hundreds of speakers. The results of the MMNm ECD analysis revealed that this process seems to be strongly lateralized to the left auditory cortex when natural speech sounds are used. A further aim in the research was to replicate and extend the findings of recent studies suggesting the existence of a phonotopic (phonemotopic) map in the human auditory cortex. Our experiment demonstrated that the different phonemic categories represented by multiple exemplars elicit differential brain responses. Two methodological issues with respect to MMN and MMNm recording were addressed throughout the research work: first, the MMN and P3a functions of sound source location were studied; and second, a new paradigm, called the varying-standard paradigm was developed to record a category-specific MMNm.

Page 7: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI
Page 8: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

8

ABBREVIATIONS

EEG electroencephalogram ERP event-related potential FFT fast Fourier transform LDN late difference negativity MEG magnetoencephalogram MMN mismatch negativity MMNm magnetic mismatch negativity MRI magnetic resonance image NLM Native Language Magnet (theory) N1, negative ERP peaking at approximately 100 ms from stimulus onset N1m magnetic N1 RON reorienting negativity SOA stimulus onset asynchrony SSG semisynthetic speech generation method

Page 9: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

9

CONTENTS ACKNOLEDGEMENTS ............................................................................................................................... 5 ABSTRACT ................................................................................................................................................... 6 LIST OF PUBLICATIONS............................................................................................................................ 7 ABBREVIATIONS........................................................................................................................................ 8 INTRODUCTION........................................................................................................................................ 11 LITERATURE OVERVIEW ....................................................................................................................... 12

Speech perception guided by recognition traces....................................................................................... 12 Mismatch negativity, MMN ..................................................................................................................... 13 MMN in pre-school and school-aged children ......................................................................................... 14 Emergence of language-specific memory traces ...................................................................................... 14 MMN-P3a-LDN complex......................................................................................................................... 15 Categorical perception of phonemes guided by language-specific memory traces .................................. 16 Phonotopical organization of auditory cortex........................................................................................... 17 Aims of the present studies....................................................................................................................... 21

METHODS................................................................................................................................................... 22 Participants ............................................................................................................................................... 22 Stimuli and experimental conditions ........................................................................................................ 22 Frequency analyses of the stimuli used in the MEG studies..................................................................... 24

PROCEDURE .............................................................................................................................................. 25 Stimulus presentation ............................................................................................................................... 25 EEG recordings and data analysis ............................................. Virhe. Kirjanmerkkiä ei ole määritetty. MEG recordings and data analysis ........................................................................................................... 27 Statistical analyses.................................................................................................................................... 29 Experimental designs in brief ................................................................................................................... 29

RESULTS..................................................................................................................................................... 30 Studies I and II. ERPs obtained from the children upon joining the French school ................................. 30 Study III. MMN and P3a as functions of location .................................................................................... 33 Study IV. MMNm in response to category change................................................................................... 35 Study V. Phonotopical gradient as indexed by N1m ................................................................................ 35

DISCUSSION .............................................................................................................................................. 38 Children learn to discriminate non-native speech: ERP associated with second-language phonological awareness.................................................................................................................................................. 38 Direction of children’s visual attention affects involuntary attention allocation to sounds...................... 39 Hemispheric asymmetry of the processing of the category change in speech perception. Evidence from the MEG studies ....................................................................................................................................... 40 Phonemotopic representations in the human auditory cortex ................................................................... 41

CONCLUSIONS .......................................................................................................................................... 42 References .................................................................................................................................................... 44

Page 10: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

10

Page 11: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

11

INTRODUCTION

Recently, using the mismatch negativity (MMN) component of the auditory event-related potential (ERP), and its magnetic counterpart MMNm, a major breakthrough was made in discovering mechanisms specific to the processing of speech sounds, and especially those characteristic of one’s mother-tongue. According to the MMN theory, after some repetition each sound develops a neural representation corresponding to the percept of this sound in the sensory memory. In practice, the MMN appears in a difference signal which is elicited when the regularities of the ongoing stimulation are violated (Näätänen et al., 1978). MMN data showed that the native-language phonetic prototypes evoked specific cerebral responses when compared to non-prototypes, thus postulating the existence of language-specific memory traces (Näätänen et al., 1997). Second-language acquisition seems to be dependent on recognition patterns for a newly acquired language, too (Winkler et al., 1999a; Dehaene-Lambertz et al., 2000). However, the time course of the development of the second-language memory traces remained uncovered. The aim of the research reported in this dissertation was to find the neural correlates of the second-language acquisition as a function of time in young children. MMN has been known to be a probe of the auditory discrimination function, and consequently of the auditory sensory memory. However, recent findings have suggested the existence of some higher cognitive processes in the sensory cortical areas than previously thought, such as the extraction of common invariant features shared by a number of acoustically varying sounds, and categorical speech perception guided by language-specific memory traces. This led to the development of the new concept of “primitive sensory intelligence” in the auditory cortex (Näätänen et al., 2001). How and where the information on phonemes is encoded in the brain remain undiscovered, bearing in mind the fact that the perceptual constancy of speech is preserved in the presence of its inherent physical variability. We wanted to investigate whether MMNm could be evoked in response to change that was more abstract than basic physical changes. In one of the studies reported in this dissertation, therefore, we introduced a substantial variation in the natural speech stimuli in order to investigate vowel-category discrimination. The tonotopic organization of the auditory cortex is a robust finding and has been studied extensively in animals and humans. The frequency maps first appear at the level of the cochlea and are maintained through the ascending auditory pathways up to the cortex. Thus, tonotopy can be referred to as a basic principle of the information processing in the auditory system. The idea of a phonotopic map which, according to some researchers (Ohl and Scheich, 1997; Diesch and Luce, 2000; Obleser et al., 2003), would enable the human brain to selectively respond to each phoneme depending on its category was put forward in order to explain how the categorical phoneme representations could be

Page 12: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

12

implemented in the human brain. In order to investigate whether such mapping could be preserved in the presence of vast acoustical variation, a separate magnetoencephalographic (MEG) study was conducted using natural stimuli. Altogether, the studies that comprise the present dissertation aimed at extending the notion of the role of the auditory cortical areas in the processes of a cognitive nature, such as categorical perception, thus developing the idea of ‘primitive sensory intelligence’ in the auditory cortex with regard to speech recognition.

LITERATURE OVERVIEW

Speech perception guided by recognition traces

Language can be studied on a number of levels, which could be roughly described as follows: semantics (the knowledge of meaning), syntax (word order), morphology (the structure of words), phonology (the study of sounds within a given language), and phonetics (the study of sounds irrespective of the language context). Phonology and phonetics are thus considered on different levels in the study of speech sounds. To take an example, the sounds “l” and “r” are clearly different phones, and in English they are also different phonemes. In Japanese, however, they represent the same phoneme. Therefore, a pair of phones (i.e., speech sounds) are said to belong to different phonemes of a given language if a difference between the phonemes makes a difference to the meaning of words in which they are incorporated. Phonemes have been widely used in various disciplines of modern science that concern speech perception, notwithstanding the ongoing debate on whether phonemic abstractions are the real primitives of phonology, or whether they are needed at all to understand the mystery of speech perception and production. Accordingly, phonemes are often referred to as “basic building blocks of a language’s sound structure” (Sussman, 2000). It became apparent as early as in the 19th century that there are some peculiar characteristics of speech perception that set it apart from the processing of other sounds. First, it was shown to be categorical (Liberman et al., 1967). The human auditory system is tuned so as to discriminate phonemes that cross the phonetic boundaries between but not within categories. Second, the perception of speech sounds in individuals older than six months is language-specific (Eimas, 1975). Näätänen et al. (1997) found neural correlates that lay behind phonological perception. Their study showed that native-language prototypical phonemes evoked special responses in native speakers of Finnish compared with speakers of Estonian. Mismatch negativity (MMN), a neural correlate of the speech-specific representation that can be measured as a difference between the two brain responses to auditory stimuli elicited by the Estonian / õ/ was smaller in amplitude

Page 13: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

13

than that elicited by the Finnish /ö/ even though the acoustic difference between / õ / and /e/ was larger than that between /ö/ and /e/. Thus, representations of native speech sounds, also called language-specific memory traces, were found.

Mismatch negativity, MMN

It has recently become evident that mismatch negativity (MMN), an endogenous component of event-related potential (ERP), provides the means to objectively study phoneme processing in the human auditory cortex (Näätänen, 2001). MMN was identified by Näätänen et al. (1978) as a change-detection response in auditory discrimination. Typically peaking at about 150 ms from change onset, it is generated by neural mismatch between a deviant sensory input and the neural representation, or ‘sensory memory trace’, formed by a repetitive ‘standard’ sound. It is thought to be a probe of the time-integrated sound representations in sensory memory (Näätänen and Winkler, 1999). Another component indexing auditory change detection, N1, is a stimulus-onset response. This exogenous stimulus-specific ERP component peaks at about 100 ms from stimulus onset. While N1 indexes the transient encoding of physical stimulus features, and does not always correspond to the level of conscious perception (Näätänen and Picton, 1987), the MMN amplitude correlates nicely with the accuracy of behavioural discrimination (Sams et al., 1985) and could be used as an objective measure of the auditory-discrimination function in the brain. Thus, N1 and MMN (their magnetic counterparts being termed N1m and MMNm, respectively) reflect the activation of two distinct change-detection mechanisms (Näätänen and Winkler, 1999). Importantly, unlike other endogenous components, MMN can be elicited in the absence of the subject’s attention (Näätänen et al., 1993; Paavilainen et al., 1993), which makes the MMN response extremely useful for studying young children. Although the involvement of the subcortical-level structures in the MMN generation cannot be completely ruled out, and is even suggested in some studies on animals (Kraus et al., 1994; Ruusuvirta et al., 1996), the predominant role of the auditory cortex in the discrimination of complex sounds is supported by results obtained from intracranial MMN recordings (Kropotov et al., 1995). Evidence of the essential role of the auditory cortex in the discrimination of complex sounds was first obtained some 40 years ago in studies on animal lesions (Dewson, 1964; Dewson et al., 1969). Moreover, Talwar et al. (2001) found recently that the application of muscimol, a neural inhibitor causing temporal inactivation of the primary auditory areas, led to profound deficit in simple tone detection and frequency discrimination in the rat. Furthermore, recent PET (Tervaniemi et al., 2000) and fMRI (Opitz et al., 2002) findings are in agreement with MEG and ERP studies that localize the main sensory-specific component of MMN in the auditory cortex (Hari et al., 1984; Giard et al., 1990; Alho, 1995; Alho et al., 1998), and the other

Page 14: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

14

component — the one associated with the involuntary switching of attention to deviant stimuli — in the frontal cortex (Giard et al., 1990). Likewise, several brain areas are suggested to be involved in the generation of the auditory N1 — supratemporal and frontal (Hari et al., 1982; Näätänen and Picton, 1987; Giard et al., 1994). Although the main generators of the N1/N1m are bilaterally located in the supratemtporal cortices, the sources of the MMN and N1 are clearly separate, at an approximate distance of 1 cm from each other (Sams et al., 1991; Hari, 1992 et al., Huotilainen et al., 1993; Korzyukov et al., 1999).

MMN in pre-school and school-aged children

MMN/MMNm response can be obtained already at birth (Alho et al., 1990; Cheour et al., 1998a; Ceponiene et al., 2002a; Kushnerenko et al., 2002; Huotilainen et al., 2003). Findings concerning MMN in pre-school and school-age children are robust; the MMN latency and amplitude are comparable to those of adults (Kraus et al., 1992; Korpilahti and Lang, 1994; for reviews, also see Csepe, 1995; Kurtzberg et al., 1995; Cheour et al., 1998b; Cheour et al., 2000; Ceponiene et al., 2002b; Jansson-Verkasalo et al., 2004), notwithstanding the developmental changes that occur in the early and later periods of childhood. Such changes result in (1) a U-shaped amplitude function of age — an early decrease and a later increase of the MMN amplitude (Cheour et al., 2000) — and (2) a slight gradual decrease in the MMN peak latency throughout childhood (Kraus et al., 1995; Shafer et al., 2000; Morr et al., 2002). The former presumably occurs due to the maturation of thalamocortical afferents in the deeper cortical layers during early childhood, as well as to the maturation of the commissural and association axons in the superficial cortical layers in later childhood (Moore and Guan, 2001; Moore, 2002).

Emergence of language-specific memory traces

As has been shown in behavioural and psychophysiological studies, speech perception is altered in the early ontogenesis under the influence of ambient language (Cheour et al., 1998b; Cheour et al., 2000; Kraus and Cheour, 2000; Kuhl, 2000). Unlike adults, infants under six months of age demonstrate categorical perception across the phonetic units of all languages (Streeter, 1976). However, after six months they start to show greater sensitivity to the native-language than to foreign vowel categories (Kuhl et al., 1992), and finally, after some time (about 12 months) lose the ability to discriminate non-native phonemic contrasts (Werker and Tees, 1983). Kuhl (2000) proposed a mechanism for establishing the origin of human differential sensitivity to the sounds of one’s native vs. non-native languages in the Native Language Magnet (NLM) model. This mechanism, called the magnet effect, can be best explained using the opposition of good representatives of a phonetic category (prototypes) vs. bad

Page 15: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

15

ones (nonprototypes). According to the NLM model, the language experience of an infant warps the acoustic dimensions underlying speech so that his or her perceptual space is shrunk in the vicinity of the prototype and stretched at the boundaries. As the NLM theory suggests, native-language memory traces in man appear to emerge very early in life, i.e., by 12 months of age (Kuhl et al., 1992; Cheour et al., 1998b; Dehaene-Lambertz and Baillet, 1998). Winkler et al. (1999a) suggested that acquiring a second language would require the formation of cortical memory representations for the phonemes of the speech sounds that are specific to it. This hypothesis was probed in Hungarian and Finnish subjects using MMN (Winkler et al., 1999a). The authors found that MMN for a contrast between two Finnish phonemes [a] and [æ] that were irrelevant to Hungarian was elicited in Hungarians who were fluent in Finnish but not in those with no knowledge of Finnish. This result indicates that the fluent Hungarians had developed cortical memory representations for the Finnish phoneme system, which enabled them to preattentively categorize phonemes specific to this language. The temporal characteristics of the afore-mentioned processes of language acquisition would be of great interest, however.

***

For the studies reported in the present thesis we recorded longitudinal ERP data to show the neurobiological manifestation of second-language learning in children (Study I). Native monolingual pre-school and school-aged Finnish speakers were monitored for six months (starting slightly before they had been enrolled at either the French school or in daycare) to determine how quickly and accurately they developed cortical memory traces and discrimination abilities for non-native speech sounds during this period using MMN. The aforementioned responses were then compared with those of monolingual age-peers.

MMN-P3a-LDN complex

In some experimental conditions, a child’s MMN can be followed by the P3a component (Squires et al., 1975), a positive deflection in ERP and, especially in children, late difference negativity (LDN), a negative peak of the difference waveform (Korpilahti and Lang, 1994; Korpilahti et al., 2001; Ceponiene et al., 2002b). Both P3a and LDN have amplitude maxima over the central and frontal areas. Escera et al. (2000) recently proposed a model of involuntary attention, according to which the supratemporal MMN subcomponent may be related to the memory representation of the auditory regularity involved in the MMN generation, whereas the frontal source might generate the signal leading to the attention-switching response (Näätänen and Michie, 1979; Giard et al., 1990; Näätänen, 1992; Schröger, 1996). P3a is assumed to reflect the involuntary allocation of attention towards novel or deviant stimuli (Picton, 1992; Alho et al., 1997;

Page 16: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

16

Escera et al., 2000). The reorienting of attention back to the main task appears to be indicated by reorienting negativity (RON), a late frontal negativity as recently identified by Schröger and Wolff (1998) in the condition of auditory distraction of the concurrent task performance. While the MMN and P3a components are well documented in the literature, the functional role of LDN remains obscure. Consequently, P3a and LDN were systematically observed during the MMN recordings in Study I. Given that the perceptual salience of the foreign-language stimuli used for Study I, as well as their distracting effects, had been increasing throughout the three-month exposure period, the functional role of LDN and its relation to the processes indexed by the MMN and P3a were considered in the light of the aforementioned model of involuntary attention.

***

Each of the two components ― LDN and MMN ― has a special role in language processing. Study II was devoted to analysing the functional significance of LDN in learning the second language, as well as its relation to the processes of automatic auditory discrimination and involuntary attention indexed by MMN and P3a, respectively.

***

Furthermore, in Study III we addressed some methodological issues arising from the recording of the MMN-P3a complex in response to speech stimuli in children. A typical MMN experiment with children is such that a child’s attention to the auditory stimuli is distracted using a movie or a cartoon presentation. To make the experimental situation more comfortable, the sound stimulation is often delivered via loudspeakers (not headphones). However, the possible effect of the location of the stimulus source and its integration with the source of the visual distraction task is typically neglected. We hypothesized that preattentive auditory discrimination might be sensitive to the azimuth of the sound source. To test this hypothesis, three different types of stimuli varying in complexity were used: sinusoidal tones, complex tones, and vowels. We were looking for the differences between the two brain processes — sensory memory (underlying the MMN elicitation) and involuntary attention (underlying the P3a generation) — as functions of stimulus location.

Categorical perception of phonemes guided by language-specific memory traces

MMN is typically elicited when there is a many-to-one ratio among the stimuli: infrequent deviant sounds interrupt a sequence of identical sounds, the standards.

Page 17: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

17

However, the results of recent several studies suggest that more complex events such as stream segregation (Sussman et al., 1998; Sussman et al., 1999), extracting the abstract sound patterns and invariant sound relationships (Tervaniemi et al., 1994; Paavilainen et al., 1999; Paavilainen et al., 2001), and categorical speech perception guided by language-specific memory traces (Dehaene-Lambertz, 1997; Näätänen et al., 1997; Winkler et al., 1999a; Winkler et al., 1999b; Phillips et al., 2000; Phillips, 2001) may take place preattentively in the auditory cortex. Moreover, the neural correlates of the above-mentioned processes can be recorded using MMN (or its magnetic counterpart MMNm). These and other findings indicate that sensory cortical areas are the substrata for higher-order cognitive processes rather than for the mere detection of a physical change in auditory input (Näätänen et al., 2001). It has been shown that the permanent traces of phoneme categories can accommodate acoustic variation (Aulanko et al., 1993; Dehaene-Lambertz et al., 2000). However, thus far, some studies of cortical phoneme perception have been constrained in that only a few stimuli have been used (Näätänen et al., 1997; Winkler et al., 1999a). Moreover, different studies of categorical sound perception have been restricted to synthetic or semisynthetic stimuli (Dehaene-Lambertz, 1997; Alku et al., 1999; Phillips et al., 2000). By and large, many authors have agreed on the importance of using natural or biologically valid stimuli in studies of neural coding in the auditory system and its plasticity (for a review, see Eggermont, 2001). According to Brugge (1992), for example, “In order to qualify as a neural code for acoustic information, it must be shown first that the pattern in question occurs in the auditory system under natural conditions or is evoked by natural stimuli”. Even though this statement originally referred to an animal model, it may be of equal importance in the studies of a cerebral substrate of speech, the perceptual constancy of which is preserved in the presence of its inherent physical variability.

***

On the one hand, using stimuli that are well controlled in frequency, duration, and intensity allows one to carefully balance and control for the acoustic parameters; but on the other hand, evidence may be weakened by the use of only a few stimuli in studying speech perception. In Study IV, we used 450 different natural stimuli to determine how the auditory cortices may automatically discriminate phoneme category irrespective of extensive inter-speaker variability.

Phonotopical organization of auditory cortex

The auditory cortex plays an important role in processing complex sounds and speech. Its general characteristics are as follows: first, frequency is mapped in an orderly fashion across its surface forming a one-dimensional tonotopic axis (Merzenich and Brugge,

Page 18: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

18

1973); second, there is a columnar arrangement perpendicular to the surface of the cortex within which neurons show little variation in frequency sensitivity (Abeles and Goldstein, 1970). As reported by Versnel et al. (1998), these columns provide a systematic representation of the periodicity of the spectral envelope of a complex sound. Tonotopy is a general principle of the functional organization of the auditory system. It arises in the cochlea and is maintained throughout the central auditory pathways. Projections from the cochlea travel via the eighth nerve to the three main divisions of the cochlear nucleus, the targets of which include the superior olive and nuclei of the lateral lemniscus, which in turn innervates the neurons of the inferior colliculus. The neurons of the inferior colliculus send out the axons to the medial geniculate body of the thalamus, which is the last relay before the auditory cortex (Goldstein, 2002). According to the modern model of auditory cortical organisation, the primate auditory cortex consists of three major regions: the core, the belt, and the parabelt (Hackett et al., 1998; Kaas and Hackett, 1998; Kaas and Hackett, 1999; Kaas and Hackett, 2000; Hackett et al., 2001). In animals, the primary auditory cortex and some of the divisions of the belt are tonotopically organized (Rauschecker et al., 1995; Rauschecker, 1998; Wessinger et al., 2001). However, their belt and parabelt regions are inhomogeneous in their cytoarchetictonical structure and thalamocortical and intercortical projections, and therefore can be subdivided into different areas that are sometimes organised in a reversed tonotopic order (Merzenich and Brugge, 1973; Morel et al., 1993; Kaas and Hackett, 1998). Although the relationship between the physiological observations and the cytoarchitectonical data remains unclear, it has nevertheless been suggested that the aforementioned model may also apply to humans (Hackett et al., 2001). Indeed, MEG has been used to demonstrate the tonotopical organization of the auditory cortex in humans with the main gradient running along the lateromedial axis, with more lateral equivalent current dipole (ECD) locations for low frequencies and more medial locations for high ones (Romani et al., 1982; Pantev et al., 1988; Pantev et al., 1994; Pantev et al., 1995). Several N1m studies have also found an additional gradient in the tangential plane, with low frequencies being represented posteriorly and high frequencies anteriorly (Romani et al., 1982; Pantev et al., 1994; Pantev et al., 1995; Langner et al., 1997). Some researchers argue that the MEG findings of the orderly representations of frequency in the human auditory cortex cannot be directly compared with those obtained with the help of multi-unit electrophysiological recordings in animals; this is because the spatial resolution of the MEG is not enough to tackle the multiple tonotopic maps of the different core and belt regions, and because of the interindividual variability in anatomy of the Heschl’s gyrus (HG) (Schönwiesner et al., 2002; Lütkenhöner et al., 2003). Conversely, remarkable results have been obtained in recent studies of the periodotopic

Page 19: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

19

and phonemotopic organization of the human auditory cortex and, in particular, in analyses of their spatial relationships with the tonotopic gradient (Langner et al., 1997; Obleser et al., 2003). The main findings from these comparisons establish the independence of the phonotopic and tonotopic maps. It was also found that the N1m ECD trajectories for stimuli varying in timbre were almost orthogonally displaced from those varying in pitch in both lateral and top views (Langner et al., 1997). Indeed, pitch and timbre are largely independent and may be considered orthogonal perceptual parameters related to temporal envelope periodicity (tonotopy) and spectral content (periodotopy), respectively (Krumhansl and Iverson, 1992). For example, different musical instruments and human voices differ in timbre from each other when producing the same note. A number of groups have proposed a phonotopic mapping principle, which would enable the human brain to selectively respond to each phoneme depending on its category (Ohl and Scheich, 1997; Diesch and Luce, 2000; Obleser et al., 2003). Data from different studies, even if not unanimously, suggest that vowel categories can be represented by separate neuronal populations in the auditory cortex (Diesch et al., 1996; Diesch and Luce, 1997; Ohl and Scheich, 1997; Poeppel et al., 1997; Diesch and Luce, 2000; Obleser et al., 2003). Reasons for such a differentiation have been found in the different encoding of information among the frequency, amplitude, and time domains (Bladon and Lindblom, 1981; Diesch and Luce, 2000; Ito et al., 2001). Diesch’s et al. (2000) hypothesis is based on lateral inhibition, another classical organizational principle of the auditory system: the excitation or neural activity caused by stimulation at one characteristic frequency is reduced by the presence of excitation or neural activity at the adjacent characteristic frequencies (Moore, 2003). Therefore, inhibition is a nonlinear process response. For example, a pair of simultaneously presented stimuli is not a linear sum of the response inputs to each stimulus presented separately. However, in search of the factor pertaining to invariant speech-sound processing in humans, the aforementioned N1m studies employed only a few speech stimuli. This complicated their interpretations of phonotopy because both physical characteristics differing across the stimuli and invariant features enabling one to classify a different speech sound as belonging to one or another category could be used during phoneme encoding in the brain.

***

Our aim in Study V, was to investigate invariant speech-sound processing by introducing a large variation to the natural stimuli. We also tested the hypothesis that any differences in the loci of cortical vowel representations using the N1m could be attributed to any common acoustic features characterizing the groups of the vowel exemplars of each of the three categories studied.

Page 20: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

20

Dual nature of MMN

Initial evidence of the dual nature of MMN, i.e. the simultaneous effect of acoustic and phonetic stimulus characteristics in it, was obtained by Näätänen et al. (1997). They found that the left-hemisphere MMNm enhancement for prototype deviant phonemes was enhanced compared with that elicited by a non-prototype stimulus. This was explained by the occurrence of two parallel change-detection processes — the acoustic and the phonemic — for the prototypes, whereas only the acoustic change-detection process occurred when the deviant stimulus was a non-prototype phoneme. It is noteworthy that the language-specific change detection took place in the left auditory cortex, whereas the opposite hemisphere merely detected the acoustic change. Further striking evidence of the two parallel processes for acoustic and phonetic inputs come from studies on duplex perception (Liberman and Mattingly, 1985; Whalen and Liberman, 1987). Duplex perception is a phenomenon in which the same acoustic event, namely a short third-formant transition, evokes two different percepts — one that is perceived as speech (the consonant part of the CV syllable) and another that is perceived as non-speech (a tone glide, which sounds like ‘chirp’) — when combined in a dichotic listening experiment with the syllable base, including the first two formant transitions and the steady-state portion of the vowel (Mattingly et al., 1971). Whalen and Liberman suggested that processing for the phonetic module has priority over processing for the auditory module. The results of the studies on duplex perception strongly supported this view (Liberman and Mattingly, 1985; Whalen and Liberman, 1987). Since in their study, a tone glide was not perceived until the intensity of the third formant transition was increased relative to the base, Whalen and Liberman surmised that the incoming signal was processed first in the phonetic module, and then forwarded to the auditory module (Whalen and Liberman, 1987). It is not only language-specific MMN, but also abstract-pattern MMN (Näätänen et al., 2001) that apparently incorporates two types of automatic sensory memory mechanism: the first is a short-term, sound-pattern encoding mechanism, and the second a long-term mechanism, which provides a template or recognition pattern. Thus, the dual function of the MMN has been uncovered. The long-term memory it indexes is ‘primitive’ unconscious long-term memory, yet it appears to exert a powerful influence on declarative memory and behaviour, particularly speech perception and possibly production. However, the question arises as to how the connection between the sensory and long-tem representations is implemented. In an attempt to find the answer, an elegant study was conducted by Huotilainen et al. (2001) showing that very few repetitions were enough to activate phoneme prototype memory traces as opposed to non-prototype ones. Thus long-term memory traces seem to facilitate short-term representations. The authors

Page 21: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

21

proposed that the link from long-term back to short-term memory could be considered a property of the feature-analysis system. This system may respond more readily to a certain combination of acoustic features if the neural connections underlying the preformed long-term memory trace to this combination in the brain are strengthened. Pharmacological studies of the role of cortical N-methyl-D-aspartate (NMDA) receptors in auditory sensory memory provide indirect evidence to support this view, too. On the one hand, glutamate (a mediator of the NMDA system) has been observed in a process called long-term potentiation that has emerged as a model for the creation of long-term memories. In fact, NMDA receptor activation plays a critical role in ‘Hebbian’ learning whereby “those (cells) who fire together wire together”— a classical form of learning by association or conditioning. On the other hand, as a recent study showing that phencyclidine, an NMDA antagonist, abolished the MMN generation in monkeys suggests (Javitt et al., 1996), transient representations of short-term memory are also dependent on the function of the NMDA receptors.

Aims of the present studies

The aim of the present dissertation is to determine some of the unknown properties of the auditory cortex in extracting invariant patterns from the physical variability of speech sounds. In order to elucidate the dynamic nature of cortical memory representations of phonemes, we compared the MMN responses of school-age children learning a second language at school with those of a monolingual group of age peers naïve in that language (Study I). Study II assessed the functional behaviour of the MMN-P3a-LDN complex during learning in detail. An additional goal was to determine the functional role of LDN and its relation to MMN and P3a. Study III tested the hypothesis that preattentive auditory discrimination is sensitive to the azimuth of the sound source, i.e., this experiment compared the MMN and P3a components with each other with respect to a function of stimulus location. In Studies I-III we chose to use EEG. Studies IV and V shed light on the neural mechanisms whereby the human brain automatically processes phoneme categories irrespective of the large acoustic inter-speaker variability. Study IV was carried out in order to determine whether MMNm could be recorded in response to changes in features that are more abstract than basic physical characteristics, e.g., cognitive features. Here we studied the MMNm elicited by a vowel-category change. The aim of Study V was to find evidence of a phonotopic (sometimes called phonemotopic) map, which appears to be a prerequisite of categorical speech

Page 22: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

22

perception. We used MEG recordings in the last two studies, since this method has high temporal resolution as well as fine spatial accuracy in localizing active brain areas.

METHODS

Participants

All of the subjects, both children and adults, participating in the present studies were healthy volunteers with normal hearing and no record of neurological disease. All of the families gave informed consent for their children to participate in Studies I, II, and III. All the studies were approved by the Ethical Committee of the Department of Psychology, University of Helsinki, or by the Coordinating Ethical Committee of the Helsinki University Central Hospital (Studies IV, V). Pre-school and early school-age Finnish-speaking children (aged 3-6 years; N=17, 4 males) had been invited to the experiments just before they joined a French school or a day-care centre in which French was spoken 50-90% of the day (the experimental group). They were monitored for a six-month period as they started to acquire the second language. This study system is called the immersion method, according to which groups of learners are taught exclusively through the medium of the foreign language (Harley, 1995). The MMN recorded in the children from the experimental group was compared with that recorded in their age-peers (also Finnish speakers) from the control group (N=17, 4 males) who were learning French (Studies I and II). Seven-to-ten-year-old Finnish-speaking children (N=22, 10 males) participated in the MMN experiment conducted for Study III, half of them in the ‘In-front’ condition and half in the ‘On-sides’ condition. For Studies IV and V, Russian adults (aged 23-33 years, N=9, 5 males, one left-handed, (Study IV; N=11, 6 males, all right-handed, Study V) volunteered to undergo the MEG measurements. The Edinburgh Handedness Questionnaire (Oldfield, 1971) was used to access their handedness.

Stimuli and experimental conditions

The stimuli used in Studies I and II included semisynthetic vowels produced according the semisynthetic speech-generation method, SSG, based on an artificial vocal-tract model with an excitation waveform extracted from natural French phonemes (Alku et al., 1999): [i] , [e], and [ε]. Prior to the experiment, adult Finnish native speakers were asked to rate the category goodness of the stimuli used in this experiment, among other French and Finnish vowels. None of the three stimuli chosen for the experiment were perceived as ‘best exemplars’ of Finnish vowels and therefore considered non-prototypes of Finnish

Page 23: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

23

(Kuhl et al., 1992). The formant frequencies of speech sounds lasting 210 ms including 10-ms rise and fall times are presented in Table 1. The fundamental frequency F0 was stable at 100 Hz for each of the three phonemes. The phoneme stimuli were used to study language-specific perception in an odd-ball experiment. The SSG method is optimal in terms of generating stimuli in this context, as it preserves the natural variation of the excitation signal, thus making the phoneme stimuli sound much more natural than phonemes synthesized using other methods. At the same time, the use of SSG allows the only differences between the stimuli to be the formant frequencies, while all other parameters are exactly the same. This is crucial, since any acoustic difference in any feature of the stimuli will give rise to a response in an odd-ball experiment. The SSG method was also used Study III to generate vowels in addition to two other stimulus classes: complex tones and single-partial tones. The complex tones were composed of sinusoidal partials, the frequencies of which corresponded to the main frequencies of the vowel formants. The frequency of the standard single-partial sinusoidal tone (further referred to as ‘simple tone’) was 458 Hz, in other words, it matched the strongest harmonic component F1 of the standard vowel. Both standard and deviant vowels were perceived as the vowel /ö/ ([ø] phoneme which sounds like the French ‘eux’) in Finnish. The fundamental frequency and the lowest formants of the standard vowel were as follows: F0=115, F1=450, F2=1420, F3=2200, and F4=3500 Hz. Within each stimulus class, a difference of 10% was introduced between the formant/partial frequencies of the 260-ms standard and the deviant sounds. In Studies IV and V, 450 natural Russian speech stimuli were used, each uttered by a different speaker, belonging to three vowel categories: [a] (as in English ‘father’), [i] (e.g., ´willow´), and [u] (e.g., ‘boom’). These three phoneme categories were chosen so as to create maximum distance from each other in the two-formant coordinate space of

Table 1. Frequency characteristics of French vowels. Studies I and II.

Word example F1 F2 F3 F4

[e] Peiner 475 1790 2370 3580

[i] Dix 235 2070 3210 3790

[ε] Treize 370 1440 2045 3170

Formant frequencies are given in Hz.

Page 24: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

24

Russian (Potapova, 1997). The stimuli were recorded via a fixed telephone network channel digitized with 8000 Hz and 16 bits, edited to have a duration of 250±20 ms, and RMS-normalized in amplitude.

Frequency analyses of the stimuli used in the MEG studies

Information on the amplitude spectra and dispersion of the stimuli used in Studies IV and V was obtained as follows. For each stimulus, a spectral envelope was obtained by using FFT (Fast Fourier Transform). The mean of the spectrum, which is characteristic of the energy distribution, was separately calculated for each sound (Fig. 1). Group-average spectra were obtained for each of the three phoneme categories (Fig. 2). In addition, to roughly approximate the grand-average spectrum envelope, and to further compare its differences across the categories, a normalized spectrum mean was calculated for each of the three vowel categories as an area below the corresponding group-average spectral curve.

0

20

40

60

80

100

120

140

3 8 13 18 23 28 33 38Spectrum mean

Num

ber o

f vow

els

AIU

[a][i][u]

Figure 1. Stimulus distribution: the abscissa shows the normalized spectrum mean for each vowel exemplar in the given category; the ordinate shows the ordinal number of each vowel exemplar in the given category.

Page 25: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

25

PROCEDURE

Stimulus presentation

The oddball paradigm was used to obtain ERPs in Studies I, II, III and V, in which deviant sounds were infrequently and randomly presented in sequences of frequent standard sounds. The probabilities of the standard and the deviant sounds of each experiment are given in Table 2. A new paradigm based on a modification of the classic oddball paradigm called the roving-standard (Cowan et al., 1993) applied in Study IV was implemented. It is illustrated in Fig. 3 and can be characterised as follows. Speech stimuli produced by different speakers, but belonging to the same phoneme category, were presented several times: this suggested that a memory trace for the specific vowel category in the subjects’ auditory sensory memory could be built. As a consequence of the acoustic variation in speech sounds, sensory memory traces were formed not according to any physical repetitive feature of the sounds, but according to phoneme categories. After repeating tokens from the same category between three and six times, we foiled the expectation of the specific phoneme category by presenting a stimulus from another phonetic category. The first stimulus following the change of category was regarded as a deviant, while the stimuli following more than three repetitions of the same category were regarded as standards. This modification was termed the varying-standard paradigm (see Fig. 3 for an example).

688

281

1219

281

2125

344

625

0

10

20

30

40

50

60

70

80

0 313 625 938 1250 1563 1875 2188 2500Frequency, Hz

Am

plitu

de, % [a]

[i][u]

Figure 2. Group average spectra for the three vowel categories. The numbers on the plot indicate the frequencies of the local maxima, i.e., the F1 and F2 formants for each group of vowels.

Page 26: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

26

All the stimuli were presented binaurally. Other nuances of the stimulus presentation in this and the other studies are summarized in Table 2. In order to distract subjects’ attention from the sounds, they were offered to watch self-selected movies or cartoons with the sound turned off. The stimuli were delivered through a system incorporating plastic tubes, EAR loudspeakers and earpieces in the MEG studies, while sounds were presented via loudspeakers in the EEG studies. In Study III, two experimental conditions — with and without integration of the audio and visual sources — were chosen in order to study involuntary attention among the children as a function of sound-source location. It should be noted that the ‘In-front’ setup, in which the sources of the audio and visual stimulation were integrated (i.e., were in the same direction), was used in Studies I and II.

EEG recordings and data analysis

So far, EEG has been one of the most frequently used methods in studies of the brain function and its development in pre-school- and schoolchildren, being (1) totally non-invasive (compared with PET or fMRI), (2) inexpensive, (3) and less demanding than MEG (the popularity of which is nevertheless growing along with its methodological improvement), given that it permits some head movement during the recording. In Studies I, II, and III, EEG, amplified by SynAmps at DC-30 Hz, and digitized at 250 Hz, was recorded using the NeuroScan PC-3.0 based system at the Cognitive Brain Research Unit, Department of Psychology, University of Helsinki. Individual silver/silver-chloride electrodes were attached to the scalp sites (Table 3) according to the International 10-20 system. Eye movements were monitored via the electrooculogram (EOG) electrodes attached below and at the outer cantus of the right eye. During the recordings, the electrodes were referred to the right mastoid. In order to avoid a hemispheric bias, the data from each hemisphere were re-referenced off-line to the

Figure 3. Varying-standard paradigm. A 30-s sequence of the original stimulus block: the upper letters represent the codes for the standard (S) and deviant (D), the middle stream indicates the waveforms of the natural stimuli, and the lowest stream shows the vowel categories. The two consecutive stimuli immediately following the deviant were omitted from the standard average (hollow squares).

Page 27: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

27

ipsilateral mastoid (Studies I and II) or the average of the two mastoids (Study III). Table 3 also gives an overview of the characteristics of the EEG data analysis, including epoch lengths, after-averaging filtering parameters, artefact rejection criteria, as well as the latency windows for measuring the ERP mean amplitude.

MEG recordings and data analysis

With its absolute noninvasiveness, good accuracy in source location, and millisecond-scale time resolution, the MEG (in combination with other brain imaging methods, e.g., MRI) has become a powerful tool for studying cerebral activity in humans. Minute changes in the magnetic field can be recorded using superconducting quantum interference devices (SQUIDs). An MEG signal can carry important information about sensory as well as higher-level information processing in the brain. Because the MEG method is only sensitive to tangentially–oriented currents, and does not detect sources with an absolute radial orientation (Hämäläinen et al., 1993), it is most sensitive to activity in the fissural cortex and thus has great potential in terms of studying the auditory function in humans. Indeed, pyramidal cells — the largest cortical neurons in the auditory cortex — are oriented so that the primary current generated by them has a tangential orientation to the surface of the brain. The evoked magnetic fields are considered to be counterparts of the corresponding event-related potentials (Romani et al., 1982; Pantev et

Table 2. Parameters of the stimulus presentation in Studies I-V.

Study Stimulus probability SOA,

ms

Sound pressure level Sound-delivery software

I and II [i]-standard, p=0.8 , [e]-and [ε]-deviant,

P=0.1 for each

900 65 dB SPL STIM (NeuroScan Labs)

III [ø]-standard, p=0.86, frequency-and

duration-deviant *, P=0.07

800 55 dB SPL, ‘In-front’;

60 dB SPL , ‘On-sides’

STIM (NeuroScan Labs

IV and

V

category-standard ([a], ([i] or [u]),

P=0.8, category-deviant P=0.1 (Study

V*’).

900 55dB above the

individual hearing

threshold

Presentation 0.43,

Neurobehavioural

Systems Inc.)

*Only the ERPs elicited by frequency deviants are reported in the present dissertation. **For the stimulus probability in Study IV, see the above explanation.

Page 28: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

28

al., 1988; Hari, 1990; Tiitinen et al., 1993; Huotilainen et al., 1998). Moreover, the auditory event-related magnetic fields (ERFs), obtained by averaging MEG signals time-locked to the auditory stimulation, have proved to be very useful for studying the anatomical and functional organization of the auditory cortex in humans (Pantev et al., 1988; Hämäläinen et al., 1993; Huotilainen et al., 1995; Huotilainen et al., 1998). In Studies IV and V, the MEG responses were recorded using a whole–head magnetometer (a 306-channel Vectorview system, Elekta Neuromag Oy, Helsinki Finland) at the BioMag Laboratory, Helsinki University Central Hospital. Eye movements and blinks were recorded with vertical and horizontal bipolar EOGs. MEG epochs (sampling rate 603 Hz, passband 0.03−90 Hz) starting 100 ms before and ending 500 ms after stimulus onset were separately averaged for the standards and deviants, while epochs contaminated by extra-cerebral artefacts (EOG variation exceeding 150 µV or MEG variation in any channel exceeding 3000 fT/cm during the epoch) were automatically omitted. The averaged responses were digitally off-line filtered with a passband of 1–30, Hz and baseline-corrected off-line in relation to the mean amplitude at the 100-ms pre-stimulus interval. In addition to the temporal filtering, signal-space projection (SSP) was applied to suppress external magnetic noise (Tesche et al., 1995; Uusitalo and Ilmoniemi, 1997). The N1m responses were evaluated by using 63 channels (containing 21 triplets of 2 gradiometers and 1 magnetometer) above each temporal area separately for each subject´s left and right hemispheres. Equivalent current dipoles (ECDs) were used to model the activity in the brain. The courses of ECDs for MMNm

Table 3. Summary of the ERP analysis parameters.

Study Electrode sites Artifact

rejection

criteria, µV

Epoch length,

(prestimulus time

included), ms

Passband

(filter slope),

Hz

Latency window

(integration window)

I F3−4, C3−4,

T3−4, and P3−4

200 500 (100) 1.5−15 (24

dB/octave)

MMN 1503−50 (50)

II F3−4, C3−4,

T3−4, and P3−4

200 900 (100) 1.5−15 (24

dB/octave)

P3a 200−500 (80); LDN

350−700 (80)

III F3−4, C3−4 150 800 (100) 1−15 (24

dB/octave)

MMN 100−350 (50);

P3a 250−400 (50)

Page 29: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

29

and N1m were estimated by means of sequential ECD fitting using a spherical head model (see below) (Hämäläinen et al., 1993) in the latency range of 100–250 ms and 80–150 ms from stimulus onset, respectively. The ECD that had a local maximum in the source strength, Q, or, in its absence, a local maximum in goodness of fit, was chosen as representative of the source. For the majority of subjects participating in Study V, high-resolution volumetric magnetic resonance images were acquired using a 1.5-T Siemens (128 slices with thickness of 1.3 mm) at the Department of Radiology, Helsinki University Central Hospital. A 3-D reconstruction of the brain surface was used to determine the optimal individual location of the sphere used in the ECD fitting. For subjects having no MRI, the sphere origin was placed at (x,y,z)=(0,0,45 mm). In the search for spatial differences between the N1m ECDs in response to the different stimulus categories, the Euclidian distances were calculated in addition to the absolute locations. The former analysis was performed as a vector sum from the distances in the x, y, and z directions.

Statistical analyses

Statistical comparisons of the data were made using 1) t-tests and 2) analyses of variance (ANOVA/MANOVA). Least significant difference post-hoc test was used to calculate the sources of the significant ANOVA effects and interactions. The presence of an ERP component was verified by comparing its amplitude to 0 µV in the two-tailed t-test. Greenhouse-Geisser correction was applied to factors with more than two levels. The corrected P values are reported where appropriate. An alpha level of 0.05 was used for all the statistical tests. The correlation analysis was performed using Pearson Product-Moment correlation.

Experimental designs in brief

Studies I and II. The MMN, P3a, and LDN elicited by the vowel change were measured and compared between the experimental and the control groups across the recording sessions, stimulus types, and electrode sites. The experimental group was tested minimum three times during a six-month period. The experimental group was tested twice. Study III. The MMN and P3a elicited by the frequency change in vowels, complex tones, and simple tones were recorded in the two conditions: with and without the integrated audio- and visual stimulation. Their mean amplitudes and latencies were compared across the stimulus types and electrodes. The analysis of change-detection responses was complemented with that of the so-called obligatory responses in order to check for their contribution to the condition main effect on the difference waveform.

Page 30: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

30

Study IV. The MMNm elicited by the category change in the varying-standard paradigm was measured and compared between the two hemispheres. Similar analyses of dipole moments, latencies and source locations were performed for the N1m recorded to the standard response in the same paradigm. Study V. In a search for spatial differences between the N1m ECDs in response to the three different stimulus categories, the absolute locations of and the relative distances between the N1m ECDs, as well as the Euclidian distances, were calculated. The spectrum mean and maximal dispersion data were further compared with the N1m source-analysis data.

RESULTS

Studies I and II. ERPs obtained from the children upon joining the French school

The brain responses to the French speech sounds are presented in Fig. 4. These responses clearly imply the development of non-native memory traces in the Finnish-speaking children who were exposed to French. In the MMN analysis, a four-way ANOVA with the Group as a grouping factor revealed a significant Group × Session interaction in comparisons between the experimental and control groups (F (1,32)=11.96, P<0.002). The same results were observed in the P3a and LDN comparison: (F(1,32)=17.39, P<0.001); (F (1,32)=36.74, P<0.0001). LSD post-hoc testing showed that the MMN, P3a, and LDN amplitudes were significantly larger in the experimental study group in the last session compared with the first. Moreover, the amplitude of each of the three components in the last session was significantly larger in the experimental group than in the control group. In particular, no significant increase in the MMN, P3a and LDN amplitudes was found in the control group between the first and second recordings, nor were there any significant MMN-amplitude differences between the experimental and control groups in the first session.

Page 31: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

31

The facilitating effect of exposure to the French language can be seen in Fig. 5 as an increase of average amplitudes of the MMN, LDN and P3a components. The Session effect on MMN amplitude was tested separately for the experimental subjects using a three-way ANOVA: (Session (First, Second, Last) × Stimulus [e], [ε] × Electrode (F3, F4, C3, C4, P3, P4). This analysis was highly significant for MMN (F(1,32)=18.12, P<0.00001); P3a (F(1,32)=36.75, P<0.00001), and LDN (F(1,32)=30.77, P<0.00001).

Not-studying, first session

Not-studying, last session

Studying, first session

Studying, last session

F3 F4

C3 C4

P3 P4

-5 µV

+5 µV

-100 ms 800 ms

MMN LDN

P3a

C3

F3

P3

F4

C4

P4

Figure 4. Responses to phoneme changes. Difference waves (obtained by subtracting the average standard /i/ stimulus response from the average deviant /e/ stimulus response) of 17 Finnish-speaking children who began to attend French day-care are shown in the beginning of the test period (at a few weeks before the studies), 1.5–2 months after they had started, and about 3–4 months later. The difference waves of 17 monolingual Finnish-speaking children who were not exposed to French are also shown at the beginning of the studies and after about 3–4 months.

Page 32: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

32

To further examine the relationship between MMN, P3a, and LDN, a correlation analysis (Pearson Product-Moment correlation) was performed on the amplitudes. Based on the correspondence between the individual amplitudes for MMN−LDN, P3a−LDN, and MMN−P3a, correlations were obtained for these three respective pairs of ERPs. Table 4 summarizes these analyses. The P3a-LDN correlations were significant, whereas no correlation was observed between MMN and LDN, or between MMN and P3a. In order to check for intergroup variability in the subjects exposed to French, an additional 4-way ANOVA analysis with GROUP, Session, Deviant and Electrode factors was performed. Neither the Group nor Group × Session interaction showed significance (Fig. 6).

F3

Session1 2 3

-5

0

5

F4

C3 C4

MMN LDN P3a

F3 C3

C3 C4

ER

P a

mpl

itude

, µV

/e/ - /i/ /ε/ - /i/

Figure 5. Average amplitudes of MMN, LDN, and P3a recorded in response to the French-phoneme contrasts. Seventeen children were monitored during a six-month period upon joining the French school or day-care. Vertical bars indicate standard errors of mean.

Page 33: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

33

Study III. MMN and P3a as functions of location

MMN and P3a can be seen in the difference-waveforms as prominent peaks with latencies of 170–250 and 330–370 ms, respectively (Fig. 7). MMN, the magnitude of which was calculated as the mean voltage of the 50-ms interval centred at the corresponding peak-latencies of the left and right frontal electrodes in the grand-averaged waveforms separately for each stimulus type, was statistically significant in both conditions. The P3a mean amplitude was significant in the ‘In-front’ condition only.

Table 4. Correlations among the MMN, LDN and P3a amplitudes a.

/e/ deviant /ε/ deviant

F3 F4 C3 C4 P3 P4 F3 F4 C3 C4 P3 P4

MMN-LDN n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. 0.58 n.s. n.s.

LDN-P3a -0.83 -0.78 -0.85 -0.83 -0.58 -0.53 -0.85 -0.84 -0.88 -0.86 -0.70 n.s

MMN-P3a n.s. n.s. n.s. n.s. -0.63 n.s. 0.55 n.s. n.s. -0.73 -0.63 0.65

a n.s., not significant (p > 0.05).

third sessiionsecond sessionfirst session

MMN

-3 µV

+3 µV

400 ms

YOUNGER3−4-year-old

OLDER5−6-year-old

-3

-2

-1

01 2 3

Time

µV

YoungerOlder

* Group effect (Younger vs. Older): F(1,15)=2,02, p=0.18

* *Group×Session interaction: F(2,30)=1.02, p=0.40

n.s.*, **

a b

Figure 6. Intergroup variability. Comparison of younger and older children from the Experimental group of Study I. a. Difference waveforms recorded at the electrode F3. b. Amplitude Means across the two age-groups and sessions.

Page 34: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

34

No significant difference in the amplitude or latency of MMN elicited in the ‘In-front’, compared with the ‘On-sides’ condition was found for the vowels, complex tones, or simple tones (2-way ANOVA with the factors: Condition, Electrode). However, the P3a amplitudes for the vowels and complex tones in the ‘In-front’ condition were significantly larger than in the ‘On-sides’ condition (F(1,19)=7.34, P<0.01; and F(1,19)=5.16, P<0.03, respectively). For the simple tones, the main effect of Condition only approached significance (P<0.1). The effects were insignificant for all stimulus types in the analyses of the P3a latencies.

Complex tones

F3 F4

C3 C4

-5 µV

+5 µV

800 ms

Vowels Simple tones

In-frontOn-sides

MMN

P3a

1.8 m

LSTV

LS

In-front On-sides

0.4 m

TV

LS

Figure 7. Subtracted waveforms (standard-stimulus ERP subtracted from the deviant-stimulus ERPs) in the ´In-front´ condition interposed with those in the ´On-sides´ condition. Experimental set-up of Study III is given in the outlined box, bottom left. In the ´In-front´ condition the sources of audio (loudspeakers, LS) and visual (TV-set, TV) stimulation are integrated, whereas in the ´On-sides´ condition they are not.

Page 35: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

35

Study IV. MMNm in response to category change

MMNm responses were obtained by subtracting the response to standard stimuli from that to deviants recorded in the varying-standard condition (Fig. 8a). The ECDs of the MMNm sources explained, on average, 81% (N=8) of the magnetic signals over the left temporal cortex, and 75% (N=5) over the right. The MMNm generator sources over each hemisphere were modelled in five right-handed subjects. The locations and orientations of the dipoles are shown in Fig. 8b,c. The MMNm sources in the right hemisphere could not be modelled in the remaining three subjects because of the weakness of the response. In these cases, a ‘mirroring’ procedure of the sources locations was used to estimate the strength and direction of the source in the right temporal cortex, keeping the location symmetrical to that of the left-hemisphere MMNm ECDs. The mean values of the MMNm dipole moment were significantly larger in the left (12 nAm) than in the right (8 nAm) hemisphere in the right-handed subjects (t(7)=6.29, P<0.0005). In the left-handed subject, the MMNm dipole was stronger in the right than in the left hemisphere (Fig. 8d). In contrast, the N1m source dipole moments did not significantly differ between the hemispheres (23 vs. 21 nAm for the left and right hemisphere, respectively; t(7)=0.59, P=0.56).

Study V. Phonotopical gradient as indexed by N1m

The mean values of the spectrum for each phoneme (Fig. 1) represent perceptual properties such as timbre, loudness and pitch. It can be seen here that the most amplitude-intensive spectra belong to the exemplars from the [a] group compared with the [i] and [u] groups. The group-average spectra for each of the three categories correspond more to the formant structure of the vowels (Fig. 2). For all subjects, statistical analyses of source strengths, latencies, and source locations were performed. The ECDs explained, on average 90%, 88% and 93% of the magnetic signals of the N1m sources for the [a], [i] and [u] categories, respectively. Individual MEG data (Fig. 9a) superimposed on the individual MR image show that the N1m activity localizes to the auditory cortical areas of the superior bank of the superior temporal gyrus. The mean data for the ECD source strengths, latencies and locations across the three categories and over each hemisphere are presented in Fig. 9b. The main effect of the vowel category on N1m location was significant, which was revealed in the analyses of absolute locations [F(2,20)=4.56, P<0.05] and Euclidian distances [F(2,20)=4.81, P<0.05]. A comparison of the absolute locations revealed a significant difference only in the z coordinate (the inferior-superior axis). The [a] source (54.4 mm) was superior to that for [i] (49.5 mm). Their locations differed significantly

Page 36: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

36

from each other (P<0.01). The ECD of [u] was located between [a] and [i] (52.0 mm), but did not significantly differ from them. The Euclidian distances were further compared with the spectrum means across the vowel categories. The [i] and [u] categories with more similar spectral envelopes, as revealed in the analysis of normalized spectrum means, had close cortical representations, whereas the [a] category spanned a greater distance from the other two categories, having a dissimilar spectrum from them. A statistical analysis of the source strength revealed a main effect of the vowel category [F(2,20)=8.25, P<0.01; ε=0.76]. An LSD post-hoc test indicated that Q was the strongest for [i] (19 nAm) and significantly differed from those for [a] and [u] (14 nAm (P<0.001) and 17 nAm (P<0.01), respectively). No systematic effect on the N1m latencies across the categories, the hemispheres, or their interactions were found.

Page 37: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

37

100 fT/cm

500 ms0

DeviantStandardDifference waveform

Left hemisphere Right hemisphere

a Magnetic responses

MMNm

b MMNm isofield contours and dipoles

–50 0 50 mm

50 mm

–50 0 50 mm

50 mm

c Single-subject data Mean data

RL

RL

d MMNm dipole moments

HemisphereLeft Right

Re

lativ

e v

alu

es

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

S1 (9→7)

S2 (7→3)

S3 (22→16)

S4 (13→6)

S5 (10→7)S6 (19→15)

S7 (8→4)

S8 (10→4)

S9 (2→7 nAm)

Figure 8. (a) Data from a typical subject (S2). MEG responses from one gradiometer channel over the auditory cortex of each hemisphere. The MMNm component is seen in the difference waveforms (thick line) obtained by subtracting the responses to the standard (dashed line) stimuli from those to the deviant stimuli (thin line). (b) MMNm isofield contour maps (difference between the adjacent lines 8 fT/step) over each hemisphere superimposed on the helmet-shaped magnetometer array. The arrows indicate the locations and orientations of the MMNm ECDs. The dashed lines represent the magnetic flux entering and the solid lines exiting the head. (c). Left: locations for the MMNm generator sources in each hemisphere of a typical subject (S5), superimposed on the subjects’ MR images. Right: the corresponding mean locations indicated by circles (five right-handed subjects) of the MMNm. (d) Relative and absolute (given in parentheses) strengths of the MMNm ECDs for the eight right-handed and the left-handed subject S9. The standard errors of the mean for the MMNm ECD locations varied between 1 and 3mm.The diameters of the two circles are proportional to the corresponding mean values of the right and left dipole strengths.

Page 38: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

38

DISCUSSION

Children learn to discriminate non-native speech: ERP associated with second-language phonological awareness

The aim of Study I was to search for the neural correlates of learning a foreign language in pre-school and early-school-age children. MMN — the probe for language-specific memory representations (also called traces, (Näätänen et al., 1997)) — was used to monitor the neural processes underling preconscious discrimination of foreign speech utterances. Among the children joining either a French school or a French kindergarten where French was spoken almost all the time the MMN amplitude to French phoneme contrast increased strikingly in only a couple of months as a result of mere exposure to French. No such results were observed in the control group of age-matched Finnish-speaking children, who were naïve in French, proving that the increase was not merely due to age-related development or exposure to French during the recordings. These results not only corroborate the findings of Winkler et al. (1999a), who showed that

Figure 9. (a) Data from a typical subject (S2). The right-hemisphere N1m ECD source locations and orientations derived from MEG recordings are superimposed on the subject’s MR image (coronal slice), corresponding to the dipoles for the [a], [i], and [u] vowel categories. (b) The corresponding mean locations (based on the mean x, y, and z coordinates, in mm) of the N1m ECDs of all subjects are indicated by the spheres in the 3-D space. The diameters of the spheres are proportional to the dipole strengths for each of the three vowel categories for the right and left hemispheres.

10

20

30

40

50

60

-60-40

-200

2040

-10-5

510

15

Z

X Y

Left

hemisphere

Righ

hemisphere

[a], 15 nAm

[u], 17 nAm

60

a b

[i], 20 nAm

PA

Page 39: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

39

MMN for a contrast between the Finnish [e] and [ä] phonemes was elicited in Hungarians fluent in Finnish but not those naïve in it, but also uncover the temporal dynamics of second-language learning. Indeed, our findings imply that the memory representations of foreign phoneme systems develop remarkably quickly in young children. Amplitude comparison between the younger (3–4 years of age) and older (5–6 years) subgroups of children exposed to French revealed that the younger subjects developed cortical memory representations of the French vowels slightly (but not significantly) more quickly across the three experimental sessions. This supports the maturation-state hypothesis of early-age advantage in phoneme acquisition (Johnson and Newport, 1989), according to which the capacity to acquire language gradually declines through the critical period, but does not disappear entirely. The P3a and LDN components also increased throughout the six-month period within which an ability to discriminate French stimuli in a language-specific manner developed. Thus far, the P3a responses have been associated with the processing of novel stimuli and less with the subtle deviants (Picton, 1992; Alho et al., 1997; Escera et al., 2000). Indeed, speech stimuli in the language that has become increasingly familiar throughout the experimental sessions might be more attention-catching and thus, perhaps, more distracting. We suggest that the gradual P3a growth reflects the growing relevance of the foreign language and its ability to distract attention. In other words, the more effective detection of stimulus change based on the language-specific memory traces indexed by MMN could trigger the switching of involuntary attention to the deviant stimuli more effectively. Within the framework of the model for involuntary attention proposed by Escera et al. (2000), LDN appears to indicate a reorienting of attention that is similar to reorienting negativity, RON, recorded in adults (Schröger and Wolff, 1998; Berti and Schröger, 2001). The significant correlation between the LDN and the P3a mean amplitudes, but not between MMN and LDN, further supports this view.

Direction of children’s visual attention affects involuntary attention allocation to sounds

The ability to automatically re-orient attention towards salient sounds is of great biological significance. Vision is implicitly involved in orienting towards auditory stimuli, as demonstrated by the audio-visual reflex, and in involuntary eye-turning towards unexpected salient sounds (Paulsen and Ewertsen, 1966). Under natural conditions, observers relate what they see to what they hear, which may undergo transformation in ontogenesis from a series of empirical incidences to an automatized neural phenomenon. Phylogenetically, this reflex may have armed us with an inherent predisposition to enhance the sensory weight of the events occurring within our sight. Although MMN is not fully attention-independent (Alain and Woods, 1997; Gomes et al.,

Page 40: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

40

2000), its amplitude has been found not to be affected by visual processing load (Alho et al., 1992; Woods et al., 1992). The results of Study III showed that the MMN amplitude was not affected by the location of the sound source in the horizontal plane. In contrast, the P3a amplitudes were larger when vowels and spectrally rich complex tones, which the subjects were instructed to ignore, were presented from the visually attended space in front of the subject. Interestingly, the P3a elicited by the sinusoidal tones was not affected by the location of the sound source. It is conceivable that a poor acoustic content of single-partial tones was insufficient to exceed the perceptual threshold or to trigger an involuntary attention switch, irrespective of the sound-source location. The present results indicate that the stimulation conditions in which the sound is coupled with the direction of visual attention appear to be beneficial for the P3a elicitation in children, even when there are small differences between the standard and deviant stimuli.

Hemispheric asymmetry of the processing of the category change in speech perception. Evidence from the MEG studies

The main finding of Study IV was that, even when we used natural speech stimuli that were constantly changing and varying acoustically, MMNm to the phoneme-category channel was elicited. This result confirms and extends previous findings involving phonological MMN recorded using synthesized stimuli that were constrained by relatively few exemplars in order to control for the acoustical variation (Dehaene-Lambertz, 1997; Dehaene-Lambertz et al., 2000; Phillips et al., 2000). However, in the present study, in which speech sounds were uttered by hundreds of speakers and recorded outside the laboratory, the experimental situation was modelled so as to resemble the real-life one as much as possible. Consistently with previous findings (Näätänen et al., 1997; Rinne et al., 1999; Kasai et al., 2001), the dipole moments indicated the dominance of the MMNm responses in the left hemisphere in the right-handed subjects, while the opposite pattern was revealed in the left-handed subject. The strong lateralization observed during vowel categorization, which in some subjects resulted in the absence of the MMNm ECD in the contralateral hemisphere, corroborates the view of left-hemisphere predominance in the process subserving the automatic extraction of invariant phoneme features, and consequently the perception of across-category change. The presence in N1m hemispheric differences reported earlier by Gootjes et al. (1999) was not confirmed in our study. The inconsistency in the N1m lateralization results between the two experiments was presumably due to the differences in the experimental tasks: no attention was paid to the stimuli in our experiment, whereas Gootjes et al. used an active discrimination task in which the top-down control could change the N1m function from the acoustic-specific to the phonetic-specific.

Page 41: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

41

Phonemotopic representations in the human auditory cortex

The aim of Study V of the present dissertation was to identify a phonemotopic map of vowel phonemes in the human brain. Using stimuli that have been well controlled in frequency, duration, and intensity, some researchers have reported evidence suggesting the existence of such a phonemotopic map (Ohl and Scheich, 1997; Obleser et al., 2003). We attempted to reinvestigate this issue using speech sounds uttered by 450 different speakers and recorded outside the laboratory, thus including natural background noise. Consistently with previous studies, we found that the strengths and locations of the N1m cortical sources were affected by the vowel category. The strongest sources were found for [i]. The attenuated amplitude in response to [a] compared with those for [i] and [u] corroborates the results reported earlier by Obleser et al. (2003) for the German [a], [i] and [e] phonemes. This could be explained by the formant inhibition hypothesis (Diesch and Luce, 1997; Diesch and Luce, 2000), according to which closely neighbouring formant peaks elicit weaker cortical responses. The N1m data obtained in the present work comply with this hypothesis: the strongest response was elicited for the [i] phoneme with its bigger interformant distance than the other two phonemes. Our finding that spectral distances were mirrored in the cortical representations of the vowels is in line with the results of previous MEG studies on phonemotopy (Diesch and Luce, 1997; Diesch and Luce, 2000; Obleser et al., 2003). In their N1m study, Obleser (2003) found that Euclidian distances correlated with spectral dissimilarities across the vowels of the German language. Likewise, the ECD locations of the acoustically dissimilar Russian vowel pairs [a]-[i] and [a]-[u] spanned greater distances than the more similar [i] and [u] pair. The effect of vowel category on the absolute dipole locations was significant along the inferior-superior axis. Thus, our results closely mirror those of Obleser et al. (2003). Importantly, the electrophysiological evidence obtained in a study on vowel discrimination in gerbils (Ohl et al., 1997) is in agreement with the formant inhibition hypothesis: the single- and multi-unit activity of the AI neurons was proportional to a given formant distance of the vowel stimulus. In the autoradiographic imaging part of their animal study, Ohl et al. (1997) discovered that topographic representations of vowels occurred along the isofrequency bands that are known to be oriented perpendicular to the surface of the cortex. This finding supports the notion of tonotopic- and phonemotopic-gradient orthogonality (Diesch and Luce, 2000). Accordingly, our observation is consistent with the view that phonotopic and tonotopic maps are independent of each other (Diesch and Luce, 1997; Ohl and Scheich, 1997; Diesch and Luce, 2000).

Page 42: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

42

CONCLUSIONS

The ability to categorically discriminate speech sounds depends on language-specific memory traces reflected by brain responses such as MMN. The studies that comprise this dissertation were the first to expose the dynamics of the formation of second-language memory traces using MMN. According to the data, children learned to discriminate non-native phonemes within two months: the MMN amplitude increased during the test period, and its latency decreased. Moreover, the P3a and LDN amplitudes also increased during the six-month period within which the ability to discriminate French stimuli in a language-specific manner developed. We interpreted this gradual P3a growth as reflecting the growing relevance of the foreign language and its ability to distract attention. The results of the correlation analysis allowed us to conclude that LDN presumably indicates a reorienting of attention that is similar to reorienting negativity, RON, recorded in adults. The positive correlation between the LDN mean amplitude and the P3a, not MMN, supports this view. We also found that during extensive acoustic variation, robust MMNm can be recorded in response to the change across phonemic categories. This process seems to be strongly lateralized to the left auditory cortex in adults when natural speech sounds are used and is in line with the hypothesis that language-specific memory traces are located in the left hemisphere. The fact that MMNm occurred earlier in the left than in the right hemisphere supports the view that the left auditory cortex is predominant in the speech-perception function. Taken together, the findings concerning language-specific MMN in children and abstract MMN in adults obtained in the studies strengthen the ‘sensory intelligence’ hypothesis. These results imply the presence of the language-specific memory traces that underlie the invariant perception of phonemes, a pre-attentive cognitive operation in the human brain. The existence of a phonemotopic map in the human brain is supported by the evidence obtained in these studies that the N1m vowel representations differ in strength and location. Even when there was wide acoustic variation, we found that the spectral distances and differential distributions of the vowel categories were mirrored in the N1m cortical representations. In terms of the two methodological issues that were addressed in the present dissertation, and the results of these studies revealed the following. (1) MMN seems not to be affected by the location of a sound source in the horizontal plane. In contrast, the P3a amplitude increases when complex stimuli are presented from the attended space in front of the subject. This suggests that the stimulation condition in which the sound is coupled with the direction of visual attention appears to be beneficial to the P3a elicitation in children, and could be used to study distraction. (2) A more robust category-specific MMNm can

Page 43: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

43

be recorded when the varying-standard paradigm — a newly developed modification of the conventional oddball paradigm — is used. This work was aimed at achieving a better understanding of the effects of stimulation paradigms and conditions when recording change-related responses to speech sounds in children and adults. The specific findings of this thesis contribute to the general knowledge about MMN in speech, and could lead to improvements in the practical arrangements of studies in the future. It is especially important to work towards minimizing the acoustic component and obtaining results that reflect linguistic processing in the brain.

Page 44: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

44

References

Abeles M, Goldstein MH (1970) Functional architecture in cat primary auditory cortex: columnar organization and organization according to depth. J Neurophysiol 33:172-187.

Alain C, Woods DL (1997) Attention modulates auditory pattern memory as indexed by event-related brain potentials. Psychophysiology 34:534-546.

Alho K (1995) Cerebral generators of mismatch negativity (MMN) and its magnetic counterpart (MMNm) elicited by sound changes. Ear Hear 16:38-51.

Alho K, Escera C, Diaz R, Yago E, Serra JM (1997) Effects of involuntary auditory attention on visual task performance and brain activity. Neuroreport 8:3233-3237.

Alho K, Sainio K, Sajaniemi N, Reinikainen K, Näätänen R (1990) Event-related brain potential of human newborns to pitch change of an acoustic stimulus. Electroencephalogr Clin Neurophysiol 77:151-155. Alho K, Winkler I, Escera C, Huotilainen M, Virtanen J, Jaaskelainen IP, Pekkonen E, Ilmoniemi RJ (1998) Processing of novel sounds and frequency changes in the human auditory cortex: magnetoencephalographic recordings. Psychophysiology 35:211-224.

Alho K, Woods DL, Algazi A, Näätänen R (1992) Intermodal selective attention. II. Effects of attentional load on processing of auditory and visual stimuli in central space. Electroencephalogr Clin Neurophysiol 82:356-368. Alku P, Tiitinen H, Näätänen R (1999) A method for generating natural-sounding speech stimuli for cognitive brain research. Clin Neurophysiol 110:1329-1333. Aulanko R, Hari R, Lounasmaa OV, Näätänen R, Sams M (1993) Phonetic invariance in the human auditory cortex. Neuroreport 4:1356-1358. Berti S, Schröger E (2001) A comparison of auditory and visual distraction effects: behavioral and event-related indices. Brain Res Cogn Brain Res 10:265-273. Bladon RA, Lindblom B (1981) Modelling the judgment of vowel quality differences. J Acoust Soc Am 69:1414-1422. Brugge JF (1992) An overview of central auditory processing. In: The Mammalian Auditory Pathway: Neurophysiology (Popper AN, Fay RR, eds), pp 1-33. New York: Springer Verlag. Ceponiene R, Kushnerenko E, Fellman V, Renlund M, Suominen K, Näätänen R (2002a) Event-related potential features indexing central auditory discrimination by newborns. Brain Res Cogn Brain Res 13:101-113.

Ceponiene R, Yaguchi K, Shestakova A, Alku P, Suominen K, Näätänen R (2002b) Sound complexity and 'speechness' effects on pre-attentive auditory discrimination in children. Int J Psychophysiol 43:199-211. Cheour M, Alho K, Ceponiene R, Reinikainen K, Sainio K, Pohjavuori M, Aaltonen O, Näätänen R (1998a) Maturation of mismatch negativity in infants. Int J Psychophysiol 29:217-226.

Page 45: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

45

Cheour M, Ceponiene R, Lehtokoski A, Luuk A, Allik J, Alho K, Näätänen R (1998b) Development of language-specific phoneme representations in the infant brain. Nat Neurosci 1:351-353. Cheour M, Leppänen PH, Kraus N (2000) Mismatch negativity (MMN) as a tool for investigating auditory discrimination and sensory memory in infants and children. Clin Neurophysiol 111:4-16. Cowan N, Winkler I, Teder W, Näätänen R (1993) Memory prerequisites of mismatch negativity in the auditory event-related potential (ERP). J Exp Psychol Learn Mem Cogn 19:909-921. Csepe V (1995) On the origin and development of the mismatch negativity. Ear Hear 16:91-104.

Dehaene-Lambertz G (1997) Electrophysiological correlates of categorical phoneme perception in adults. Neuroreport 8:919-924.

Dehaene-Lambertz G, Baillet S (1998) A phonological representation in the infant brain. Neuroreport 9:1885-1888. Dehaene-Lambertz G, Dupoux E, Gout A (2000) Electrophysiological correlates of phonological processing: a cross-linguistic study. J Cogn Neurosci 12:635-647.

Dewson JH, 3rd (1964) Speech sound discrimination by cats. Science 144:555-556. Dewson JH, 3rd, Pribram KH, Lynch JC (1969) Effects of ablations of temporal cortex upon speech sound discrimination in the monkey. Exp Neurol 24:579-591. Diesch E, Eulitz C, Hampson S, Ross B (1996) The neurotopography of vowels as mirrored by evoked magnetic field measurements. Brain Lang 53:143-168. Diesch E, Luce T (1997) Magnetic fields elicited by tones and vowel formants reveal tonotopy and nonlinear summation of cortical activation. Psychophysiology 34:501-510. Diesch E, Luce T (2000) Topographic and temporal indices of vowel spectral envelope extraction in the human auditory cortex. J Cogn Neurosci 12:878-893. Eggermont JJ (2001) Between sound and perception: reviewing the search for a neural code. Hear Res 157:1-42.

Eimas, P.D. Auditory and phonetic coding of the cues for speech: discrimination of the [r-l] distinction by young infants. Perception and Psychophysics, 18 (5), 341-347.

Escera C, Alho K, Schröger E, Winkler I (2000) Involuntary attention and distractibility as evaluated with event-related brain potentials. Audiol Neurootol 5:151-166.

Giard MH, Perrin F, Echallier JF, Thevenet M, Froment JC, Pernier J (1994) Dissociation of temporal and frontal components in the human auditory N1 wave: a scalp current density and dipole model analysis. Electroencephalogr Clin Neurophysiol 92:238-252. Giard MH, Perrin F, Pernier J, Bouchet P (1990) Brain generators implicated in the processing of auditory stimulus deviance: a topographic event-related potential study. Psychophysiology 27:627-640.

Goldstein EB (2002) Sensation and Perception: Pacific Grove, CA Wadsworth-Thomson Learning cop.

Gomes H, Molholm S, Ritter W, Kurtzberg D, Cowan N, Vaughan HG (2000) Mismatch negativity in children and adults, and effects of an attended task. Psychophysiology 37:807-816.

Page 46: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

46

Gootjes L, Raij T, Salmelin R, Hari R (1999) Left-hemisphere dominance for processing of vowels: a whole-scalp neuromagnetic study. Neuroreport 10:2987-2991.

Hackett TA, Preuss TM, Kaas JH (2001) Architectonic identification of the core region in auditory cortex of macaques, chimpanzees, and humans. J Comp Neurol 441:197-222.

Hackett TA, Stepniewska I, Kaas JH (1998) Thalamocortical connections of the parabelt auditory cortex in macaque monkeys. J Comp Neurol 400:271-286.

Hämäläinen M, Hari R, Ilmoniemi RJ, Knuutila J, Lounasmaa OV (1993) Magnetoencephalography - Theory, instrumentation, and applications to noninvasive studies of the working human brain. Reviews of Modern Physics 65:413-497. Hari R (1990) Magnetic evoked fields of the human brain: basic principles and applications. Electroencephalogr Clin Neurophysiol Suppl 41:3-12. Hari R, Hämäläinen M, Ilmoniemi R, Kaukoranta E, Reinikainen K, Salminen J, Alho K, Näätänen R, Sams M (1984) Responses of the primary auditory cortex to pitch changes in a sequence of tone pips: neuromagnetic recordings in man. Neurosci Lett 50:127-132.

Hari R, Kaila K, Katila T, Tuomisto T, Varpula T (1982) Interstimulus interval dependence of the auditory vertex response and its magnetic counterpart: implications for their neural generation. Electroencephalogr Clin Neurophysiol 54:561-569. Hari R, Rif J, Tiihonen J, Sams M (1992) Neuromagnetic mismatch fields to single and paired tones. Electroencephalogr Clin Neurophysiol. 82:152-154. Harley TA (1995) The psychology of language. From data to Theory. Hove: Psychology Press. Huotilainen M, Ilmoniemi RJ, Lavikainen J, Tiitinen H, Alho K, Sinkkonen J, Knuutila J, Näätänen R (1993) Interaction between representations of different features of auditory sensory memory. Neuroreport 4:1279-1281. Huotilainen M, Kujala A, Alku P (2001) Long-term memory traces facilitate short-term memory trace formation in audition in humans. Neurosci Lett 310:133-136.

Huotilainen M, Kujala A, Hotakainen M, Shestakova A, Kushnerenko E, Parkkonen L, Fellman V, Näätänen R (2003) Auditory magnetic responses of healthy newborns. Neuroreport 14:1871-1875. Huotilainen M, Tiitinen H, Lavikainen J, Ilmoniemi RJ, Pekkonen E, Sinkkonen J, Laine P, Näätänen R (1995) Sustained fields of tones and glides reflect tonotopy of the auditory cortex. Neuroreport 6:841-844.

Huotilainen M, Winkler I, Alho K, Escera C, Virtanen J, Ilmoniemi RJ, Jaaskelainen IP, Pekkonen E, Näätänen R (1998) Combined mapping of human auditory EEG and MEG responses. Electroencephalogr Clin Neurophysiol 108:370-379. Ito M, Tsuchida J, Yano M (2001) On the effectiveness of whole spectral shape for vowel perception. J Acoust Soc Am 110:1141-1149. Jansson-Verkasalo E, Korpilahti P, Jantti V, Valkama M, Vainionpaa L, Alku P, Suominen K, Näätänen R (2004) Neurophysiologic correlates of deficient phonological representations and object naming in prematurely born children. Clin Neurophysiol 115:179-187.

Page 47: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

47

Javitt DC, Steinschneider M, Schroeder CE, Arezzo JC (1996) Role of cortical N-methyl-D-aspartate receptors in auditory sensory memory and mismatch negativity generation: implications for schizophrenia. Proc Natl Acad Sci U S A 93:11962-11967. Johnson JS, Newport EL (1989) Critical period effects in second language learning: the influence of maturational state on the acquisition of English as a second learning. Cognitive Psychology 21:60-99.

Kaas JH, Hackett TA (1998a) Subdivisions of auditory cortex and levels of processing in primates. Audiol Neurootol 3:73-85.

Kaas JH, Hackett TA (1999) 'What' and 'where' processing in auditory cortex. Nat Neurosci 2:1045-1047.

Kaas JH, Hackett TA (2000) Subdivisions of auditory cortex and processing streams in primates. Proc Natl Acad Sci U S A 97:11793-11799.

Kasai K, Yamada H, Kamio S, Nakagome K, Iwanami A, Fukuda M, Itoh K, Koshida I, Yumoto M, Iramina K, Kato N, Ueno S (2001) Brain lateralization for mismatch response to across- and within-category change of vowels. Neuroreport 12:2467-2471. Korpilahti P, Krause CM, Holopainen I, Lang AH (2001) Early and late mismatch negativity elicited by words and speech-like stimuli in children. Brain Lang 76:332-339. Korpilahti P, Lang HA (1994) Auditory ERP components and mismatch negativity in dysphasic children. Electroencephalogr Clin Neurophysiol 91:256-264. Korzyukov O, Alho K, Kujala A, Gumenyuk V, Ilmoniemi RJ, Virtanen J, Kropotov J, Naatanen R (1999) Electromagnetic responses of the human auditory cortex generated by sensory-memory based processing of tone-frequency changes. Neurosci Lett 276:169-172.

Kraus N, Cheour M (2000) Speech sound representation in the brain. Audiol Neurootol 5:140-150. Kraus N, McGee T, Carrell T, King C, Littman T, Nicol T (1994) Discrimination of speech-like contrasts in the auditory thalamus and cortex. J Acoust Soc Am 96:2758-2768. Kraus N, McGee T, Carrell T, Sharma A, Nicol T (1995) Mismatch negativity to speech stimuli in school-age children. Electroencephalogr Clin Neurophysiol Suppl 44:211-217. Kraus N, McGee T, Sharma A, Carrell T, Nicol T (1992) Mismatch negativity event-related potential elicited by speech stimuli. Ear Hear 13:158-164. Kropotov JD, Näätänen R, Sevostianov AV, Alho K, Reinikainen K, Kropotova OV (1995) Mismatch negativity to auditory stimulus change recorded directly from the human temporal cortex. Psychophysiology 32:418-422.

Krumhansl CL, Iverson P (1992) Perceptual interaction between musical pitch and timbre. J. Exp. Psychol 18:739-751.

Kuhl PK (2000) A new view of language acquisition. Proc Nat Acad Sci U S A 97:11850-11857. Kuhl PK, Williams KA, Lacerda F, Stevens KN, Lindblom B (1992) Linguistic experience alters phonetic perception in infants by 6 months of age. Science 255:606-608.

Page 48: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

48

Kurtzberg D, Vaughan HG, Kreuzer JA, Fliegler KZ (1995) Developmental studies and clinical application of mismatch negativity: problems and prospects. Ear Hear 16:105-117.

Kushnerenko E, Ceponiene R, Balan P, Fellman V, Näätänen R (2002) Maturation of the auditory change detection response in infants: a longitudinal ERP study. Neuroreport 13:1843-1848.

Langner G, Sams M, Heil P, Schulze H (1997) Frequency and periodicity are represented in orthogonal maps in the human auditory cortex: evidence from magnetoencephalography. J Comp Physiol [A] 181:665-676. Liberman AM, Cooper FS, Shankweiler DP, Studdert-Kennedy M (1967) Perception of the speech code. Psychol Rev 74:431-461. Liberman AM, Mattingly IG (1985) The motor theory of speech perception revised. Cognition 21:1-36. Lütkenhöner B, Krumbholz K, Seither-Preisler A (2003) Studies of tonotopy based on wave N100 of the auditory evoked field are problematic. Neuroimage 19:935-949. Mattingly LG, Liberman AM, Syrdal AK, Halwes T (1971) Discrimination in speech and nonspeech modes. Cognitive Psychology 2:131-157. Merzenich MM, Brugge JF (1973) Representation of the cochlear partition of the superior temporal plane of the macaque monkey. Brain Res 50:275-296. Moore BCJ (2003) An introduction to the psychology of hearing, 3rd Edition. San Diego (CA): Academic Press. Moore JK (2002) Maturation of human auditory cortex: implications for speech perception. Ann Otol Rhinol Laryngol Suppl 189:7-10. Moore JK, Guan YL (2001) Cytoarchitectural and axonal maturation in human auditory cortex. J Assoc Res Otolaryngol 2:297-311. Morel A, Garraghty PE, Kaas JH (1993) Tonotopic organization, architectonic fields, and connections of auditory cortex in macaque monkeys. J Comp Neurol 335:437-459.

Morr ML, Shafer VL, Kreuzer JA, Kurtzberg D (2002) Maturation of mismatch negativity in typically developing infants and preschool children. Ear Hear 23:118-136.

Näätänen (1992) Attention and Brain Function. Hillsdale: Lawrence Erlbaum. Näätänen R (2001) The perception of speech sounds by the human brain as reflected by the mismatch negativity (MMN) and its magnetic equivalent (MMNm). Psychophysiology 38:1-21. Näätänen R, Gaillard AW, Mantysalo S (1978) Early selective-attention effect on evoked potential reinterpreted. Acta Psychol (Amst) 42:313-329. Näätänen R, Lehtokoski A, Lennes M, Cheour M, Huotilainen M, Iivonen A, Vainio M, Alku P, Ilmoniemi RJ, Luuk A, Allik J, Sinkkonen J, Alho K (1997) Language-specific phoneme representations revealed by electric and magnetic brain responses. Nature 385:432-434.

Näätänen R, Michie PT (1979) Early selective-attention effects on the evoked potential: a critical review and reinterpretation. Biol Psychol 8:81-136.

Näätänen R, Paavilainen P, Tiitinen H, Jiang D, Alho K (1993) Attention and mismatch negativity. Psychophysiology 30:436-450.

Page 49: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

49

Näätänen R, Picton T (1987) The N1 wave of the human electric and magnetic response to sound: a review and an analysis of the component structure. Psychophysiology 24:375-425.

Näätänen R, Tervaniemi M, Sussman E, Paavilainen P, Winkler I (2001) "Primitive intelligence" in the auditory cortex. Trends Neurosci 4:483-288.

Näätänen R, Winkler I (1999) The concept of auditory stimulus representation in cognitive neuroscience. Psychol Bull 125:826-859.

Obleser J, Elbert T, Lahiri A, Eulitz C (2003) Cortical representation of vowels reflects acoustic dissimilarity determined by formant frequencies. Brain Res Cogn Brain Res 15:207-213.

Ohl FW, Scheich H (1997) Orderly cortical representation of vowels based on formant interaction. Proc Natl Acad Sci USA 94:9440-9444.

Oldfield RC (1971) The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia 9:97-113. Opitz B, Rinne T, Mecklinger A, von Cramon DY, Schröger E (2002) Differential contribution of frontal and temporal cortices to auditory change detection: fMRI and ERP results. Neuroimage 15:167-174. Paavilainen P, Jaramillo M, Näätänen R, Winkler I (1999) Neuronal populations in the human brain extracting invariant relationships from acoustic variance. Neurosci Lett 265:179-182. Paavilainen P, Simola J, Jaramillo M, Näätänen R, Winkler I (2001) Preattentive extraction of abstract feature conjunctions from auditory stimulation as reflected by the mismatch negativity (MMN). Psychophysiology 38:359-365. Paavilainen P, Tiitinen H, Alho K, Näätänen R (1993) Mismatch negativity to slight pitch changes outside strong attentional focus. Biol Psychol 37:23-41.

Pantev C, Bertrand O, Eulitz C, Verkindt C, Hampson S, Schuierer G, Elbert T (1995) Specific tonotopic organizations of different areas of the human auditory cortex revealed by simultaneous magnetic and electric recordings. Electroencephalogr Clin Neurophysiol 94:26-40.

Pantev C, Eulitz C, Elbert T, Hoke M (1994) The auditory evoked sustained field: origin and frequency dependence. Electroencephalogr Clin Neurophysiol 90:82-90.

Pantev C, Hoke M, Lehnertz K, Lütkenhöner B, Anogianakis G, Wittkowski W (1988) Tonotopic organization of the human auditory cortex revealed by transient auditory evoked magnetic fields. Electroencephalogr Clin Neurophysiol 69:160-170. Paulsen J, Ewertsen HW (1966) Audio-visual reflex: determination of the audio-visual reflex in directional hearing by employment of electronystagmography. Acta Otolaryngol :Suppl:211. Phillips C (2001) Levels of representation in the electrophysiology of speech perception. Cognitive Science 25:711-731. Phillips C, Pellathy T, Marantz A, Yellin E, Wexler K, Poeppel D, McGinnis M, Roberts T (2000) Auditory cortex accesses phonological categories: an MEG mismatch study. J Cogn Neurosci 12:1038-1055. Picton TW (1992) The P300 wave of the human event-related potential. J Clin Neurophysiol 9:456-479.

Page 50: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

50

Poeppel D, Phillips C, Yellin E, Rowley HA, Roberts TP, Marantz A (1997) Processing of vowels in supratemporal auditory cortex. Neurosci Lett 221:145-148.

Potapova PK (1997) General and Applied phonetics, 2nd Edition. Moscow: Moscow State University.

Rauschecker JP (1998) Cortical processing of complex sounds. Curr Opin Neurobiol 8:516-521. Rauschecker JP, Tian B, Hauser M (1995) Processing of complex sounds in the macaque nonprimary auditory cortex. Science 268:111-114. Rinne T, Alho K, Alku P, Holi M, Sinkkonen J, Virtanen J, Bertrand O, Näätänen R (1999) Analysis of speech sounds is left-hemisphere predominant at 100-150ms after sound onset. Neuroreport 10:1113-1117.

Romani GL, Williamson SJ, Kaufman L (1982) Tonotopic organization of the human auditory cortex. Science 216:1339-1340. Ruusuvirta T, Korhonen T, Arikoski J, Kivirikko K (1996) Multiple-unit responses to pitch changes in rabbits. Neuroreport 7:1266-1268.

Sams M, Kaukoranta E, Hamalainen M, Näätänen R (1991) Cortical activity elicited by changes in auditory stimuli: different sources for the magnetic N100m and mismatch responses. Psychophysiology 28:21-29. Sams M, Paavilainen P, Alho K, Näätänen R (1985) Auditory frequency discrimination and event-related potentials. Electroencephalogr Clin Neurophysiol 62:437-448. Schönwiesner M, von Cramon DY, Rubsamen R (2002) Is it tonotopy after all? Neuroimage 17:1144-1161. Schröger E (1996) A neural mechanism for involuntary attention shifts to changes in auditory stimulation. J Cogn Neurosci 8:527-539. Schröger E, Wolff C (1998) Attentional orienting and reorienting is indicated by human event-related brain potentials. Neuroreport 9:3355-3358.

Shafer VL, Morr ML, Kreuzer JA, Kurtzberg D (2000) Maturation of mismatch negativity in school-age children. Ear Hear 21:242-251.

Squires NK, Squires KC, Hillyard SA (1975) Two varieties of long-latency positive waves evoked by unpredictable auditory stimuli in man. Electroencephalogr Clin Neurophysiol 38:387-401. Streeter LA (1976) Language perception of 2-month-old infants shows effects of both innate mechanisms and experience. Nature 259:39-41. Sussman E, Ritter W, Vaughan HG (1998) Attention affects the organization of auditory input associated with the mismatch negativity system. Brain Res 789:130-138. Sussman E, Ritter W, Vaughan HG (1999) An investigation of the auditory streaming effect using event-related brain potentials. Psychophysiology 36:22-34. Sussman H (2000) Phonemic representation: a twenty-first century challenge. Brain Lang 71:237-240.

Page 51: ‘SENSORY INTELLIGENCE’ IN THE AUDITORY CORTEX: BRAIN ...ethesis.helsinki.fi/julkaisut/kay/psyko/vk/shestakova/sensoryi.pdf · RESPONSES TO NATIVE AND NON-NATIVE PHONETIC STIMULI

51

Talwar SK, Musical PG, Gerstein GL (2001) Role of mammalian auditory cortex in the perception of elementary sound properties. J Neurophysiol 85:2350-2358.

Tervaniemi M, Maury S, Näätänen R (1994) Neural representations of abstract stimulus features in the human brain as reflected by the mismatch negativity. Neuroreport 5:844-846.

Tervaniemi M, Medvedev SV, Alho K, Pakhomov SV, Roudas MS, Van Zuijen TL, Näätänen R (2000) Lateralized automatic auditory processing of phonetic versus musical information: a PET study. Hum Brain Mapp 10:74-79. Tesche CD, Uusitalo MA, Ilmoniemi RJ, Huotilainen M, Kajola M, Salonen O (1995) Signal-space projections of MEG data characterize both distributed and well-localized neuronal sources. Electroencephalogr Clin Neurophysiol 95:189-200.

Tiitinen H, Alho K, Huotilainen M, Ilmoniemi RJ, Simola J, Näätänen R (1993) Tonotopic auditory cortex and the magnetoencephalographic (MEG) equivalent of the mismatch negativity. Psychophysiology 30:537-540. Uusitalo MA, Ilmoniemi RJ (1997) Signal-space projection method for separating MEG or EEG into components. Med Biol Eng Comput 35:135-140. Versnel H, Shamma SA (1998) Spectral-ripple representation of steady-state vowels in primary auditory cortex. J Acoust Soc Am 103:2502-2514. Whalen DH, Liberman AM (1987) Speech perception takes precedence over nonspeech perception. Science 237:169-171 Werker JF, Tees RC (1983) Developmental changes across childhood in the perception of non-native speech sounds. Can J Psychol 37:278-286. Wessinger CM, VanMeter J, Tian B, Van Lare J, Pekar J, Rauschecker JP (2001) Hierarchical organization of the human auditory cortex revealed by functional magnetic resonance imaging. J Cogn Neurosci 13:1-7. Winkler I, Kujala T, Tiitinen H, Sivonen P, Alku P, Lehtokoski A, Czigler I, Csepe V, Ilmoniemi RJ, Näätänen R (1999a) Brain responses reveal the learning of foreign language phonemes. Psychophysiology 36:638-642.

Winkler I, Lehtokoski A, Alku P, Vainio M, Czigler I, Csepe V, Aaltonen O, Raimo I, Alho K, Lang H, Iivonen A, Näätänen R (1999b) Pre-attentive detection of vowel contrasts utilizes both phonetic and auditory memory representations. Brain Res Cogn Brain Res 7:357-369. Woods DL, Alho K, Algazi A (1992) Intermodal selective attention. I. Effects on event-related potentials to lateralized auditory and visual stimuli. Electroencephalogr Clin Neurophysiol 82:341-355.