1

Project Proposal STW/Cognition programme

On-line Interpretation of Imagined TemporalPatterns from EEG:The next step towards neuronal control of motor andcommunication prostheses.

Dr. P. DesainProf. Dr. C. GielenUniversity of Nijmegen

2

1 Project title

On-line Interpretation of Imagined Temporal EEG Patterns: The next steptowards controlling neuronal motor prostheses.

2.1 ApplicantsDr. ir. P. Desain'Music, Mind, Machine' groupNijmegen Institute for Cognition and Information (NICI)Nijmegen UniversityP.O.Box 91046500 HE NIJMEGENThe Netherlandstel: ++ 31-24-3615885fax: ++ 31-24-3616066email: Desain@nici.kun.nl

Prof. dr. C.C.A.M. GielenLab. Medical and Biophysics (MBFYS)Nijmegen UniversityGeert Grooteplein 216525 EZ NIJMEGENtel: ++ 31-24-6314242fax: ++ 31-24-6341435email: stan@snn.kun.nl

2.2 Research Group

Medical Physics and Biophysics (NICI, KUN)Staff: Prof. Dr. C. Gielen

Dr. H. KappenPost doc.: Dr. YpmaTechnical staff: Drs. G. van Lingen

Ing. G. Windau

'Music, Mind, Machine' group (NICI, KUN)Staff: Dr. P. DesainPhD: K. JenksTechnical staff: P. Trilsbeek

St. MaartenskliniekStaff: Prof. Dr. J. Duysens

Dr. D. van Kuppevelt

3

3 Project Summary

3.1 ResearchThis project aims to combine recent advances in brain imaging and signal

processing to develop new tools for direct man-machine interaction (BMI, brain-machine interaction). Recent studies have shown that imagining the execution of aparticular sensori-motor task gives rise to almost the same pattern of neuronal activityin the central nervous system as actual performance of the sensori-motor task. Wewill use this observation and new neuro-imaging methods to develop new methodsthat enable patients with impairments of the motor system to control devices and evento communicate (via a text display or speech synthesizer).

The idea to “read out” the brain by measuring EEG signals as a measure forintended actions and to use these signals for rehabilitation in patients with motordisorders is not new. The state of the art is that correct decoding of the EEG signal ispossible to a very large extend. Although this may seem impressive, it is still not goodenough for applications since the erroneous responses in a remaining 10% can lead tocompletely wrong actions.

The key idea of this proposal is to use temporal modulation of EEG byimagining temporal patterns, such as musical rhythm, imagining a task at a frequencyof 1 Hz or a Morse-like code, to enhance the correct decoding rate. Traces of neuralactivity caused by mental imagery will be picked up by sensors (EEG) and will beclassified in discrete categories. These categories will correspond to specificcommands, characters or words. In a pilot study we already have been able to proofthe feasibility of this method. In contrast with most other BMI work, our approachwill use non-invasive measurement techniques, and discrete (symbolic) output.Furthermore, it will not rely on intensive (bio-feedback) training of the user. Instead,the system itself will be adaptive to recognize EEG patterns as representations ofresponse categories on the basis of known examples of measurements of eachcategory separately. To achieve this goal, sophisticated methods to extract therelevant information from the noisy single trial data (e.g. Independent ComponentAnalysis) will be applied, as well as state-of-the-art classification methods.

This proposal deals mainly with the recording and data analysis of EEGsignals. Using the results for clinical studies (e.g. functional electro-stimulation ofmuscles or nerves) would become feasible when the success rate of correctinterpretation of EEG signals has been improved.

3.2 Utilization

There is a great demand to measure neuronal activity and to interpret brain signals tobypass damaged brain structures with the aim to assist patients during rehabilitation,or to assist patients with motor disorders.

There are several companies in The Netherlands, which focus on the analysisand interpretation of biomedical signals. Biosemi and Brain Research Companyconsider the proposed project as a logical next step to the existing techniques forneuroengineering. Their interest in the results of this proposal becomes evident fromthe major investments that these companies are willing to contribute.

The Sony research center in Paris, with various directions of research in musicand language, also works on new data-analysis techniques (software and hardware) in

4

the field of neuroscience. Sony is interested in these applications for the control ofvarious appliances and recently founded a new research group for that aim in Tokyo.

The St. Maartenskliniek is one of the major rehabilitation institutes in TheNetherlands, and is associated to the University Medical Center Nijmegen. Thisinstitute is eager to serve as a test-bed for new advanced technology for rehabilitation.Their interest in using EEG classification for symbolic manipulation can beunderstood from the goal to enable patients with severe paralyses to control devices,like a TV set or a wheel chair.

3.3 Nederlandse samenvattingHet doel van het project is om recente ontwikkelingen op het gebied van 'brainimaging' en signaal verwerking te gebruiken voor mens-machine interactie doormiddel van classificatie van EEG signalen (BMI, Brain-Machine Interaction). Recentestudies hebben aangetoond, dat het in gedachten uitvoeren van een taak (waarnemingof beweging) bijna hetzelfde patroon van neuronale activiteit oplevert als dewaarneming of beweging zelf. Dit gegeven wordt nu al door diverse groepen gebruiktom op basis van de EEG activiteit acties te ondernemen. Zeer experimenteletoepassingen bestaan reeds bij patienten met dwarsleasies of met verlammingen tgvhersenbloedingen. Een probleem hierbij is dat de klassificatie/decodering van de EEGsignalen slechts in ongeveer 90% succesvol is, waardoor er in 10% van de gevallenfoutieve acties worden ondernomen. Dit maakt de methode voor practischetoepassingen onbruikbaar. Het centrale idee van onze aanvraag is om temporelemodulatie van EEG activiteit te gebruiken, waarbij een patient volgens een bepaaldritme zich een taak voorstelt. Temporele modulatie leidt tot een betere signaal-ruisverhouding en daardoor tot een hogere succes-score om de EEG activiteit tedecoderen.

We zullen dit gegeven gebruiken, samen met de nieuwe meet- enanalysemethodes voor hersenactiviteit, om patiënten met stoornissen in staat te stellenom huishoudelijke apparaten te besturen (b.v. de TV) en zelfs om te communiceren(met tekstscherm of spraaksynthese). De patiënt stelt zich de taak voor als kortetemporele patronen (bijvoorbeeld volgens een modulatie met 1 Hz. of volgens eenmuzikaal ritme of Morse-achtige code). De hersenactiviteit gedurende deze mentalevoorstelling wordt gemeten met sensors (EEG) en geklassificeerd in een aantalcategorieën. Deze kunnen corresponderen met specifieke commando's, tekens ofwoorden.

In een pilotstudie hebben we reeds kunnen bewijzen dat deze aanpak inprincipe haalbaar is en tot een betere performance leidt. In tegenstelling tot het meesteBMI werk gebruikt onze benadering niet-invasieve meettechnieken. Verder steunt demethode niet op langdurige (bio-feedback) training van de gebruiker. In plaatsdaarvan zal het systeem zich aanpassen aan de gebruiker om EEG patronen teherkennen als representatie van response categorieën op basis van bekendevoorbeelden. Om dit doel te bereiken moeten complexe analyse en klassificatiemethodes (b.v. Independent Component Analysis ) worden aangewend om derelevante informatie te extraheren uit de ruizige EEG signalen.

5

3.4 Utilisatie

Er is een grote behoefte ontstaan om neuronale signalen van de hersenen metbetrekking tot waarnemen en bewegen te interpreteren en eventueel te gebruiken omde functie van beschadigde delen van de hersenen te kunnen herstellen en ompatiënten met ernstig hersenletsel (bijv. na een hersenbloeding of dwarslaesie)functionaliteit terug te geven. Dit zou deze patiënten in staat stellen om weerzelfstandig te kunnen bewegen en communiceren.

Er zijn in Nederland diverse bedrijven, die zich specialiseren op de analyse eninterpretatie van biomedische signalen. Biosemi en Brain Research Companybeschouwen het voorgestelde project als een logische volgende stap in aanvulling opde bestaande technieken voor neuro-engineering. Hun interesse komt duidelijk totuiting in de bereidheid aanzienlijk te investeren in het project.

Het Sony research center in parijs, met verschillende richtingen van onderzoekin muziek en taal werkt ook aan nieuwe data analyse technieken (software andhardware) voor de neurowetenschappen. Sony is geïnteresseerd in deze toepassingenvoor het besturen van verschillende apparaten en heeft recent een nieuweonderzoeksgroep opgezet in Tokio.

De St. Maartenskliniek is een van de grootste revalidatie centra in Nederlanden is geassocieerd met het University Medical Center Nijmegen. De St.Maartenskliniek wil investeren in nieuwe technieken voor revalidatie. Zij zijn metname geïnteresseerd in het gebruik van EEG signalen als methode om discrete actieste kunnen initiëren, zoals bijv. aan/uit zetten van een TV, selecteren van een kanaal,besturen van een gemotoriseerde rolstoel bij patiënten met ernstigeverlammingsverschijnselen.

4 Description of research group and expertiseDr. P. Desain combined his background in both computer science and psychology todevelop a multi-disciplinary approach to the computational modeling of rhythmperception and production. He is leader of the PIONIER project 'Music Mind,Machine'. In the final year of this project he has been concentrating on tracing theneural processes involved in perceiving and imagining rhythm in EEG signals. Nextto leading the MMM group (between 5 and 10 fte) was is involved in supervision of aPost-doc project on ERP and basic rhythm perception. This year he is elaborating theERP analyses methods as visiting researcher at Stanford University (CSLI andCCRMA). Projects and publications can be found at http://www.nici.kun.nl/mmm.

Prof. dr. C.C.A.M. Gielen is full-professor of Biophysics. His research group focuseson neuronal information processing using experimental and theoretical tools. Thebiophysics group is involved in neuroimaging studies at the recently established FCDonders Center, using EEG, MEG and fMRI. The group also houses three set-ups forsingle-unit recordings in primates and advanced equipment for visual, auditory andvestibular stimulation. The theoretical research focuses on modeling of biological andartificial neural networks. Prof. Gielen is director of the Dutch Foundation for NeuralNetworks (SNN) that initiates and coordinates research and applications of neuralnetworks in the Netherlands. More information about the expertise and researchactivities can be found on the web-sites http://www.mbfys.kun.nl/ andhttp://www.snn.kun.nl/nijmegen/index.php3.

6

5 Project durationThe project duration is 4 years

6 Description of the project

6.1 AimThe goal of this project is to apply recent signal processing and brain imagingtechniques for the development of on-line EEG classification methods, which enablepatients with impairments of the motor system (limb movements, but also speech) tocontrol devices and to communicate by imagining temporal patterns, such as(musical) rhythms or Morse code.

6.2 State of the art of Brain Machine Interfaces (BMI) resultsThe use of Brain Machine Interfaces (Donoghue, 2002), and more specifically the useof neuro-imaging signals for rehabilitation, is a very rapidly expanding field ofresearch. This type of research is still in a preliminary phase. It started withelectrophysiological research in monkeys. In these experiments multi-unit activity(activity of a large number of neurons) is recorded with chronically implantedelectrodes subdurally while the monkey is focusing on visual stimuli or is makingvarious types of movements. With the increasing body of knowledge about theinvolvement of cortex in perception and motor control, activity from cortex can beused as a measure for the percept or intended movement of the monkey (see e.g.Nicolelis, 2001).

BMI has been used so far only in experimental situations and mainly in monkeys touse signals from visual cortical areas and from motor cortical areas for reaching andgrasping objects. A leading group in this field is the group headed by RichardAndersen at CalTech, who started with studies in monkeys who focus on a videoscreen that shows a variety of visual stimuli. While fixating these stimuli, neuronalactivity is measured that provides information about how neuronal structures areinvolved in visual perception and how this information is used to initiate motoractivity. Experiments by this group using EEG in humans to use visual inducedneuronal activity for the initiation of goal-directed arm movements are in preparation.

Miguel Nicolelis (2001) presented a technique developed at Duke University, torecord neuronal signals from various cortical regions in monkeys during movements.These neuronal signals were used to control a robot arm to make the same movementsas the monkey while reaching for food. A German group (Mehring et al., 2003)recently reported that measuring local-field potentials in monkey motor cortex can beused to correctly predict the direction and velocity of arm movements to varioustargets in 90% of the cases. The next step in this line of research will be to use EEGsignals from motor cortex in human subjects to electrically stimulate muscles, suchthat patients with hemiplegia will be able to make movements by imagining thesemovements.

At Case-Western University (Cleveland Ohio) the group headed by Patrick Crago isworking on a project to measure brain activity to control a hand neuroprosthesis (theso-called Freehand) for patients with high-level spinal lesions. The aim of this project

7

is to use brain signals from patients to control a (robot) arm to provide the patientwith new motor capabilities.

The use of invasive techniques (e.g. electrodes for local field potentials or multi-unitactivity) cannot be used on a routine-basis in human subjects. Therefore, we have torely on non-invasive techniques, such as EEG. However, the main problem with EEGsignals, as opposed to subdural recordings of neuronal activity in monkeys, is the poorsignal-to-noise ratio. This problem has been addressed in the literature by averagingthe EEG signal in time. Although this procedure increases the signal-to-noise ratio,the gain is small and this procedure decreases the temporal resolution. Moreover,aberaging over repeated trials is very cumbersome for patients. Therefore, otheralternatives have to be explored.

The work of Birbaumer e.a. (1999) pursues the same aim as this project, but with adifferent method. Slowly varying scalp potentials and very extensive bio-feedbacktraining have been used to allow completely immobile or locked-in patients to controla cursor on a computer screen and thus to select characters to compose text or operatemenu's. Although successful to some extent, the training procedure often lasts manymonths and is extremely cumbersome and demanding for the patient.

At Graz University, the group of Pfurtscheller is one of the leading groups in Europe.They have extensive experience in recording and analyzing EEG signals with the aimto use these signal to restore functionality in patients who lost the ability to move theirlimbs (tetraplegia, see e.g. Pfurtscheller et al. (2003) and Muller et al. (2003) for anoverview of their work). This group has also a strong tradition on basic research,focussing on beta- and gamma synchronization of cortical EEG activity. Theseoscillations in the frequency range between 15 and 70 Hz provide information aboutattention for perception and action. The signal-to-noise ration in this frequency rangeis not high enough to use these signals for reliable on-line control. Pilot studies in ourgroup (see sections 7.1 and 7.2) show that the use of low-frequency temporalmodulation (range 1 to 8 Hz) provides a good alternative to extract relevantinformation.

The work of Suppes e.a. (1998) aims at directly identifying imagined words andsentences (internal speech) from EEG. The words and sentences in these study aretaken from sets around 10-100 in size. Although promising, it appeared to be easier toidentify perceived speech from EEG data than to identify imagined (or associated)words from EEG. It is still an open question whether the better performance for theperceived stimuli could be attributed to the temporal patterning of the sounds, andthus the rhythm, or whether it was due to the difference in perception versusimagining. One finding of this group has implications for the required adaptability ofany proposed system: there are large differences between the signals obtained fromdifferent subjects.

6.3 Our ApproachWe plan to exploit the fact that imagining temporal patterns (e.g. auditory rhythm)creates fluctuating characteristic signatures in brain activity that can be identified andextracted. These signals could be used as an on-line way of communication,especially if imagining simple patterns (e.g. Morse code or rhythmic trials in the1-2Hz frequency range) would be useful. Part of the construction of a robust analysis

8

method will be to extract understanding as to the source of the EEG signals and theirrelation to the imagined pattern. Combining source analysis and temporal modulationwill allow us to extract a wide range of intended actions from the EEG signal. Thisinformation will be used to assist in the development of a proper set of analysestechniques to allow for a reliable and fast identification.

The main innovative idea of this research project is to instruct subjects to imaginetemporal sequences, which, similar to FM-modulation in radio signals, will improvethe signal-to-noise ratio, one of the largest problems in BMI research using EEGsignals. In contrast to most approaches, which output continuous data, our system willclassify the data and the output will consist of a sequence of discrete symbols from afixed set. We will use advanced techniques to make the system adaptive to recognizethe signals from the noisy data, rather than attempting to train the subject to generatebrain signals that can be picked up. Furthermore, the measurements can be done in anon-invasive way (EEG).

6.4 Problems and SolutionsNoise. Single trial EEG data is very noisy, with data stemming from many sources.The characteristic responses to specific events are usually obtained by averagingsignals from many trials, like in evoked potentials. To successfully match single trialdata the relevant source of the signal needs to be separated out, before it can bematched to average templates.

In this project spatio-temporal templates will be used. The spatial component of thetemplates is related to a proper weighting of the EEG signals from the electrodes onthe skull. As it turns out, Independent Component Analysis (Hyvärinen, e.a. 2001;Ypma, e.a. 2001) can extract relevant sources of information from such spatiallyseparated noisy signals without any prior knowledge about the spatio-temporal sourceof these signals. Applying the resulting 'un-mix' matrix to both single trial data andtemplates will allow us to separate any noise signals (e.g. due to blinks, muscleactivation) and to extract the neuronal signals. This will greatly improve theclassification process.

Spectrum. In matching, a similarity measure is applied to compare the single trialwith each template. This measure is still easily distorted by signals from non-relevantparts of the frequency spectrum.

The temporal aspect of the templates relates to the various frequency components inthe EEG recordings, which reflect temporal aspects of the rhythm and that of theneuronal activity. Using the similarity measure on a full spectral analysis (e.g.wavelet) helps successful matching. In addition to evoked potential templates, there isevidence that induced potentials (oscillations that are time-locked, but not phase-locked to the events) in the higher frequency bands (theta-rhythm) contain muchpattern-linked information that can be exploited. By ignoring the phase information ofthe spectral analysis, meaningful templates can be constructed for this kind ofinformation as well and can be used in the matching process.

Synchronization. When subjects have to generate a temporal pattern, subjects willreveal variations in the speed of the temporal pattern, which makes it hard to matchthe signal to a temporal template. This problem can be solved by dynamic time-

9

warping, which is a well-known technique, which has been successfully used in theDept. of Medical Physics and Biophysics for on-line quantization of temporal patternsfor automatic music transcription. Time-warping techniques, as used in speechrecognition and automatic music transcription, may be applied to solve this problem.Since the data-analysis procedure is rather complex, we propose to start the projectwith the restriction to a domain in which time locking is easily enforced, e.g. by anexternal metronome, before we make the next step towards dynamic time-warping.

Overfitting. After matching, a vector of similarity measurements for the differentchannels and the different templates needs to be classified into a category judgment.Optimal methods need to be found that do not over-fit the data. The latter could resultin a categorization system, that works well on the training set, but fails on new trials.

In a pilot study (Desain & Honing, 2003) a rigorous “leave-n-out” separation in testand training data was used, along with common statistical methods like logisticregression and discriminant analysis, with promising results. Minimization of over-fitting is the same as maximizing generalization, which is a well-known problem inmany classification techniques (like neural networks.) The Nijmegen group has alarge expertise in methods to avoid over-fitting, and the vectors of similarity, thatwere used in the pilot study, will be used as a first step to improve the performance.

Confidence. A rating of confidence of the judgment is much needed, as outputting awrong symbol may have a high cost in some situations (like in wheelchair control).This aspect is lacking in all procedures that have been proposed in the literature sofar. Yet, it is crucial for a good performance: the system should only work in regionsof high confidence.Several classification methods exist, that can output such a confidence measure. It canbe used in conjunction with a threshold to prevent unreliable output (“falsepositives”), or trigger a retry. The robustness of such an approach has already beendemonstrated by us (see figure 1).

Measurement. EEG electrodes may bring many practical problems, like sensitivity toelectromagnetic radiation, difficulty to place and position, varying conductance,usually a limited number of channels, and discomfort when used for a longertime.Within the FC Donders Center, we have ample experience with these aspects ofmeasurement. Moreover, we have BioSemi in our “gebruikerscie”. With theintroduction of the ActiveOne system by BioSemi (with support from STW) two newconcepts were introduced in the design of EEG measurement systems:- miniaturized, battery-powered front-end close to the patient, with fiber optic datatransfer to the signal processing PC- use of active electrodes, which have the property that the first amplifier stage isintegrated within the electrode.The product was further improved with the ActiveTwo system (introduced in 2002),which featured higher system integration (256 channels) and higher resolution (24-bitconversion). The principles used in these systems allow the measurement of low-noise EEGs without the need for time-consuming head preparation, and in situationswhere interference sources are nearby the subject. This has simplified the recording ofEEGs with a high number of channels and reduced the discomfort for the subject.

10

6.5 FeasibilityThough much research has been conducted on visual imagery (e,g. Kosslyn, e.a.2001), the process of mentally performing rhythmic patterns has not yet beenaddressed often. In our studies (Desain & Honing, 2003) we have been able toachieve a 50% correct single trial identification rate for a perceived musical rhythmfrom a set of 5. Perfect identification proved possible when the method was allowedto disregard (“don't know”) half of the cases, when the confidence level ofidentification was not sufficiently high.

Figure 1. Classification results (for perception), when a confidence threshold allowsinputs to be rejected.

Identification of imagined rhythms is harder, but the method's success proved highlysignificant, the imagery of the simplest pattern in the set was identified correctly inabout 35% of the cases.

6.6 Experience in diagnostics and automatic medical diagnostics

The Biophysics group has recently developed a tool for automatic detection andclassification for Parkinson’s Disease (see Keijsers et al., 2003a,b,c). The problem isthat adjustment of the proper Levodopa medication in Parkinson patients is hard todetermine after several years of medication. After a period of 5 to 10 years, too largea dose gives rise to dyskinesia and too small a dose leaves the classical Parkinsonsymptoms. Since the effects depend on the time of the day, on attention and mentalload, a proper adjustment of levodopa dose requires 24-hours monitoring for about aweek. Therefore, it is very difficult for a neurologist to find the proper medicationdose. Using advanced data analysis and accelerometers, we have successfullydeveloped a system to classify movement disorders.

7 Research Issues

7.1 NeuroscienceThe question of what kind of signal is picked up, it’s source, and it’s relationship tothe imagined pattern, will need to be investigated. David, Garnero, Cosmelli andVarela (2002) have developed a method, using MEG and EEG, to estimate neuraldynamics of cortical sources. Their use of this new approach on non-averaged data

11

with a poor signal-to-noise ratio has resulted in source localization results weaklysensitive to noise. This offers a promising method to determine the localization of thesignals recorded in response to mental imagery, which will give important insightsinto the nature of the neural signal. These new insights can be used both to assist inthe choice of the patterns to imagine and in the design of the most successful analysismethod.

Previous studies have used an EEG activity averaged over several or tenths of secondsto improve the signal-to-noise ratio. In this approach, we will focus on temporallymodulated neuronal signals, which allow template matching in order to improve thesignal-to-noise ratio. This approach also offers the advantage of a better temporalresolution of signals to be picked up. In this context, we have to choose the temporalstructures that are most reliably picked up. One option would be pure musical rhythm,with its internal structure and expectancies (Cooper & Meyer, 1960; Desain, 1992).The second could be sequential but not rhythmically structured expectancy generatingstructure, like isochronous musical melody. This we are currently undertaking at theStanford CSLI lab, with the stimuli based on a model of melodic expectancy(Margulis, 2003). The third type of patterns are temporal patterns like Morse code.This project will conduct the critical experiments needed to choose the best domain,connecting to areas of music cognition and auditory perception research, that havebeen conducted outside the area of brain mapping (Longuet-Higgins, 1976).

In general, signals resulting from pure mental imagery are harder to detect in the EEGthan signals in response to the perception of real stimulation. The use of mentalimagery has the capacity to offer a larger number of potential patterns. However, ifthe detection and identification of the resulting signals is too difficult, it would bepossible to present a few auditory stimuli simultaneously with the instruction to thesubject to pay attention to the stimulus which is associated to the desired task.Contradictory as it may seem, 'control by listening' is a feasible method as signals inresponse to perception are modulated to a great extent by (selective) attention. Thisoffers the possibility of developing an easier task by simultaneously providingdifferent auditory stimuli. Which of the streams is being attended to can bedistinguished from the EEG signals and the patients choice of which stimulus toattend to can be assigned a meaning (see Miranda, 2003). Working with dynamicallymodulated attention, as in 'subjective rhythmization' of an isochronous sequence, thesubject mentally grouping the sounds in 2 or 3, an experiment was designed and datais currently being collected by Kathleen Jenks at NICI. This will show us how muchmore reliable temporally modulated attention can be detected from ERP as opposed topure imagery which lacks a firm time locking.

A robust method, but still needing repeated measurements, is the detection of P300components in EEG. It has long been known that violated expectancy can bemeasured in the case that a stimulus is not expected but nevertheless does occur, inthe form of high P300 amplitude (Duncan-Johnson & Donchin, 1977). It is alsopossible to measure violated and confirmed expectancies in response to non-occurringstimuli (Jenks e.a. 2002), as a high amplitude omission evoked potential (OEPs) iselicited. Although these results were obtained using averaging of a few EEG signals,this methods can be used for practical purposes for single trial data by using amatched filter, since we know where to look for (absence or presence of P300). Thisis the method used in the visual domain with flashing columns and rows of letters

12

(Donchin ea. 2000).

Another issue is the size of the alphabet of symbols that is recognized. Although thismay be restricted by the analysis method, the availability of a few choices alreadyallows navigation through complex menu structures or pre-structured dialogs. The lastidea, structured multiple-choice conversations, has be patented as PCT/NL00/00425by Desain and may be the best solution to support a sense of full interaction using alimited control device.

7.2 Signal processingThe second issue focuses on the treatment of the signal, regardless of it neurologicalsource. In a pilot study (Desain & Honing, 2003) the setup of Figure 2 was used forthe classification procedure. From this study it became clear that application of theunmix matrix resulting from blind source separation methods (ICA) (van der Veen,1998; Cardoso, 1999), a linear transformation, is beneficial for the automaticclassification.

Figure 2. Signal processing in matching new single trials against 5 ERP templates.

However, several questions regarding advanced methods for signal processing (SP)and machine learning (ML) have to be answered. How can the success of ICA inconcentrating information be expressed/optimized? How can we combine ICA with

13

spectral separation and induced potentials? How can we exploit the temporal structureof the patterns?

Recently, extensions of ICA have been proposed in the literature, which, to someextent, deal with these issues. Some of the research has been done fortelecommunication signals but is applicable to EEG data as well. Prior knowledgeabout the sources can be incorporated in a general manner by exploiting the second-order statistics of the sources (Ypma, Leshem, & Duin, 2002) or by modeling thesource separation problem as a generative probabilistic model (e.g. Attias, 1999).Algorithms for convolutive demixing are based on cancellation of higher-order cross-moments (Nguyen Thi & Jutten, 1995 ), blind separation in the frequency domain(Mejuto & Principe, 1999), or using a proper (more complex) generative probabilisticmodel (Attias & Schreiner, 1998). Our aim is to compare the separation quality ofstandard (linear instantaneous) ICA methods with one or two instances of theadvanced ICA methods mentioned above and to decide which extensions areultimately necessary for adequate classification.

An important issue of both practical and conceptual importance is that the 'imaginedpattern detection' should be adaptable to changing user preferences, novelenvironments, and even to changing neural responses of the user. It is known (Kilgardand Merzenich, 1998; Xerri et al., 1998; Buonomano and Merzenich, 1998) thatneural response patterns change after a prolonged learning phase. For example,massive reorganization takes place in motor cortex after a playing the piano for halfan hour by unexperienced subjects. Similar processes take place after training ofdyslexic patients. Other aspects of adaptability are related to the fact that, the usermay behave differently in a real-world setting compared to the initial (probablylaboratory-constrained) calibration phase, or the subject may change his preferentialimagined temporal 'control patterns' somewhat. The measurement of novel (stillunlabeled) patterns can lead to classifiers with improved generalization performance(Bennet, Demiriz, & Maclin, 2002). We expect to develop novel statistical criteria fordeciding when to update an existing Maximum Likelihood module ("when is a set ofincoming measurements consistently different from everything we have seen before,excluding noise?"). We expect that extending a static classifier to a nonstationarymodel (e.g. the Kalman filter implementation of a support vector classifier indeFreitas e.a. (1999) or the adaptive clustering strategy described in Guedala, et al(1999) is a good way to start this research, but it may prove necessary to developother novel adaptive machine learning methods for this purpose. Many seeminglysubjective issues like setting of thresholds, choice of model structure, and the 'optimal'(best generalizing) model parameters and finding a proper trade-off between priorknowledge and measured data can be approached in a more objective manner bytaking a Bayesian approach. In this sense, the Bayesian machinery is suitable to (a)find a proper initial model using calibration data, which is then (b) traded off withnovel (labeled and unlabeled) measurements upon utilization. We remark that theadaptive ML problem is as yet largely unresolved. However, the SNN/Biophysicsdepartment has considerable expertise in Bayesian methods for modeling non-stationary processes and in modeling with generative probabilistic models. Thisproblem, which is of considerable theoretical interest by itself, is an appropriatechallenge.

14

7.3 Patients

As the most extreme, and demanding test of our method we will test it on so-called“locked-in” patients, in patients with paralysis due to strokes, and in tetraplegicpatients. “Locked-in” patients are patients who lost all abilities to make movements(often also eye-blinks) due to a conduction block of the efferent nerve fibres (e.g. dueto ALS). The number of such patients is not large, but there are some of those patientsin the St. Maartenskliniek. For these patients the benefit would be enormous, as theuse of neuronal signals to interact with the environment is almost the only way forcommunication and control.Other groups of patients (stroke patients and tetraplegic patients) will be selected. Wewill focus on one of these groups of patients. We expect that it will take about 2 yearsbefore we will be able to apply the algorithms real-time on patients. The finaldecision, which group to start with, will be made by then. At this stage we propose towork with stroke patients, since there are two important reasons to work with strokepatients in an early stage after the stroke: our methods will help to provide somefunctionality to these patients and regaining the ability to interact with theenvironment will greatly assist the natural recovery in these patients (results from thegroup by Thomas Sinkjaer in Aalborg, Denmark). A disadvantage to test the systemon stroke patients might be that the stroke also effects the EEG signals. Based onprevious reports we expect that the remaining EEG activity will be sufficient and thatthe signal-to-noise ration will be even better in these patients. However, the ability tomodulate EEG activity by attention seems to be lost. In summary: the technology thatwill be developed will enhance the already existing tools to assist patients with thelocked-in syndrome and tetraplegic patients (see the work by Birbaumer andPfurtscheller). Exploration of the usefulness of the method for various other groupsof patients is part of the project.

The St. Maartenskliniek recently obtained permission to use electrostimulation of legmuscles in hemiplegic patients to use a specially designed bike. Since the EEGrecordings are non-invasive and without risk for the patient, we do not foresee anymajor problems obtaining the formal ethical approval from the appropriate committeefor these experiments.

We would like to emphasize that the main effort in this proposal will be focused onmeasurement, analysis, and interpretation of the EEG data. Using these signals fordaily practice would require a next step: interfacing of the results with hardware(wheel chair, prosthesis, etc.). This would be another in itself since it requires thedevelopment of various tools, which will need approval by governmental agencies. Inthis proposal we will rely on simple point and click interfaces, thereby makingaccessible and building the common methods of Internet technology.

8 Proposed budget

8.1 PersonnelFor this project to become successful expertise from two different domains is requiredto successfully design a method and a mental task that can reliably be used for controland communication. First of all, the inherent noisy character of EEG signals requiresadvanced signal analysis tools to measure the EEG signals and to classify these EEG

15

signals for control and communication. Second, knowledge is required aboutcognitive modulation of brain activity in order to interpret EEG signals as traces ofmental activity. Therefore, either two post-docs (3 years) or one post-doc (3 years)plus one PhD student (4 years) with complementary backgrounds from signalprocessing and from cognitive neuroscience will be needed to conduct the research. Apart-time technical assistant will assist in programming, the online processing of thedata, constructing the stimuli, and their presentation for the experiments, andprogramming the control of the output devices. A rehabilitation physician will helpand supervise the testing of the system with patients, some of whom are severelyhandicapped. Furthermore, this kind of expertise is needed for the evaluation of thesystem in practical use, and to be able to design the adaptations for the individualneeds of patients.There are already experienced candidates available for some of these positions.

Post-doctoral Fellow, signal processing, 3 years * 1 ftePost-doctoral Fellow or PhD student, neuroscience, 3 or 4 years * 1 fteProgrammer/technical assistant, 4 years * 0.5 fteRehabilitation physician, 3 years * 0.1 fte

8.2 EquipmentThe requirements for the EEG measurement system are quite high. It needs to beportable and to function in a clinical environment, outside a shielded cabin and itneeds to support a large number of electrodes to provide a high scalp sensor density.Furthermore, it needs to be inherently safe, to fit the patient comfortably and it shouldwork reliably for a long time. It needs to support on-line processing of the data. Noneof the available EEG equipment, neither at NICI/KUN nor at the F.C. Donders centercan meet these requirements. Therefore, we propose to buy the Biosemi equipment,that does meet all desired requirements. This inherently safe and reliable active-electrode solution (developed with STW support) is less than half the price of systemswith passive electrodes offered by other companies.

The recording, storage, and on-line analysis of EEG data put great demands on acomputer system. An inadequate system can result in slow analysis time andunnecessary project delays. We, therefore, have to opt for large and fast storage andfast computers for analysis.

9 Time planning and assignments of tasksIn the first stage of the project we will focus on the development of the basic softwareto analyse the EEG data and on the collection of a representative set of EEG data totest the software tools and algorithms. In the second stage of the project, we will testthe software tools and algorithms on data to be collected from a representativenumber of patients.Control and communication will be realized in small steps. First we will explore onebit messages (Yes/No, alarm), then we will extend the approach to a few (e.g. forcursor control), then to a full usable alphabet (like Morse code).The tasks, that can be attributed mainly to one or two persons, are indicated: theNeuroscience PhD or Postdoc (N), Signal processing postdoc (S), and programmer(P).

16

Year 1.• Definition and implementation of software analysis modules and interfaces (S)• Definition and implementation of independent performance measures of analysis

modules (S)• Conceptualization and choice of mental tasks (imagery, attention control) and

domain (rhythm, temporal codes, pitches etc.) (N)• Definition and implementation of independent performance measures for the

tasks, defined in the previous item (S)• Technical realization of interfacing to/from measurement equipment (P)• Start of first experiments to obtain a representative data set for analysis (N)

Deliverable 1: off-line demo: replication/improvement of results of pilot, healthysubjects, reliable classification (1 bit).

Year 2.• Completion of representative data set for a small number (about 3) of patients,

who will also have fMRI and MEG scans to localize the site of the neurologicaldisorders (localization). This data-set serves as a test for the software. (N)

• Design, implementation, and test of preprocessing methods (ICA) (S)• Design of similarity measurements (induced, spectral) (S)• Design, implementation of classification methods (S)• Choice of output device and it’s implementation in the set-up. (N,P)• Interfacing and programming output device (P)• IP issues

Deliverable 2: on-line demo: proof of principle, healthy subjects, more complex andfaster control.

Year 3• Technical adaptations for in-clinic use (P)• Measurements with patients, which includes on-line testing and performance in

the tasks defined in year two (N)• Analysis and design of adaptation methods and on-line system training (S)

Deliverable 3: demo with one patient, proof of usability in daily life.

Year 4• Exploration of other output devices, control of structured conversation, evaluation

of limitations in performance (e.g. which tasks can/cannot be done; suitability forvarious types of patients) (P,N)

• Exploration of the usefulness and comparison of the performance for three patientgroups: patients with the locked-in syndrome, stroke patients and tetraplegicpatients.

• Study of improvement of performance of system under bio-feedback (N)• Transfer of knowledge and software tools to commercial partners• Exploration of commercialization

Deliverable 4: on-line demo with a few patients with various types of neurologicaldisorders. Various output devices. Evaluation/usability results.

17

10 Scientific relevanceThe scientific relevance can be summarized by the following items:

1. The spatio-temporal mapping of EEG activity on mental processes willprovide new insights into the structure of the signal for the different tasks, andon the role and functionality of human cortical areas.

2. The study will provide new insights in the ability of adaptation by the humanbrain. Studies on adaptivity have become possible by the availability of newneuro-imaging techniques and on-line processing. Insight into the ability ofadaptation will be highly relevant for understanding the neuronal mechanismswhich underlie adaptation, but will also be important for further use in patientswith physical or neuronal handicaps.

3. The development of adaptive techniques to detect temporal patterns in noisysignals will be another contribution of the project. This is a rapidly developingfield, which is making fast progress due to new developments from variousresearch areas: neural networks, machine learning, artificial intelligence andsignal analysis.

4. Though driving towards robust control, the methods developed will be alsouseful as tools in the scientific study of neuro-cognition, where the excessiveamount of noise in the signal still hampers much experimental work.

11 Collaboration

11.1 International collaborationThe work will be done in close contact with Prof. Suppes (Stanford, CSLI), whoworks on similar problems in language (internal speech), and Prof S. Makeig, (SCCN,UCSD), an expert in ICA methods for analysis of brain signals. Dr. Desain has visitedthese colleagues and laboratories and works at Stanford as visiting researched during2003-2004. There are many contacts with the 'neurosciences of music' researchcommunity that can provide valuable insights in the mental coding of rhythmicpatterns.One of the leading groups on Functional Electrical Stimulation (FES) using neuronalsignals (like EEG and EMG) is the group headed by Thomas Sinkjaer (AalborgUniversity in Denmark) and the research group by Peter Veltink (University Twente).The Biophysics group in Nijmegen has collaborated with both groups in the past andif the proposal will be granted, we will contact these groups for further collaboration.Gielen also spent several periods (range from 8 months in 1993, 4 months in 1995 andone month in 2002) in the Chicago Rehabilitation Institute of NorthwesternUniversity where neuro-imaging signals are used in handicapped patients (amputatedlimbs, paralysis of limbs) to control prosthesis movements. We have a longstandingcollaboration with this group (dr. Lee Miller, prof. Dr. J.C. Houk, and prof. Dr. Z.Rymer), which will contribute to the success of this project.Prof. Gielen is also a speaker at the yearly meetings of the Summerschool onNeuroEngineering (2002 in Genova; 2003 in Venice) where scientists from variousdisciplines meet to discuss recent results and to train excellent Ph.D. students.

18

11.2 National collaborationOn a national level, SMK-Research works together with the Universities ofAmsterdam, Groningen and Utrecht and with the Roessingh Research Center inEnschede. On the international level, an EU concerted action (Eurokinesis project)exists with Balgrist (Prof. Dr. Dietz, Zurich) and KUL (Prof Dr Swinnen, Leuven).The recently founded F.C. Donders Center (prof. Dr. Hagoort) in Nijmegen providesan excellent background of expertise in brain imaging and measurement facilities.This center has many international affiliations (for example with Dusseldorf (Zilles),London (Frackowiak), and Leipzig (Max Planck Institute). Between these institutesthe most advanced expertise and computer programs for recording and analysis ofneuro-imaging signals is exchanged in order to have the best available tools forresearch in this rapidly expanding field.As mentioned above, the Biophysics group has good contacts with the research groupat the university of Twente (prof. Dr. veltink), who studies the use of functionalelectrostimulation in the rehabilitation of physically handicapped patients.

12 Utilization

12.1 Summary EnglishThe outcome of the project, relevant for utilization, is a reliable prototype of systemto control and communicate by means of rhythmic EEG patterns. This direct interfacebetween brain and machine has many applications, like interactions with theenvironment, multi-modal work environments, etc. We will focus in this project onone application, a robust system that allows patients with severe paralyses to interactwith the environment. Interactions could imply communication with other people (forpatients who lost the ability to speak), control of a robot prosthesis (for patients withparalysis of the arms), or simple communication with the environment for so-called“locked-in” patients. Although we are not in a stage yet where a directcommercialization is to be considered, we have assembled a group of companies andan institute as partners with keen interest in the use of this technology. It is in theirinterest to help in realizing the goal of a good utilization of the outcome of the project.

12.2 Summary DutchHet resultaat van dit project, gezien als product, is een betrouwbaar systeem voorbesturing en communicatie door middel van ritmische EEG patronen. Voor dezedirecte interface tussen hersenen en machine zijn veel toepassingen voorzien, zoalsinteractie met de omgeving, multi-modale werkomgevingen en interfaces voorcomputerspellen. In dit project zullen we ons richten op één toepassing, eenbetrouwbaar systeem dat patiënten met ernstige verlammingen in staat te stellen tecommuniceren met andere personen (voor patienten, die het vermogen van spraakverloren hebben) , robot prothesen te besturen (voor patienten met verlammingen aande armen) en voor eenvoduige communicatie met de ongeving bij zogenaamde“locked-in” patienten.In dit project participeert een groep van bedrijven en een instituut met elk een groteinteresse voor het gebruik van deze technologie. Het is in hun belang te helpen ook dedoelstelling van een goede utilisatie van de resultaten van het project te doen slagen.

19

12.3 DeliverablesThrough the years we plan to deliver partial demonstrations of the method (seesection 9). The most extensive final deliverable will be an online demonstration of thesystem, proving the feasibility of direct neuronal-control and communication anddemonstrating its robustness in everyday use by patients. The demonstration willinclude• on-line control by at least 4 patients (presumably stroke patients with severe

paralyses)

This prototype should be a sound basis for commercialization by a company.Depending on the success of the method developed, it may range from a device tosimply sound an alarm buzzer by mind-control, to a speech synthesizer controlled bythought Morse-code. In between are applications like the remote control for AVequipment, a 'thought-mouse' to control a www browser or any computer application,and the control of communications tools like icon-boards or pre-structuredconversation item selection.

12.4 DisseminationIn addition to publications in the leading international scientific journals in thevarious scientific disciplines involved and the presentation of the results ininternational meetings, we plan to organize an international seminar on mind-control,which brings together scientists and potential users.

Our partners are a rehabilitation institute (the St. Maartenskliniek), which hasmany patients that could greatly benefit from the results of this project, and severalcompanies who develop specific software and hardware to measure and analyse EEGsignals. Their interest is to strengthen their leading position in the field by developingnew advanced tools for the use of EEG in medical applications. For a more extendeddescription of the relevance of the research proposal for these partners, see section12.5 which provides an overview of the Utilization Committee members.

Because the appealing character of the proposed work it will not be difficult toattract media attention. We hope that the NWO public relations dept. will help inorganizing this.

12.5 User committeeExperts from very different backgrounds will participate in the user committee,helping to steer the research in its various aspects: its usability in future consumerproducts, the technical realization of robust semi-permanent EEG measurements, thecommercial exploitation of databases and standardized analysis methods for braindata, and addressing the needs of patients in the proper way allowing a smoothintegration in care-taking practices. The following institutes and companies havecommitted themselves to participate in a user committee. Together they willcontribute 35,200 Euro to the project.

Sony Computer Science Laboratory Paris (A. Tanaka)6, rue AmyotF-75005 Paris FRANCEtel: +33-1-44.08.05.12email: tanaka@csl.sony.fr

20

Sony is a large company with pioneering consumer products in media, games androbotics. Sony Computer Science Laboratories, Inc. (Sony CSL) was founded inFebruary 1988 as an independent corporate entity, for the sole purpose of conductingresearch in computer science. Its objective is to contribute extensively to social andindustrial development through original research that looks ahead and has thepotential to achieve breakthroughs in computer technology. In 1996, Sony CSL Pariswas founded and joined CSL Tokyo as a European laboratory committed tofundamental research. This department specializes in natural language processing,music and recently also neuroscience. It is CSL policy to make public the results ofthe research.Sony CSL will support the project by providing the work and expertise of A. Tanaka(4 years, 0.05 fte). This amounts to a contribution to the project of 10KEuro.

Biosemi (Dr. C. Metting van Rijn)Oude Looiersstraat 181016 VJ AmsterdamNetherlandstel: +31 650 626354email: metting@biosemi.com

BioSemi biomedical instrumentation is the initiative of electronic engineers RobertHonsbeek and Ton Kuiper and physicist Coen Metting van Rijn. After working formore than a decade in the field of research into biomedical instrumentation at theMedical Physics department of the University of Amsterdam, they decided to starttheir own company.During the past decade, this group has analysed the fundamental problems in modernmultichannel biopotential measurements within the framework of academic researchprojects sponsored by the Technology Foundation STW (projects AGN1667,AGN3416, AGN4098).Based on the results of these research projects BioSemi offers a range of state-of-the-art equipment for the most demanding biopotential measurements. The underlyingprinciples were described in various scientific publications.

Biosemi will contribute 10,000 Euro to buy a portable EEG recording system.

The Brain Resource Company (Drs. Martijn Arns)Toernooiveld 100Postbus 310706503 CB Nijmegentel: 024-3528878/: 06-48177919email: marns@qeeg.nl

The Brain Resource Company (BRC) is a biotechnology company focussed onbringing advances in human brain science from the laboratory into BRC clinics. Itsglobal consortium of scientists has developed computerized methods of analyzing andmodeling the brain. These software tools are complemented by a large qualitycontrolled database of normative subjects and subjects with a range of clinicaldisorders. The database provides a reference of normative variability and patterns ofclinical instability in the brain. Measures of the brain are collected in a rigidly defined

21

manner by a world-wide network of accredited clinics and transmitted over theinternet to BRC’s Central Analysis Facility for processing (including the profile ofeach client's brain function referenced to the database). Fee-for-service analysisreports are generated and returned over the internet to the referring BRC clinic.Activation tasks and psychological tests provide a profile of the brain’s majornetworks and functions. The brain profile consists of “When”(electroencephalography and event related potential electrical activity in response toactivation tasks),“Where” (functional MRI in response to activation tasks and MRI),and “How” (a battery of psychological tests) processes. The ethos of BRC is that allanalysis is transparent and published in international refereed publications.

The Brain Resource Company will contribute access to their commercial database fora possible large scale validation of our analysis methods.

Maartenskliniek (prof. Dr. J. Duysens)Berg en DalsewegNijmegentel: 024- 36 59 140email: J.Duysens@smk-research.nl

The Sint Maartenskliniek is a major hospital in Nijmegen for rehabilitation ofdisorders of posture and movement, as well as for behavioral dysfunctions as a resultof brain damage. The clinic has a large center for orthopedics, a center forrheumatology, and a major rehabilitation center. The Sint Maartenskliniek is wellaware of the fact that advanced medical practice is only possible when it is based onscientific research. The main purpose of Sint Maartenskliniek-Research bv (SMK-Research) is to translate modern scientific insights concerning motor disorders, re-learning, rehabilitation, orthopedics, movement science, biomechanics, engineeringand (neuro)psychology into applied clinical knowledge. The research departmentplays a crucial role in the stimulation of patient-oriented clinical research and worksclosely together with all three medical centers of the hospital. The staff consists of amulti-disciplinary group of scientists trained in movement science, neuropsychology,experimental psychology, clinical psychology, biomechanics, medical engineering,biology, physics, reumatology, neurology, rehabilitation medicine, anaesthetics,physical therapy, occupational therapy and epidemiology. About 15 researchers areworking in the department.Although SMK-Research is part of the Sint Maartenskliniek, it is firmly rooted in thescientific community of the Nijmegen University. There is not only a closerelationship with the NICI and with the Neurological Institute, but also intensive andfruitful relationships exist with several departments of the University of Nijmegen.Prof. Dr. Duysens, head of SMK-research, has a joint appointment at the university atthe Dept. of Biophysics.

The contribution of the St. Maartenskliniek will be0.1 fte for a period of 4 years: supervision by prof. Dr. J. Duysens, Head Research

Dept. of St. Maartenskliniek.0.1 fte clinical specialist for a period of 4 years: 15200 Euro

Other potential users might be the Roessingh Research and Development (prof. Dr.Hermens) and TMSi (ir. Peuschen). The group of prof. Gielen has good contacts with

22

the research group in Twente (Roessingh). Since we expect that we need about 2years to develop the algorithms and software implementation, we will contact thesegroups when the preliminary work has been done and when recording and analyzingEEG signals from various types of patients becomes feasible.

12.6 Protection of intellectual propertyAn application for a patent on the method has been submitted by Desain (EPO-02076635.8) after searches in the patent literature demonstrated the innovativecharacter of this invention. This application is still open for amendments.The structured dialog invention (PCT/NL00/00425) may form the solution that allowsthe proposal to support textual conversation through very limited channels.

13. Recent publication by the applicantsOutput by Desain since 1998 in international, peer-reviewed journals:

Desain, P. & Honing, H. (2003) The formation of rhythmic categories and metric priming.Perception.

Repp, B. H., Windsor, W. L., & Desain, P. (2002) Effects of tempo on the timing of simplemusical rhythms. Music Perception. Vol. 19, No. 4, 563-591.Cemgil, T., Kappen, B., Desain, P., and Honing, H. (2001) On tempo tracking: TempogramRepresentation and Kalman filtering. Journal of New Music Research. 29 (4), 259-273.

Honing, H. (2001) From time to time: The representation of timing and tempo. ComputerMusic Journal, 35(3), 50-61.

Windsor, W. L., Aarts, R., Desain, P., Heijink, H., and Timmers, R. (2001) The timing ofgrace notes in skilled musical performance at different tempi: a preliminary case study.Psychology of Music, 29, 149-169.

Cemgil, T., Desain, P., and Kappen, B. (2000) Rhythm Quantization for Transcription.Computer Music Journal, 24(2), 60-76.

Cemgil, T., Kappen, B., Desain, P., and Honing, H. (2000) On tempo tracking: TempogramRepresentation and Kalman filtering. In Proceedings of the International Computer MusicConference, 352-355. San Francisco: ICMA. [Received distinguished paper award]

Heijink, H., Desain, P., Honing, H., and Windsor, W. L. (2000) Make Me a Match: AnEvaluation of Different Approaches to Score-Performance Matching. Computer MusicJournal 43-56.

Heijink, H., Windsor, W. L., and Desain, P. (2000) Data processing in music performanceresearch: using structural information to improve score-performance matching. BehaviorResearch Methods, Instruments and Computers 546-554.

Timmers, R., Ashley, R, Desain, P, and Heijink, H. (2000) The influence of musical contexton tempo rubato. Journal of New Music Research 131-158.

Desain, P. (1999) Vibrato and portamento, hypotheses and tests. Acustica 348. ISSN: 1436-7947.

Desain, P. and Honing, H. (1999) Computational Models of Beat Induction: The Rule-BasedApproach. Journal of New Music Research, 28(1), 29-42.

23

Desain, P., Honing, H., Thienen, H. van, and Windsor, W. L. (1998) Computational Modelingof Music Cognition: Problem or Solution ? Music Perception 151-166. ISSN: 0730-7829.

Output by Gielen since 1998 in international peer-reviewed journals:

B.M. van Bolhuis, C.C.A.M. Gielen, G.J. van Ingen SchenauActivation patterns of mono- and bi-articular arm muscles as a function of force and movementdirection of the wrist. J. of Physiology, 508, 313-324, 1998.

C.C.A.M. Gielen, B.M. van Bolhuis, E. VrijenhoekOn the number of degrees of freedom in biological limbs.In: Progress in Motor Control: Bernstein’s Traditions in Movement Studies.M.L. Latash (Ed.), Human Kinetics, Champaign, USA. 1998. pp. 173-190.

W.P. Medendorp, B.J.M. Melis, C.C.A.M. Gielen, J.A.M. van GisbergenOff-centric rotation axes in natural head movements: implications for vestibular reafference andkinematic redundancy. J. Neurophysiol. 79, 2025-2039, 1998.

M.D. Klein Breteler, R.G.J. Meulenbroek, S.C.A.M. GielenGeometric features of workspace and joint-space paths of 3D reaching movements.Acta Psychologica 100, 37-53, 1998.

C.C.A.M. Gielen, B.M. van BolhuisTask-dependent reduction of the number of degrees of freedom in sensori-motor tasks.Brain Research Reviews, 28, 136-142, 1998.

W.P. Medendorp, S. van Asselt, C.C.A.M. GielenPointing to remembered visual targets after active one-step self-displacements within reachingspace. Exp. Brain Res., 125: 50-60, 1999.

S. GielenWhat does EMG tell us about muscle function ? Motor Control, 3, 9-11, 1999.

Van der Laar, P., Heskes T., Gielen S.Partial retraining: a new approach to input relevance determination.International Journal of Neural Systems 9, 75-85, 1999.

W.P. Medendorp, B.J. Bakker, J.A.M. van Gisbergen, C.C.A.M. GielenHuman gaze stabilization for voluntary off-centric head rotations.Proceedings National Academy of Sciences, 871: 426-429, 1999.

B.M. van Bolhuis, C.C.A.M. GielenA comparison of models explaining muscle activation patterns for isometric contractions.Biological Cybernetics, 81, 249-262, 1999.

W.P. Medendorp, J.A.M. van Gisbergen, M.W.I.M. Horstink, C.C.A.M. GielenDonders’ law in torticollis. Journal of Neurophysiology, 82, 2833-2838, 1999.

R. Stroeve, B. Kappen, S. GielenStimulus segmentation in a stochastic neural network with exogenous signals. Artificial NeuralNetworks 9, Vol.2, pp. 732-737, 1999.

C.C.A.M. Gielen

24

Too early to explain all experimental data with a single model. Motor Control, 4, 600-63,2000.

L.N. Cornelisse, W.J.J.M. Scheenen, W.J.H. Koopman, E.W. Roubos, S.C.A.M. GielenMinimal model for intracellular Calcium oscillations and electrical bursting in Melanotropecells of Xenopus Laevis. Neural Computation, 13, 113-137, 2000.

W.P. Medendorp, J.D. Crawford, D.Y.P. Henriques, J.A.M. van Gisbergen, C.C.A.M. GielenKinematic strategies for upper arm-forearm coordination in three dimensions. J.Neurophysiol. 84, 2302-2316, 2000.

S. GielenCoordination of redundant manipulators. Rediction of degrees of freedom.In : Prerational Intelligenc: Adaptive Behavior and Intelligent Systems without Symbols andLogic- Volume 1. Eds. H. Cruse, J. Dean, H. Ritter. pp. 305-322. Kluwer Academic Publishers,Dordrecht.

N. Keijser, M. Horstink, J. van Hilten, C.C.A.M. GielenDetection and assessment of the severity of Levodopa induced dyskinesia in patients withParkinson’s Disease by Neural Networks. Movement Disorders, 15, 1104-1111, 2000.

T. Welter, B. van Bolhuis, C.C.A.M. Gielen, M. BobbertRelevance of the force-velocity relationship in the activation of mono- and bi-articular muscles inslow arm movements in humans. Motor Control, 4, 420-438, 2000.

W.P. Medendorp, J.A.M. van Gisbergen, S. van Pelt, C.C.A.M. GielenContext compensation in the vestibulo-ocular reflex during active head rotations.J. Neurophysiol. 84, 2904-2917, 2000.

S. GielenHelmholtz: Founder of the Action-Perception Theory.In: Classics in Movement Science. Editor: M. Latash. Pages 221-242. Human Kinetics,Champaign, Illinois, USA.

Handbook of Biological Physics, Vol. IVEditors: S. Gielen and F. Moss. Elsevier, Amsterdam, 2001.

Population coding: efficiency and interpretation of neuronal activity.C.C.A.M. GielenIn: Handbook of Biological Physics, Vol. IV, pp. 853-886. Editors: S. Gielen and F. Moss,Elsevier, Amsterdam, 2001.

S. Stroeve and C. GielenCorrelation between uncoupled conductance-based integrate-and-fireneurons due to common and synchronous presynaptic firing. NeuralComputation, 13, 2005-2030, 2001.

C. Apolloni, C. Orovas, J. Taylor, W. Fellenz, S. Gielen, M. WesterdijkA general framework for symbol and rule extraction in neural networks.Proceedings International Joint Conference on Neural Networks, pp. ??-??, 2000.

M.D. Klein Breteler, S.C.A.M. Gielen, R.G.J. MeulenbroekEnd-point constraints in aiming movements: effects of approach angle and speed.Biological Cybernetics 85, 65-75, 2001

25

B. Kappen, S. Gielen, W. Wiegerinck, A.T. Cemgil, T. Heskes, M. Nijman, M. LeisinkApproximate Reasoning: Real-world applications of graphical models.In: Foundations of Real-World Intelligence. Y. Uesaka, P. Kanerva, H. Asoh (Eds.). CLSIPublications, Stanford, California, USA. Pp. 73-122 (2001)

M.A. Admiraal , W.P. Medendorp, C.C.A.M. GielenThree-dimensional head and upper arm orientations during kinematically redundantmovements. Experimental Brain Research 142: 181-192, 2002.

W.P. Medendorp, J.A.M. van Gisbergen, C.C.A.M. GielenHuman gaze stabilization during active head translations.J. Neurophysiol. 87, 295-304, 2002.

L.P. Pantic, J.J. Torres, H.J. Kappen, S.C.A.M. GielenAssociative memory with dynamic synapses. Neural Computation, 14, 2903-2923, 2002.

M.A. Admiraal , W.P. Medendorp, C.C.A.M. GielenThree-dimensional head and upper arm orientations during kinematically redundantmovements. Experimental Brain Research 142: 181-192, 2002.

Klein-Breteler M.D., Meulenbroek R.G.J., Gielen C.C.A.M. An evaluation of the minimum-jerk and minimum torque-change principles at the path, trajectory, and movement-cost level.Motor Control, 6, 69-83, 2002.

L.N. Cornelisse, R. Deumens, J.J.A. Coenen, E.W. Roubos, C.C.A.M. Gielen, D.L. Ypey,B.G. Jenks, W.J.J.M. ScheenenSauvagine regulates Ca2+ oscillations and electrical membrane activity of melanotrope cells ofXenopus laevis.Journal of Endocrinology, 14: 778-787, 2002.

S.F. Gabel, H. Misslisch, C.C.A.M. Gielen, J. DuysensResponses of neurons in area VIP to self-induced and external visualstimuli. Experimental Brain Research, 147: 520-528, 2002.

N.L.W. Keijsers, M.W.I.M. Horstink, S.C.A.M. GielenAutomatic assessment of levodopa-induced dyskinesias in daily-life by neural networks.Movement Disorders, 18, 70-80, 2003a.

N.L.W. Keijsers, M.W.I.M. Horstink, S.C.A.M. GielenMovement parameters which distinguish between normal voluntary movements anddyskinesia in patients with Parkinson’s Disease.Human Movement Science, 22, 67-89, 2003b.

F. Hermens and S. GielenVisual and haptic matching of perceived orientation of lines.Perception, 32, 235-248, 2003.

F. Hermens and S. GielenCatching oriented objectsActa Psychologica, 114, 17-39, 2003.

N.L.W. Keijsers, M.W.I.M. Horstink, S.C.A.M. GielenOnline monitoring of dyskinesia in patients with Parkinson’s disease.IEEE Engineering in Medicine and Biology. 22, 96-103, 2003c.

26

D. Y. P. Henriques, W. P. Medendorp, C. C. A. M. Gielen, J. D. CrawfordGeometric computations underlying eye-hand coordination:orientations of the two eyes and the headExperimental Brain Research, 152, 70-78, 2003.

M.A. Admiraal , N.L.W. Keijsers, C.C.A.M. GielenInteraction Between Gaze and Pointing Toward Remembered Visual TargetsJ. Neurophysiol. 90, 2136-2148, 2003.

27

14 ReferencesAmorim MA, Lang W, Lindinger G, et al. (2000) Modulation of spatial orientationprocessing by mental imagery instructions: A MEG study of representationalmomentum. J Cognitive Neurosci 12 (4): 569-582.

Attias, H. (1999) Independent Factor Analysis. Neural Computation, 11, 803-851.

Attias, H. & Schreiner, C. E. (1998) Blind Source Separation and Deconvolution: theDynamic Component Analysis Algorithm. Neural Computation. (10) 1373-1424.

Bennet, K. P., Demiriz, A. & Maclin, R. (2002) Exploiting unlabeled data inensemble methods. Proceedings of Int. Conference on Knowledge Discovery andData mining KDD'02, 289 - 296.

Besson, M., Faïta, F. & Requin, J. (1994). Brain Waves Associated with MusicalIncongruities Differ for Musicians and Non-musicians. Neuroscience Letters, 168,1010-105.

Birbauer, N., Ghanayim, N., Hinterberger, T., Iversen, I., Kotchoubey, B., Kubler, A.,Perelmouter, J. Taub, E. & Flor, H. (1999). A spelling divice for the paralyzed.Nature, 398, 297-298.

Bregman, A.S. (1990). Auditory scene analysis. Cambridge, MA., MIT Press.

Buonomano D.V.& Merzenich M.M. (1998) Cortical plasticity: From synapses tomapsAnnu. Rev. Neurosci. 21, 149-186.

Cardoso, J.F. (1999) High-order contrasts for ICA. Neural Computation. 11(1), 157 -192.

Cooper, G. & Meyer, L.B. (1960). The rhythmic structure of music. Chicago:University of Chicago Press.

David, O., Garnero, L., Cosmelli, D. & Varela, F. J. (2002). Estimation of neuraldynamics from MEG/EEG cortical current density maps: application to thereconstruction of large-scale cortical synchrony. Transactions on BiomedicalEngineering, 49, 975-987.

deFreitas, J. F. G. and Milo, M. and Clarkson, P. and Niranjan, M. and Gee, A. H.(1999) Sequential support vector machines. Proceedings of Int. Workshop on neuralnetworks for signal processing IX, 31 - 40.

Desain, P. (1992). A (de)composable theory of rhythm perception. Music Perception,9, 439-454.

28

Desain, P. & Honing, H. (2003). Reading the musical mind, detection of perceivedand imagined rhythm from ERP. International Joint Conference on ArtificialIntelligence. Rencon workshop. Acapulco.

Desain, P. & Honing, H. (2003) The perception of time: the formation of rhythmiccategories and metric priming. Perception.

Desain, P., Honing, H., Van Thienen, H. & Windsor, L.W. (1998). Computationalmodeling of music cognition: problem or solution? Music Perception 16 (1), 151-166.

Desain, P. & Windsor, L.W. (eds.) (2000). Rhythm perception and production. Swets& Zeitlinger, Lisse.

Donchin, E., Spencer, K.M., Wijensinghe, R. (2000). The mental prosthesis:Assessing the speed of a P300-based brain-computer interface. IEEE Trans. Rehab.Eng. 8:174-179.

Donoghue, J.P. (2002) Connecting cortex to machines: recent advances in braininterfaces. Nature neuroscience Supplement. 5. 1085-1088.

Duncan-Johnson, C.C. & Donchin, E. (1977). On quantifying suprise: The variationof event-related potentials with subjective probability. Psychophysiology, 14, 456-467

Guedala, I. D. and London, M. and Werman, M. (1999) An on-line agglomerativeclustering method for nonstationary data. Neural computation, 11, 521 - 540.

Mejuto, C. and Principe, J. C. (1999) A second-order method for blind sourceseparation of convolutive mixtures. Proceedings of Int. Workshop on ICA '99, 395 -400

Honing, H. (2002) Structure and interpretation of rhythm and timing. Tijdschrift voorMuziektheorie, 7 (3), 227-232.

Hyvärinen, A., Karhunen, J. & Oja, E. (2001). Independent Component Analysis(New York, John Wiley & Sons.

Janata, P. (2001). Brain electrical activity evoked by mental formation of auditoryexpectations and images. Brain Topography, 13, 169-193.

Jasper, H.H. (1958). The 10-20 system of the international federation.Electroencephalography and Clinical Neurophysiology, 10, 371-375.

Jenks, K. M., Jongsma, M.L.A, Desain, P., Honing, H. & van Rijn, C.M. (2002).Omission evoked potentials (OEPs) in rhythmically trained and non-trained subjects.Journal of Cognitive Neuroscience. B90 Suppl.

Knight, R.T.( 1996). Contribution of human hippocampal region to novelty detection.Nature, 383, 256-259.

29

Jongsma, M.L.A, Desain, P, Honing, H., van Rijn, C.M, Jenks, K.M. & CoenenA.M.L. (2002). AEP P300 modulation by two different temporal contexts in bothrhythmically trained and nontrained subjects. Journal of Cognitive Neuroscience. A94Suppl..

Kilgard MP, Merzenich MM (1998) Plasticity of temporal information processing inthe primary auditory cortex. Nature Neuroscience, 1, 727-731.

Knight, R.T.( 1996). Contribution of human hippocampal region to novelty detection.Nature, 383, 256-259.

Kosslyn, S.M., Ganis, G. & Thompson, W.L. (2001). Neural foundations of imagery.Nature Reviews Neuroscience, 2, 635-642.

Jung, T.-P., Makeig, S., Humphries, C., Lee, T.-W., Mckeown, M. J., Iragui, V. &Sejnowski, T. J. (2000). Removing electroencephalographic artifacts by blind sourceseparation. Psychophysiology, 37, 163-178.

Large, E.W. & Jones, M.R. (1999). The dynamics of attending: How people tracktime-varying events. Psychological Review, 10(1), 119-159.

Longuet-Higgins, H.C. (1976). The perception of melodies. Nature, 263, 646-653.

Mehring C., Rickert J., Vaadia E., Cardoso de Olineira S., Aertsen A., Rotter S.(2003) Inference of hand movements from local field potentials in monkey motorcortex. Nature Neuroscience 6, 1253-1254.

Michon, J.A. & Jackson, J.L. (1985). Time, mind, and behavior. Springer, Berlin.

Kuller GR, Neuper C., Pfurtscheller G. (2003) Implementation of a telemonitoringsystem for the control of an EEG-based brain-computer interface. IEEE Trans NeuralSystems & Rehabilitation, 11, 54-59.

Näätänen, R. (2001). The perception of speech sounds by the human brain as reflectedby the mismatch negativity (MMN) and its magnetic equivalent (MMNm).Psychophysiology, 38, 1-21.

Nguyen Thi, H.L. and Jutten, Ch. (1995) Blind sources separation for convolutivemixtures. Signal Processing, (45)2, 209 - 229.

Nicolelis, M.A.L. (2001). Actions from thoughts. Nature, 409, 403-407. (and seehttp://www.nicolelislab.net/NLNet/Load/Papers/Signals.pdf).

Picton, T.W., Bentin, S., Berg, P., Donchin, E., Hillyard, S.A., Johnson, R., Miller,G.A., Ritter, W., Ruchkin, D.S., Rugg, M.D. & Taylor, M.J.( 2000). Guidelines forusing human event-related potentials to study cognition: recording standards andpublication criteria. Psychophysiology, 37, 127-152.

30

Pfurtscheller and Neuper (2001). Motor imagery and direct brain-computercommunication, Proc. of the IEEE, pp. 1123-1134.

Pfurtscheller G, Muller GR, Pfurtscheller J, Gerner HJ, Rupp R (2003) 'Thought' -control of functional electrical stimulation to restore hand grasp in a patient withtetraplegia. Neuroscience Letters 351 (1): 33-36 Nov. 6

Regan, D. (1989). Human brain electrophysiology: evoked potentials and evokedmagnetic fields in science and medicine. Elsevier, New York.

Requin, J. (1978). Attention and performance VII. Erlbaum, Hillsdale, N.J

Suppes, P. Han, B. & Lu, Z.-L. (1998). Brain-wave recognition of sentences.Proceedings of the National Academy of Sciences of the United States of America, 95,15861-15866.

van der Veen, A.J. (1998) Algebraic methods for deterministic blind beamforming.Proceedings of the IEEE, (86)10, 1987 - 2008.

Ypma, A., A. Leshem and R.P.W. Duin (2002). Blind separation of rotatingmachinesources: bilinear forms and convolutive mixtures. Neurocomputing 49/1-4.349-368.

Ypma, A. C. Melissant, O. Baunbaek-Jensen and R.P.W. Duin (2001) Healthmonitoring with learning methods. Proceedings of Int. Conf. on Artificial NeuralNetworks ICANN'01, Vienna. 541 - 546.

Xerri C, Merzenich MM, Peterson BE, Jenkins W (1998) Plasticity of primarysomatosensory cortex paralleling sensorimotor skill recovery from stroke in adultmonkeys J. Neurophysiol. 79, 2119-2148.

31

Appendix NWO Cognition Programme Motivation

In this proposal we exploit what is known about the EEG signal and the traces thatneural activity of processing imagined temporal patterns leaves. We combine thiswith state of the art stochastic techniques from artificial intelligence to classifypatterns in the noisy signal. The patterns are used to allow users (patients withparalysis) to communicate and to control their environment. That is the aim.

It is a daring step, to go immediately for direct mind-control, while so much is stillunknown about human cognition. But is just the right time for this step. We seebeginning results emerging from labs around the world. The results obtained withneural control of robot arms using invasive techniques in monkeys are spectacular.But progress for non-invasive Brain-Machine-Interfaces is needed before it can beused for a wide range of human patients. For the group of locked-in patients, whocannot communicate at all, the need for a neural interface that works, howeverlimited, is pressing. We also foresee wider application in other domains (as reflectedby SONY CSL participation).

It is our experience that working towards an application is not to be valued less thanfundamental research. On the contrary, the need for reliable methods often pushes thepurely scientific work in a different direction, and frames the questions in a new light.High-level cognitive processes, like mental imagery, which take place in the absenceof any direct stimulus have only recently become the topic of investigation forneuroscience, but exciting discoveries have already been made (like the activation ofperceptual areas during visual imagery, and the omitted evoked auditory response),and our work may stimulate research in this direction.

Furthermore, ways to reduce noise and classify single EEG trials reliably can be useddirectly in fundamental research, which is often hampered by the noisiness of thesignal and still needs a prohibitively large number of trials. The real-time character ofour software will open up the methodology of on-line interactive and learningexperiments, an area that is difficult to realize with the present setups.

We have a fantastic team covering the whole field: music, experimental psychology,ERP methodology, signal processing, artificial intelligence, and the clinical domain.And we have found an enthusiastic user committee spanning consumer electronics,high tech bio-measurement devices, large EEG databases, and patient care. Theparticipants are willing to invest. Awaiting funding, small pilot experiments arerunning at NICI and at Stanford University. All expertise is on board and ready tobegin.

The Cognition programme strives to understand how humans perceive theirenvironment, reason and think, and initiate action. These insights may now help thedevelopment of a device that takes over the last steps when people can no longer puttheir thoughts into action. This constitutes a great demonstration of the aim of theprogramme and of the usefulness of Cognition research for a general audience.