Emotion Recognition of EEG Underlying Favourite Music by Support Vector Machine

Kevin C. Tseng Ph.D., Member IEEE Product Design and Development Laboratory

Department of Industrial Design College of Management Chang Gung University

Taoyuan, Taiwan ktseng@pddlab.org

Bor-Shyh Lin Ph.D., Member IEEE Institute of Imaging and Biomedical Photonics

National Chiao-Tung University Tainan, Taiwan

borshyhlin@mail.edu.tw

Chang-Mu Han Product Design and Development Laboratory

Department of Industrial Design College of Management Chang Gung University

Taoyuan, Taiwan chan@pddlab.org

Psi-Shi Wang Department of Physical Medicine and Rehabilitation

Chang Gung Medical Foundation Taoyuan, Taiwan

w750322@yahoo.com.tw

Abstract—This study aims to research the relationship between electroencephalography (EEG) at the prefrontal cortex (PFC) and emotion in the condition of different preference levels of music by applying a support vector machine (SVM). To achieve this, this study presents an EEG-based brain computer interface (BCI) music player, which can simultaneously analyse brain activities in real time and objectively provide therapists with physiological data for emotion detection in the experiment. The SVM result shows that more than 80% accuracy of elicited emotion based on 28 participants was analysed under the two factors of the frontal midline theta and alpha relation ratio. As such, it might suggest that significantly different stimuli are capable of enticing discernible EEG responses at frontal lobes, which is an indication of emotion and of providing an effective approach for application to multimedia with the abilities of EEG interpretation.

Index Terms—EEG, emotion, and support vector machine

I. INTRODUCTION Dementia is a more serious long-term decline of cognitive

functions than that associated with normal aging, affecting memory, attention, and emotion, etc. Through the investigation of the Taiwan Alzheimer Disease Association, the growing population with dementia in Taiwan is inevitable. To date, more than 170,000 people with dementia have lived in Taiwan; it is conceivable that the population number will rise to 300, 000 underlying the influence of aging society in the next 25 years [1]. Without proper treatments, people with dementia would gradually lose the abilities to complete daily life activities, thus dramatically burdening social and medical resources.

So far, many studies have proposed various approaches to alleviate multi-syndromes of dementia and to help the patients to control the emotional illness since it is incurable yet preventable [2]. For example, music therapy clinically effects positive outcomes on behavioural and psychiatric disorders, such as emotion disorder [3]. Undoubtedly, music enables listeners to moderate and control negative psychological attitudes. In many studies, however, the methods applied for evaluation of mental and cognitive functions are subjective assessments, such as the mini-mental state examination [4], abbreviated mental test [5], and Montreal cognitive assessment. Such assessments and therapy greatly depend on the doctor’s experience and lack of physiological evidences to support assessed outcomes. To help the patients to manage negative emotions, assist therapists in providing proper treatments, and inform caregivers via real-time responses, it is developing a reliable device for detection of brain activities in order to interpret emotion is necessary.

Currently, vast researches on electroencephalography (EEG) for investigating various responses of brain activities to different stimulations are in progress. Reflection of neurons for specific events from the external environment can be modulated by temporal frequency. Event-related synchronization for interpreting brain activities, however, is not only temporal but also spatial. In other words, different anatomical regions have specialized features. To detect and understand the musical effects on emotional response and control, the prefrontal cortex (PFC) is a eligible brain area because it represents the abilities of cognitive control, emotional processing, and mental function for activities in daily life. For example, frontal-midline theta significantly

978-1-4673-5936-8/13/$31.00 ©2013 IEEE 155

increases with positive emotion, such as pleasance [6], whereas it decreases with negative emotion, such as anxiety. Alpha power from the left PFC decreases when listening to positive music, along with that of the right frontal cortex when listening to negative music, indicating that alpha power in the right and left hemispheres of the PFC is inversely in line with emotion; only few studies have stated that beta power is related to emotion [7]. However, the inverse correlation of alpha with emotion in the PFC is still in debate. For instance, Wheeler et al. found that the left frontal cortex became more activated by positive films, whereas Sammler et al. indicated no significant difference at the left and right PFC while listening to pleasant music [7]. Therefore, current EEG research to investigate the links between music and psychological responses and to assess psychological characteristics is still in the developing stage [8].

Accordingly, this study aims to develop an EEG-based brain computer interface (BCI) device to evaluate emotion not only for patients with dementia but also for therapists. Using this device, the patients are able to achieve emotion detection and control. In the meantime, the therapists obtain objective EEG responses and emotion indication for providing proper treatment and further investigation. The present study interpreted EEG evoked by different preference levels of music, denoting emotional levels from neutral to arousal. The study investigated EEG responses associated with emotion by using a support vector machine (SVM). We hypothesized that the more favourite music would evoke stronger EEG responses at the PFC than less favourite music would. Otherwise stated, the listeners should receive more positive effects with more favourable music. For conducting the research, a real-time wireless EEG-based BCI system was developed and applied for fetching EEG responses. Also, a more effective approach, continuous response digital interface (CRDI), for quantifying subjective preference levels of music was used to record listeners’ feelings toward the music in real time [9]. This paper was organised as the following: The system architecture and overall experimental details are described in Sections II and III. Section IV shows the experiment results. Finally, discussion and conclusion of this study is given in Section V.

II. MATERIALS The proposed EEG-based BCI music system contains a

multi-channel EEG acquisition system, a Bluetooth module, a real-time analysis module, and a music controller. The overall scheme of the system is shown in Fig. 1. The wireless multi-channel EEG acquisition system integrated with the Bluetooth module is designed to simultaneously acquire multi-channel EEG signals and to wirelessly transmit the EEG signals to the computer for analyzing real-time responses. The EEG signals went through a noise filter, amplifier, and analog-to-digital (AD) converter to the Bluetooth for transmission. The music controller contains a music control program built with LabVIEW for control of the following experiment. It will receive and analyze the acquired EEG signal and manage the data for further analysis of emotion.

Fig. 1. The overall scheme of the EEG-based BCI system

Stimuli To investigate EEG responses under the stimulation of

music, three distinctive categories of music, including high focus (HF), K448, and favourite song (FS), were used for the subjects to test the correlates of brain activities and positive emotion. HF, released by the Brain Sync Company for brain wave therapy, was composed of two tracks of pure music without voice. HF is full of natural sounds, such as a cricket’s cry, and slow-tuned sound waves. It claims that listeners should quickly reach their high-performance brain states for thinking and concentration under HF. In this study, the second track of HF blending ambient sounds was applied to the experiment.

In addition, K448 composed by Mozart is a famous concerto. As popularly used as a stimulus in many studies, K448 brings a well-known Mozart effect on brain therapy and mental improvement. Therefore, serving as an indication of happiness, K448 is an index in the experiment for understanding the different musical effects. Moreover, because the total durations of HF and K448 are beyond eight minutes, this might be too long to affect the state of listener in the experiment. Therefore, the duration of HF and K448 was appropriately reduced.

FS selected by the subjects varies tremendously based on the taste of an individual. It is hard to define a regulation that distinguishes all of the types of music. Nevertheless, we define individual music preference level by CRDI to help the participants to select an appropriate song. On average, the mean duration and mean tempo of FS approximates 280s [seconds] (range: 175–491s) and 116 beats per minute [bpm] (range: 75.6–174.3bpm). Compared with HF and K448, FS has both median properties since K448 has the highest tempo but the shortest duration (156.41bpm and 258s), whereas HF has the lowest tempo and the longest duration (81.26bpm and 289s).

III. METHODS To conduct an investigation of the stimuli impacting the

listener and EEG variation with the emotions evoked by the stimuli, an EEG experiment associated with distinctive types of music was held alongside the following criteria.

156

A. Subjects A total of 28 participants, from ages 18 to 32 years old with

a mean are of 21.75 years old, have joined the experiment. Thirteen out of 28 participants are male. To reduce the implication of professional musical knowledge on preference [9], all of the participants have no formal musical training and no experience with playing instruments. Furthermore, all of the participants report no hearing problems and no neurological diseases that will influence the outcome negatively.

B. Experiment Design The overall procedure of the experiment comprises a CRDI

test and an EEG-music test, shown in Fig. 2. In the beginning of the experiment, the participants need to go through a CRDI test for rating subjective preference levels of HF, K448, and FS. Staff give a participant introduction, instruct them to answer a questionnaire for personal recording, and ensure the sleeping time of each participant is greater than six hours. Five one-minute durations of silence intersect between each stimulus during the entire experiment. The intersection prevents the participants from the interference between the songs. The EEG signals within the first minute will be neglected in order to provide time for the mind’s and body’s adaptation to the environment. The second minute is taken as a baseline for data processing. In spite of being familiar with FS, the participants need no contact with the musical pieces of HF and K448 prior to the experiment in order to minimize the implication of familiarity on the subjective preference level. During the experiment, the participant sits alone with a comfortable chair in a soundproof room that helps to filter background noise and interference by people. An auditory sound indicates the start of the musical trial for the participants. During the trial, the stimuli are played via two speakers at half volume through Windows Media Player.

Fig. 2. The experiment procedure is classified into five one-minute sections with no sound and three stimuli.

C. EEG Recording EEG signals were recorded with Ag/AgCl wet electrodes

from five electrode sites positioned on the frontal lobe according to the International 10-20 system. Fpz served as ground and left mastoid as a reference. The signals from Fp1, Fp2, and Fz were used for investigating the effects of various musical pieces on emotion processing, noted by active theta and alpha rhythms [7]. The EEG power spectrum was calculated by a 256-point Fast Fourier Transform (FFT) with 0.5 overlap to average power within the particular frequency band. Before FFT, the modified signal segments were windowed by the Hanning function via LabVIEW so as to

reduce spectral leakage. By the definition of the frequency band for EEG, theta, alpha, and beta could be obtained by calculating the mean power of 4–7Hz, 8–13Hz, and 14–40Hz, respectively.

D. Feature Extraction and Classification To reduce the individual variance of mean EEG power,

each power was normalized by the value of the baseline in the same frequency band across the scalp (N=3), showing as

∑=

×= N

SFS

FSFS

BEEGN

EEGNEEG

1,

,,

1

(1)

NEEG represents normalized EEG; S is the electrode positions, denoting Fp1, Fp2, and AFz; F is the frequency bands including θ, α, and β; BEEG represents EEG power at the period of the baseline. Furthermore, to investigate the asymmetric response of alpha power at PFC, a relation ratio (RR) was expressed as [10]

%100×+−=

LPRPLPRPRR (2)

RP denotes alpha power from the right hemisphere of the PFC (Fp2), and LP is from the left one (Fp1). This formula indicates that the subject is prone to positive emotions when the RR value increases.

In analysis, this study has employed SVM with a nonlinear kernel function for the recognition process of EEG responses and musical preference so as to classify emotion and brain activities. Its basic idea is an iterative learning process to devise an optimal hyperplane with a maximum margin for classifying data clusters. Therefore, this study applied MATLAB to build an SVM trainer and a classifier with the Gaussian radial basis function kernel for nonlinear classification.

IV. RESULTS In the CRDI test, the amplitude of the preference level of

music can be denoted by means of the level, which is shown in Fig. 3. The higher mean represents a more intensive preference that the musical excerpts brought on the subjects. Hence, according to a one-sample t test with the condition of using 128 as the median (from neutral to most favourite, 0–255), the subjects discernibly rated HF, k448, and FS as neutral, median, and high preference (test values of HF, k448, and FS are -11.08, -0.915, and 6.48 with p < 0.001, p = 0.368, and p < 0.001). Moreover, by one-way analysis of variance (ANOVA), the result also indicates that the features of the stimuli are significantly discernible from one another in order for the subjects to have different emotional arousals.

TABLE I shows the SVM classification results of elicited emotion at frontal midline theta and the result of alpha RR. As expected, they clearly reveal the high classification of EEG power underlying different stimuli—especially in the conditions of HF and FS, which drew more than 80%

157

accuracy. Compared with the CRDI result, the SVM results of both frontal theta and alpha RR indicate the distinct emotional arousal. These results might show that EEG signals interpret and recognize emotions with SVM efficiently. Nevertheless, median accuracy in the condition of K448, compared with the other two, might indicate that extreme emotion evokes more significant and distinguishable EEG responses. Moreover, no noticeable difference of accuracy in the conditions of HF, K448 and FS in frontal theta and alpha RR was found. This might suggest that the recognition by theta and alpha on emotion has similar performance.

Fig. 3 The average of preference levels of HF, K448, and FS, which the subjects rated in the CRDI test (*: p < 0.05; **: p < 0.01; ***: p <0.001).

TABLE I. SVM classification performances of the frontal midline theta and RR of alpha power

Analyzed Actual HF K448 FS

Accuracy of elicited emotion with SVM at frontal midline theta HF 85.71% 3.57% 10.71%

K448 10.71% 67.86% 21.43% FS 7.14% 10.71% 82.14%

Accuracy of elicited emotion of the RR result with SVM HF 85.71% 7.14% 7.14%

K448 7.14% 67.86% 25% FS 7.14% 10.71% 82.14%

V. DISCUSSION AND CONCLUSION This study presents EEG responses from PFC interpreting

human emotional arousal underlying discernible musical stimulation. According to the results of the CRDI test, each stimulus is significantly different from one another. However, this test differed from other studies that used tempo, rhythm, and melody as approaches for music recommendation [11]; the preference order is not line with the bpm result. The test implies that integration of the features and personal preference, instead of general features, of a song would be an effective approach for enticing a significant emotional state.

Since frontal midline theta and frontal alpha play important roles in emotional processing, the SVM results are in line with

many studies [7, 12], suggesting a comparable increase of frontal theta underlying distinguishable stimuli as well as asymmetry alpha. Interestingly, relatively low accuracy in the condition of K448, which evokes median emotion, might suggest extreme emotion would draw out significant EEG responses.

Finally, more cogent evidences on the relationship between frontal EEG powers and emotion under different daily scenes should be implemented. Also, great potential exists to figure out the correlation of EEG powers with the different stimulation of multimedia for further research on emotion control for people with dementia.

ACKNOWLEDGMENT This work was supported in part by the National Science

Council, Taiwan, ROC under Grant No. 100-2218-E-182-003- and No. 101-2218-E-182-002.

REFERENCES [1] H. C. Liu, K. N. Lin, E. L. Teng, and S. J. Wang, "Prevalence

and subtypes of dementia in Taiwan: A community survey of 5297 individuals," Journal of the American Geriatrics Society, vol. 43, pp. 144-149, 1995.

[2] J. C. d. l. Torre, "Alzheimer's disease is incurable but preventable," Journal of Alzheimer's Disease, vol. 20, pp. 861-870, 2010.

[3] M. Wall and A. Duffy, "The effects of music therapy for older people with dementia," British Journal of Nursing, vol. 19, pp. 108-113, 2010.

[4] T. N. Tombaugh and N. J. McIntyre, "The mini-mental state examination: A comprehensive review," Journal of the American Geriatrics Society, vol. 40, pp. 922-935, 1992.

[5] D. G. Swain, A. G. O'Brien, and P. G. Nightingale, "Cognitive assessment in elderly patients admitted to hospital: The relationship between the abbreviated mental test and the mini-mental state examination," Clinical Rehabilitation, vol. 13, pp. 503-508, 1999.

[6] L. Aftanas and S. Golocheikine, "Human anterior and frontal midline theta and lower alpha reflect emotionally positive state and internalized attention: high-resolution EEG investigation of meditation," Neuroscience letters, vol. 310, pp. 57-60, 2001.

[7] D. Sammler, M. Grigutsch, T. Fritz, and S. Koelsch, "Music and emotion: Electrophysiological correlates of the processing of pleasant and unpleasant music," Psychophysiology, vol. 44, pp. 293-304, 2007.

[8] Y. P. Lin, C. H. Wang, T. P. Jung, T. L. Wu, S. K. Jeng, J. R. Duann, and J. H. Chen, "EEG-based emotion recognition in music listening," IEEE Transactions on Biomedical Engineering, vol. 57, pp. 1798-1806, 2010.

[9] D. Gregory, "The continuous response digital interface: An analysis of reliability measures," Psychomusicology: Music, Mind and Brain, vol. 14, pp. 197-208, 1995.

[10] M. Popescu, A. Otsuka, and A. A. Ioannides, "Dynamics of brain activity in motor and frontal cortical areas during music listening: a magnetoencephalographic study," NeuroImage, vol. 21, pp. 1622-1638, 2004.

[11] F. Gouyon and S. Dixon, "A review of automatic rhythm description systems," Computer Music Journal, vol. 29, pp. 34-54, 2005.

[12] R. E. Wheeler, R. J. Davidson, and A. J. Tomarken, "Frontal brain asymmetry and emotional reactivity: A biological substrate of affective style," Psychophysiology, vol. 30, pp. 82-89, 1993.

158