NEUROPHONE: BRAIN-MOBILE PHONE INTERFACE USING A WIRELESS EEG HEADSET

Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu,

Matthew K. Mukerjee!, Mashfiqui Rabbi, and Rajeev D. S. Raizada

Dartmouth College, Hanover, NH, USA

Motivation• Mobile phones and neural signals are present are

accessible to many people. • Recent advances in technology has led to the

development in low-cost EEG headsets. • Smart phones are now powerful enough to run

sophisticated machine learning algorithms.• It is thus easy to interface neural signals with mobile

computing paradigms.

Introduction• This group proposed to used neural signals to control a

mobile phone. • They developed the NeuroPhone system that translates

and decodes neural signals to drive a mobile app using off-the-shelf wireless EEG headsets.

• This paper demonstrates their brain-controlled address app:• An application that uses the brain signals to select address

contacts to call.

Introduction• They implement their mobile app using two different

paradigms: P300 dialing and “Wink”-triggered dialing. • P300 signals are positive transient deflections in EEG that are

elicited in response to a rare or novel stimulus• The eye “Wink” is a type of EMG signal that is generated in

response to the contraction of skeletal muscle contraction.

Challenges• Research Grade EEG headsets

• Expensive (Often costing tens of thousands of dollars)• Offer very robust and reliable EEG signals

• Off-the-shelf EEG headsets• More affordable ($100-$500)• Electrode design and amplification are not as robust

• Results in noisy, low-quality signals.• Require more sophisticated processing techniques to classify neural

events.

• Most Off-the-shelf headsets are wireless and thus encrypt the EEG signals. • They are designed for synchronization with a computer (using wireless

dongle). • They complicate the process of developing a clean brain-mobile

interface.

Challenges• There is an energy cost for brain-mobile interfacing:

• Continuously streaming raw brain-signals wirelessly• Running classifiers on the phone introduces heavy processor

loads.

• Brain-mobile phones could likely be used in applications such as: walking, riding in a car or bicycle, shopping, etc. • Many of these cases present significant noise artifacts in the EEG

signals. • These signals will need to be filtered out to improve the brain-mobile

interface

NeuroPhone• The NeuroPhone system uses the

iPhone to display pictures of contacts in the phone’s address book.

• The pictures are displayed and flashed in random order.

• For the EEG mode, the user concentrates on a picture of the person they wish to call.

• For the wink mode, the person winks with the left or right eye to make the intended phone call

P300

• Whenever the user concentrates on a target stimulus among a pool of non-target stimulus, the target stimulus (flash) will elicit a positive peak in the EEG at around 300ms after stimulus onset (P-300).

• The P300 signal can be found on most EEG channels• Common on central and

parietal channels

NeuroPhone - P300 Paradigm

• In This case, there are 6 total stimuli on the screen (5 non-target and 1 target). The user visually attends to one of the photos while each photo is flashed in a random order. Whenever the target photo flashes, a P300 should be generated.

Wireless EEG Headset• Emotiv EPOC headset

• 14 data electrodes (2 reference electrodes)• Transmits encrypted data wirelessly to a

windows-based machine. (802.11) 2.4GHz• Low SNR• Contains build in gyroscope• ~$300

Pre-Processing• Signals were band-passed filtered to keep only the

relevant information within the P300 range. • Signal averaging was performed to increase the SNR

• This improves the quality of the signal while simultaneously adding lag to the system

Classification• To reduce complexity, only a subset of relevant channels

are used for classification. • Wink Mode

• Multivariate, naive Bayesian classifier.

• P300 Mode• Decision stump classifer

Implementation• Laptop relay is used for decoding of the encrypted Emotiv

signals• Encrypted EEG signals are sent from the phone to a laptop for

decryption (via WiFi). • Decrypted EEG signals are sent back to the phone.• Signals are sampled at 128 samples per second and transferred to

the phone at 4kbps per channel.

Wink Mode Classification• Emotiv head-set was put on

backwards to place two electrodes directly above the eyes.

• Data was collected by having the subject wink multiple times. – Data were labeled as “wink” or “non-

wink”

• A Bayesian classifier was trained by calculating the mean and variance of each wink and non-wink and building respective Gaussian models. – As can be seen, the two models do not

overlap leading to good classification

P300 Classification• The Gaussian distributions overlap too much and

therefore cannot be classified with a Bayesian classifier. • Signals from each of the six stimuli were band-passed

filtered between 0-9Hz. • The highest signal segment at around 300ms after

stimulus onset is extracted. • For classification, a decision stump is used where the

threshold is set to the maximum value of the extracted segment.

• Multiple sessions were collected on three subjects. • Subjects performed the test while sitting and while walking• The classifier was trained on five sessions from a single

subject and then tested on the remaining subjects. (I think). • Results are shown in table 1

– Precision: % of classified winks that are actual winks– Recall: % of actual winks that are classified as winks. – Accuracy: % of total events that are classified correctly

Results (Wink-Mode)

Results (P300 mode)• Data was collected with same set of subjects while sitting,

with loud background music and while standing up.

Discussion• Although data was classified using the P300 mode, large

amounts of averaging is needed to get decent classification accuracies. • This “unresponsiveness” of the system proves to be very

frustrating for the end user. • i.e. it can take 100 seconds to initiate a phone call with only 89% chance

of dialing the right person (with six to choose from).

• This System is currently not in any form to be used by subjects on a regular basis. • Looking into single trial classification techniques to speed up the

system.

Phone Loading Statistics• The CPU usage when running the application:

• 3.3% for the iPhone (iphone 3g?).

• Total memory usage:• 9.40MB memory used

• (9.14MB are for GUI elements).

• Continuous streaming raw EEG channels to the phone, and processing signals lead to battery drain (no quantitative measure given)• Looking into duty cycling to solve this phone.