A Proposal for a Research Project Investigating a Vibrotactile Musical Instrument

IntroductionTechnology and art seem to continuously influence and challenge each other in interesting ways. For

example, the development of new technologies for composing, performing, transcribing and recording

music has been substantial. As new music art forms have been derived from these technologies (e.g.,

Hip-Hop using turn table “scratching”), new technologies have been developed to fully support the new

art form (e.g., software scratching tools are now available). Digital technologies have caused an

explosion of devices, technologies and techniques that have revolutionized many art forms. For

example, devices such as electronic synthesizers allow artists to compose, record and perform music in

ways that were not possible or even conceived of prior to the invention of computers. Digital processing,

sensing and storage technologies have driven new paradigms of music such as music controlled through

gestures or dance (Camurri et al., 2000; Ng, 2002; Winkler, 1995), or music without an audio component

or that is only seen (Chew & Francois, 2003; Smith & Williams, 1997), and now tactile music (Gunther &

O'Modhrain, 2003; Karam, Nespoli, Russo, & Fels, 2009). The focus of this research proposal will be the

exploration, development and evaluation of artistic and technological tools for tactile music.

New technology has enabled the possibility of a vibrotactile music system, in which musical patterns,

delivered entirely through vibration presented to the skin can be developed. This system would be

capable of supporting the creation and delivery of music, which can contain many, if not all, of the

elements of traditional audio based music, such as intensity, tempo, rhythm, even pitch and timbre, but

is completely devoid of an audio component. Instead, this form of music would be comprised entirely of

vibrotactile stimulation which is delivered to the skin and perceived through the tactile sensory channel.

A complete vibrotactile music system or instrument will have an input controller, which I will design and

build, and a vibrotactile display. This combination of an input control and a vibrotactile display will result

in the first vibrotactile instrument.

Research Objective The purpose of this research is to begin to scientifically explore a new art form called vibrotactile music

and some of the issues related to it. The purpose of this exploration is to begin to develop a model of

vibrotactile composition that will provide a foundation for this new art form. Using a pre-existing high

definition vibrotactile display called The Emoti-Chair and several versions of “yet-to-be-built” vibrotactile

music interfaces, some of the many technological, psychological, physiological, neurological and human

factor issues that vibrotactile music raises will be investigated. Research methods traditionally employed

in the field of Human Factors will be used to determine the affect of the vibrotactile music interface on

the creation of vibrotactile music, and on further iterations of the music interface design. Using methods

from the fields of psychology, psychophysics and neurology, the abilities of the human tactile system to

perceive the resulting vibrotactile music and the affect such stimulation has on people’s attitudes,

enjoyment and experiences of music will be explored.

BackgroundThis research will draw on several areas of scholarship and research including music theory and

composition, psychology, psychophysics, neuroscience, human factors, human computer interaction and

haptics. It will consist of a technological extension of the work carried on The Emoti-Chair (C. Branje,

Karam, Russo, & Fels, 2009; Karam, Branje, Price, Russo, & Fels, 2007; Karam & Fels, 2008; Karam,

Nespoli, Russo, & Fels, 2009) as well as an evaluation of artistic processes and audience reactions

resulting from the introduction of this extension. The Emoti-chair was developed to explore the

translation of audio-based music into vibrotactile-based music in order to provide greater access to film

sound and music for the deaf and hard-of-hearing. Initial studies have shown that conveying emotional

properties of music through the Emoti-Chair is possible (Branje, Fels, Russo, & Nespoli, 2010; Karam,

Branje, Price, Russo, & Fels, 2007).

The Emoti-ChairThe Emoti-Chair (C. Branje, Karam, Russo, & Fels, 2009) is a vibrotactile display capable of delivering

complex vibrotactile information to a seated user. Embedded in the Emoti-Chair are sixteen voice coils

that serve as vibrotactile stimulators. Various models of this vibrotactile display exist, however most

commonly the Emoti-Chair has eight independent information channels, meaning eight separate

vibrotactile signals can be delivered to the display simultaneously. Each channel is independent and

capable of vibrating at all frequencies within the tactile range (20 Hz - 1000 Hz). The Emoti-Chair uses

the conceptual framework of the Model Human Cochlea (MHC) developed by Karam et. al (2008). The

primary principle of the MHC is to model the distribution of frequencies on the skin according to how it

is accomplished by the human cochlea. The frequency components of an audio signal are divided and

spread across the human back as though the cochlea were uncoiled and stretched out forming a line

along the back.

Although many vibrotactile interfaces have been developed and examined (Gemperle, Ota, & Siewiorek,

2001; Hogema, De Vries, Van Erp, & Kiefer, 2009; Nanayakkara, 2009), the Emoti-Chair is one of the few

that use voice coils as the vibrotactile stimulator. Most often other devices use motors, such as those

found in massage chairs or cell phones. Using voice coils offers several advantages over using motors to

produce the vibrotactle stimuli. First, there is independent control of the frequency and amplitude

variables. This means that many traditional aspects of audio music such as melody and intensity will

translate easily to tactile music. Second, voice coils can be driven using widely available commercial

amplification equipment, making prototyping and eventual manufacture of a vibrotactile display using

voice coils economically feasible.

Audio Psychophysics The human sense of hearing facilitates speech, the enjoyment of music, navigation of the world and

many other human endeavours. There is a great deal of literature concerning the psychophysics of

human hearing going back almost a century. Since the early 20 th century, it has been accepted that the

human frequency range of hearing is approximately 20Hz to 20000 Hz, with a significant amount of

individual variability (Pumphrey, 1951). The commonly accepted threshold for the lowest energy sound

perceptible by humans is Root Mean Square (RMS) sound pressure of 20 µPa (Gelfand, 1998) but as with

the frequency range, this threshold is subject to individual differences. It will be important to consider

any knowledge about hearing when developing a tactile instrument because although the ear will not be

studied explicitly in this thesis, information known about the human ear and the sense of hearing will

influence the study of the tactile system. This is an approach similar to the one used to study the Emoti-

Chair in that human cochlea influenced the model human cochlea (MHC) used in the Emoti-Chair

(Karam, et al. 2007). Since the two perceptual systems are so similar it is likely that the vase amount of

knowledge knowing about the sense of hearing will be useful in the study of the tactile sense.

Tactile PsychophysicsTo a much lesser degree than for human hearing, there has been some work exploring the vibrotactile

perceptual ability of humans. It has been found that the ability of humans to detect vibration through

the tactile channel is analogous to the ability to perceive vibrations through the audio channel (Von

Bekesy, 1959)(reference), although the perceptual fidelity of the skin tactile system is much lower than

the auditory system (Von Bekesy, 1959). For example, the human frequency range of hearing is 20Hz to

20000 Hz but the range of vibrotactile frequency perception in the skin is approximately 20Hz to

1000Hz, with maximum sensitivity to amplitude at approximately 250Hz. In addition, the vibrotactile

receptors in the skin detect differences in frequencies with a much lower resolution than the auditory

system (Boothroyd & Cawkwell, 1970; Mahns, Perkins, Sahai, Robinson, & Rowe, 2006; Pertovaara &

Hamalainen, 1981; Pongrac, 2008; Verrillo, 1963). These important distinctions between audio and

tactile stimuli and the human ability to perceive them, must be taken into account when a vibrotactile

instrument is developed. For example, it would make little sense to present vibrotactile stimulation with

frequencies above 1000 Hz which the skin is extremely poor at perceiving. Similarly it would be ill-

advised to create vibrotactile “notes” with frequencies that are too close together since the skin would

not be able to differentiate them. This would be analogous to creating a piano (admittedly a very large

one) with its highest note at 30 kHz and intervals between notes of only 1 Hz.

Tactile NeurologyThere has been some neurological evidence collected through fMRI studies suggesting that the

processing of vibration stimulation applied to the skin as vibrotactation and to the ear as sound, may

occur in some overlapping areas of the brain (Foxe, 2009; Schürmann, Caetano, Hlushchuk, Jousmäki, &

Hari, 2006; Yau, Olenczak, Dammann, & Bensmaia, 2009). This may have important consequences

regarding the perception of vibrotactile music and may offer some explanation as to why participants

react similarly to music played auditorily and vibrotactally (C. Branje, Fels, Russo, & Nespoli, 2010;

Karam et al., 2007; Karam & Fels, 2008; Karam, Nespoli, Russo, & Fels, 2009). Although no explicit brain

function models will be developed as a result of this work, data collected through EEG may provide

additional evidence that corroborate the data collected through physiological measures or self reports

and may help to clarify what is occurring with the human audience members.

Tactile DisplaysTactile displays have been used by the deaf and hard-of-hearing for numerous applications and over

many years. The earliest known formal use of the tactile system for perceiving sound information is the

Tadoma method (Chomsky, 1986) in which a deaf person will place his hands on a speaker’s throat and

lips in order to feel the vibrations and perceive the speech produced. As electronic technology

improved, many devices were constructed to improve on the Tadoma method. Another simple and

common tactile display for used by the deaf and hard-of-hearing were simple audio speakers used

during what is called ``speaker listening``, where during which an individuals will place her his hands on

a single speaker in order to feel the vibrations produced. These early vibrotactile tactile devices often

used motors or voice coils arranged in an array to produce vibrations that were passively detected by

the skin. Other more recent developments, such as refreshable Braille which used piston-like actuators

(American Foundation for The Blind, 2010), depend on active perception. In most of these applications,

tactile displays were used to transmit speech, written text, directional information (Hogema et al., 2009)

or even visual information (Sampaio, Maris, & Bach-y-Rita, 2001). However, there has been little formal

research on the transmission of musical information through the tactile channel.

Tactile MusicGunther (Gunther & OʼModhrain, 2003) was one of the first researchers to investigate electronically

facilitated tactile music. He used a vibrotactile suit with 13 embedded transducers to explore the

production of tactile music. He suggested that vibrotactile music should contain elements of frequency,

intensity, duration, waveform and space similar to auditory-based music, but did not examine the

impact of tactile music on users/audiences, or the design and development of interfaces that were

explicitly intended for tactile music production. He did, however, remark that composing for the tactile

channel with tools designed for the audio domain was “painstaking” suggesting there is a need for a

tool.

The Emoti-Chair has been used to translate sound from the audio domain to the vibrotactile domain.

Although the skin’s ability to perceive vibration is analogous to the ear in many ways, the vibrotactile

perception of vibration produced from sound is still limited compared to that of the ear. This means that

it is likely not possible to perceive a large portion of the audio signal presented through a vibrotactile

display like The Emoti-Chair. For example, detecting different notes which differ in frequency by small

mounts is likely not possible with a vibrotactile display because of the reduced frequency discrimination

capability of the skin. These limitations of the tactile system may mean that a direct translation of music

from the audio domain into the tactile domain is not ideal or even possible. This suggests that perhaps

more focus should be paid to pure tactile music instead of trying to translate audio music into tactile

music.

One important advantage of the vibrotactile displays is that it is possible to spatially encode the tactile

stimuli. Tactile displays such as the Emoti-Chair can present vibrotactile signals to different locations on

the body which offers another dimension of encoding in addition to frequency and amplitude.

Vibrotactile stimuli of equal intensity and frequency can be presented at different spatial locations,

effectively distinguishing them. Although extremely complex information streams can be delivered

through the auditory channel it is nonetheless still only a two channel system as much of the

multiplexing abilities of the auditory system is due to signal processing functions of the brain. This fact

will have a large influence on how a vibrotactile instrument should be physically constructed and how

the vibrotactile information system will function.

Research MethodsA system that is capable of creating and delivering vibrotactile music will contain three main elements:

1) an input interface that allows users to interact with the system to express their musical intentions; 2)

display sub-system that provides vibrotactile music to audiences; and 3) a processing sub-system that

translates user interactions into the appropriate vibrotactile signals. Each of these sub-systems contains

hardware and software elements and functionality that must be created, integrated together and then

evaluated. I intend to use iterative create and evaluate cycles in order to construct the system. To begin,

I will use the processing and display sub-systems of the Emoti-chair and make modifications as

necessary. Current audio music interfaces including traditional software application such as Garage

Band, Adobe Audition and Pro Tools will be examined for applicability to the interface/interaction

development for my system. My main contributions will be the construction of the input interface or

“instrument”, the examination of the tactile music creation process by users, and the use and evaluation

of the entire system in actual performance situations.

For the iterative design, create and evaluate cycles, traditional usability research methods such as

usability studies with representative users and tasks, and focus groups will be employed to evaluate and

study the input interface (formative evaluation). The first objective in this phase of the process is to

examine ease of use and learning, match with the user’s content creation model and acceptance of the

control interface. In addition, the interface will be revised depending on issues and/or difficulties the

users encounter. Of particular interest in this aspect of the development process is the form factor for a

vibrotactile instrument interface should have in order to best facilitate the performance of vibrotactile

music.

Determining what is required to support the tactile musical creation process based on artist’s needs,

desires and abilities is the second objective in this first phase. Questionnaires, video analysis and

ethnographic techniques such as first hand observation and interviews? will be used to elicit and

understand the creative process. Artists will be commissioned to create vibrotactile compositions which

will be used to evaluate audience reactions in the next phase of the project.

In the next phase of the project, the entire system will be integrated together and then the impact of it

on audiences with be explored. Psychological and psychophysical methods will be employed to

investigate the effect it has on audiences. Psychophysical investigations such as just noticeable

differences will be used to determine optimal placement and distances for the placement of the tactile

feedback on an individual’s skin. User studies will be used to examine the impact of the system on user’s

understanding of the content, and experience of higher level concepts such as mood, rhythm and

contour. Biometric measures such as skin conductance level, heart rate variability, respiration, EMG

(measuring zygomaticus major and corrugator supercilii) will be used to probe the emotional status of

participants exposed to vibrotactile compositions. EEG measures may be considered to further

understand what is happening between the auditory and tactile centers in the brain as a result of

exposure to tactile music. In addition to the biometric measures, self report methods such as

questionnaires, interviews and continuous self report (Lottridge, 2008) will also be used. The goal of the

summative evaluation is to examine the impact of exposure to tactile music on an audience’s

enjoyment, appreciation and acceptance of tactile music. Statistical analyses such as means and

correlation testing will be used to analyze quantitative data collected through psychophysical

experiments, EEG, continuous or discrete self reporting. Thematic analysis will be used for qualitative

data such as that obtained through interview, open ended questionnaires or video recordings.

Proposed TimetableYear Title Milestones Months Description1 Initial Research Phase On Going

Literature Review

1-12

Emoti-Chair Workshops and Concerts with Musicians

8-12 Workshops with composers, singers, instrumentalists with current prototype of the high density vibrotactile display to get high level input on what vibrotactile music should or could be will be conducted. The information gained here will influence the first version of the input prototype

Public events where deaf and hearing Emoti-chair users can offer input into the design of the interface

Use video recording, questionnaires and interviewing techniques to collect data

Thematic analysis will be conducted on the video recordings and interview transcript will statistical analysis will be performed on any questionnaire data

2 Interface Construction / Display Evaluation

Continued Literature Review

25-36

Initial Design and Build Phase

13-19 Begin initial design of Vibrotactile Instrument Interface

Begin construction of Vibrotactile Instrument Interface

Collaborate with musicians and non-musicians in an interactive design process, incorporating feedback into design and construction of the improved prototype vibrotactile interfaces.

Publish findings on musician consultations and initial design of the vibrotactile instrument

Psychophysics 19-25 Complete an experiment that examines the

Experiments aesthetic quality of various types of vibration.

Characteristics such as frequency, amplitude, wave type in isolating and in combination with each other will be tested

Additional psychophysics experiment yet to be determined

Publish psychophysical findingsVibrotactile Music Created by Several Artists (Musicians, dancers, DJ’s)

25-29 Have musicians, composers and other artists create vibrotactile music to be later evaluated.

Traditional human factors methods such as interviews, video recording and questionnaires will be used to collect data during this phase

Give these musicians an emotional goal to strive for such as sad, happy, aggressive, frightening

3 Composition Evaluation

Continued Literature Review

25-36

Conduct emotional self report experiments of vibrotactile compositions

25-31 Compare the reaction to vibrotactile composition to the reaction to audio compositions using self report as the measure of affect

Publish emotional self report findingsConduct psychophysical experiments

25-31 Compare the reaction to vibrotactile composition to the reaction to audio compositions using self report as the measure of affect

Publish emotional self report findingsConduct EEG experiments

25-31 Compare the reaction to vibrotactile composition to the reaction to audio compositions using EEG as a measure of affect

Publish EEG findings4 Write-up / Follow-up Follow up

studies37-42 Additional studies to follow up on previous

studies as to be determinedComplete 37-48

Thesis

References

American Foundation for the Blind. (2010). Refreshable braille display. Retrieved 07/28, 2010, from http://www.afb.org/Section.asp?DocumentID=3652&SectionID=7&SubTopicID=97&TopicID=330

Boothroyd, A., & Cawkwell, S. (1970). Vibrotactile thresholds in pure tone audiometry. Acta Oto-Laryngologica, 69(1), 381-387.

Branje, C., Fels, D. I., Russo, F., & Nespoli, G. (2010). The effect of vibrotactile stimulation on the emotional response to horror films. Unpublished manuscript.

Branje, C., Karam, M., Russo, F., & Fels, D. I. (2009). Enhancing entertainment through a multimodal chair interface. Toronto International Conference – Science and Technology for Humanity, Toronto.

Camurri, A., Hashimoto, S., Ricchetti, M., Ricci, A., Suzuki, K., Trocca, R., et al. (2000). Eyesweb: Toward gesture and affect recognition in interactive dance and music systems. Computer Music Journal, 24(1), 57-69.

Chew, E., & Francois, A. R. J. (2003). MuSA.RT: Music on the spiral array. real-time. MULTIMEDIA '03: Proceedings of the Eleventh ACM International Conference on Multimedia, Berkeley, CA, USA. 448-449.

Chomsky, C. (1986). Analytic study of the tadoma method: Language abilities of three deaf-blind subjects. Journal of Speech and Hearing Research, 29(3), 332.

Cohen, B., & Kirman, J. (1986). Vibrotactile frequency discrimination at short durations. The Journal of General Psychology, 113(2), 179-186.

Foxe, J. J. (2009). Multisensory integration: Frequency tuning of audio-tactile integration. Current Biology, 19(9), R373-R375.

Gelfand, S. A. (1998). Hearing: An introduction to psychological and physiological acoustics (3rd ed., rev. and expanded. ed.). New York: Marcel Dekker.

Gemperle, F., Ota, N., & Siewiorek, D. (2001). Design of a wearable tactile display. Proceedings of the Fifth International Symposium on Wearable Computers, 5-12.

Goff, G. D. (1967). Differential discrimination of frequency of cutaneous mechanical vibration. Journal of Experimental Psychology, 74(2), 294-299.

Gunther, E., & OʼModhrain, S. (2003). Cutaneous grooves: Composing for the sense of touch. Journal of New Music Research, 32(4), 369-381.

Hogema, J. H., De Vries, S. C., Van Erp, J. B. F., & Kiefer, R. J. (2009) A tactile seat for direction coding in car driving: Field evaluation.

Karam, M., Branje, C., Price, E., Russo, F., & Fels, D. I. (2007). Towards a model human cochlea: Exploring vibetracks for crosssmodal audio-tactile displays. Submitted: GI'08: 34th Graphics Interface Conference,

Karam, M., & Fels, D. I. (2008). Designing a model human cochlea: Issues and challenges in crossmodal audio-haptic displays. Proceedings of the 2008 Ambi-Sys Workshop on Haptic User Interfaces in Ambient Media Systems Table of Contents,

Karam, M., Nespoli, G., Russo, F., & Fels, D. I. (2009). Modelling perceptual elements of music in a vibrotactile display for deaf users: A field study. Advances in Computer-Human Interactions, 2009.ACHI'09.Second International Conferences on, , 249-254.

Lottridge, D. (2008). Emotional response as a measure of human performance. CHI '08: CHI '08 Extended Abstracts on Human Factors in Computing Systems, Florence, Italy. 2617-2620.

Mahns, D., Perkins, N., Sahai, V., Robinson, L., & Rowe, M. (2006). Vibrotactile frequency discrimination in human hairy skin. Journal of Neurophysiology, 95(3), 1442-1450.

Nanayakkara, S., Taylor, E., Wyse, L., & Ong, S. H. (2009). An enhanced musical experience for the deaf: Design and evaluation of a music display and a haptic chair. Proceedings of the 27th International Conference on Human Factors in Computing Systems, 337-346.

Ng, K. (2002). Interactive gesture music performance interface. Proceedings of the 2002 Conference on New Interfaces for Musical Expression, 2.

Perovaara, A., & Hamalainen, H. (1981). Vibrotactile thresholds during background vibration of long duration. Scandinavian Journal of Psychology, 22(1), 41-45.

Pongrac, H. (2008). Vibrotactile perception: Examining the coding of vibrations and the just noticeable difference under various conditions. Multimedia Systems, 13(4), 297-307.

Pumphrey, R. (1950). Upper limit of frequency for human hearing. Nature, 166, 571.

Rothenberg, M., & Verrillo, R. (1976). Vibrotactile frequency for encoding speech parameters. The Journal of the Acoustical Society of America, 59, S69.

Sampaio, E., Maris, S., & Bach-y-Rita, P. (2001). Brain plasticity: ‘visual’ acuity of blind persons via the tongue. Brain Research, 908(2), 204-207.

Schürmann, M., Caetano, G., Hlushchuk, Y., Jousmäki, V., & Hari, R. (2006). Touch activates human auditory cortex. NeuroImage, 30(4), 1325-1331.

Smith, S. M., & Williams, G. N. (1997). A visualization of music. Visualization'97., Proceedings, 499-503.

Tommerdahl, M., Hester, K., Felix, E., Hollins, M., Favorov, O., Quibrera, P., et al. (2005). Human vibrotactile frequency discriminative capacity after adaptation to 25 hz or 200 hz stimulation. Brain Research, 1057(1-2), 1-9.

Verrillo, R. T. (1963). Effect of contactor area on the vibrotactile threshold. The Journal of the Acoustical Society of America, 35, 1962.

Von Bekesy, G. (1959). Similarities between hearing and skin sensations. Psychol Rev, 66(1), 1-22.

Winkler, T. (1995). Making motion musical: Gesture mapping strategies for interactive computer music. Proceedings of the 1995 International Computer Music Conference, 261–264.

Yau, J. M., Olenczak, J. B., Dammann, J. F., & Bensmaia, S. J. (2009). Temporal frequency channels are linked across audition and touch. Current Biology, 19(7), 561-566.