Audio Engineering Society Convention Paper 8139a dancer to exert control of an accompanying audio environment. Typically, a computer receives infor-mation of the dancer’s movement

Audio Engineering Society

Convention PaperPresented at the 128th Convention2010 May 22–25 London, UK

The papers at this Convention have been selected on the basis of a submitted abstract and extended precis that havebeen peer reviewed by at least two qualified anonymous reviewers. This convention paper has been reproduced fromthe author’s advance manuscript, without editing, corrections, or consideration by the Review Board. The AES takesno responsibility for the contents. Additional papers may be obtained by sending request and remittance to Audio

Engineering Society, 60 East 42nd Street, New York, New York 10165-2520, USA; also see www.aes.org. All rightsreserved. Reproduction of this paper, or any portion thereof, is not permitted without direct permission from theJournal of the Audio Engineering Society.

The Serendiptichord: A Wearable InstrumentFor Contemporary Dance Performance

Tim Murray-Browne1, Di Mainstone, Nick Bryan-Kinns1 and Mark D. Plumbley1

1Queen Mary University of London, Centre for Digital Music, London, UK

Correspondence should be addressed to Tim Murray-Browne or Di Mainstone([email protected], [email protected])

ABSTRACTWe describe a novel musical instrument designed for use in contemporary dance performance. This instru-ment, the Serendiptichord, takes the form of a headpiece plus associated pods which sense movements of thedancer, together with associated audio processing software driven by the sensors. Movements such as trans-lating the pods or shaking the trunk of the headpiece cause selection and modification of sampled sounds.We discuss how we have closely integrated physical form, sensor choice and positioning and software to avoidissues which otherwise arise with disconnection of the innate physical link between action and sound, leadingto an instrument that non-musicians (in this case, dancers) are able to enjoy using immediately.

1. INTRODUCTION

Contemporary dance is a highly expressive art formwith a strong visual impact [1]. It encompassesa range of techniques, often allowing for a greatamount of free improvisation and exploratory move-ment. In augmenting a performance we sought totranslate a dancer’s movement into sound in an intu-itive manner whilst retaining their expressive inten-tion. Furthermore, we aimed not only to interpretmovement but to provoke it and create a dialoguebetween dancer and sound environment, and for all

of these aspects to be clear to an audience. Thisdesire for clear bidirectional feedback led early on tothe creation of an object of interaction visible to theaudience—a wearable instrument.

We first provide a brief review of similar work (§2),followed by a discussion of some issues surround-ing new musical interfaces that informed the designprocess (§3). There follows a technical descriptionof the instrument (§4) and how it is used in perfor-mance (§5) which will be followed by a discussionon how both our motivations and the development

8139

Murray-Browne et al. The Serendiptichord: A Wearable Instrument for Contemporary Dance Performance

process led this design (§6). We conclude with asummary of the key lessons that have come out ofthis project and some future avenues of research ithas presented (§7).

2. RELATED WORK

There have been a number of approaches in allowinga dancer to exert control of an accompanying audioenvironment. Typically, a computer receives infor-mation of the dancer’s movement either through avideo-feed (e.g. [2, 3, 4]), accelerometers attached tothe body (e.g. [5, 6]), or motion capture1 (e.g. [7])which are then mapped to input parameters of awide variety of bespoke music systems.

Castellano et al. [4] use the EyesWeb [8] platformto map an emotional interpretation of the user’smovement into both visual feedback and parame-ters such as tempo, loudness and articulation, whichcontrol the expressive performance of a prewrittenpiece of music. This high-level control is in contrastto Morales-Manzanares’ Tŕıo de Cuatro [9], wheremovement is mapped directly to granular synthesisparameters as well as notes produced by ‘touching’particular regions.

Direct mapping to synthesis parameters are alsoused in the interactive dance piece Lucidity [10]alongside more complex genetic algorithms drivenby the level of dance activity. In this work, mo-tion capture data is analysed to produce measuressuch as the correlation between the movements of adancer’s limbs and their position on the stage.

CoIN, a video-based work by Ng [2] making use ofthe Music via Motion [2] framework uses a mixtureof mapping strategies, including a direct one-to-onemapping from positional data onto simple musicalinput such as pitch and volume, and colour detec-tion triggering sound effects. Ng notes that the di-rect mapping has the advantage of allowing the audi-ence to easily relate movement to sound but becomes‘tiresome and uninspiring’ if used constantly. He ad-dresses this with the use of prewritten backgroundmusic (that alters certain expressive factors based onthe input) with timed pauses for the dancers to use

1Motion capture requires reflective markers to be attachedto the performer which are then recorded by a number ofinfrared cameras and converted into a 3D animation.

the direct mapping. We refer to this performance-scale organisation of themes, actions and other as-pects as the narrative of a piece, which in this caseis determined prior to the performance.

This illustrates the potential dichotomy betweenhighly reactive mapping strategies which are intu-itive and clear both to dancer and audience butlack expressive power and dialogical strategies whichhave a longer-term impact but are more complexand seem less controllable [3]. Camurri et al. [3]propose a multimodal framework where an input isanalysed on a number of layers. The first layer con-sists of a direct and reactive mapping between low-level feature analysis synthesizer parameters. Thismapping is then adjusted by a second layer, whichdraws upon high-level (and usually long-term) fea-tures such as basic emotions. A third layer then maythen alter either of the lower layers and is guided byan overall performance measure, such as the audi-ence’s engagement. This use of dialogical strategiesallows the dancers greater long-term control overthe progression of their performance. Consequentlymore freedom can be granted without the problem ofmonotony and a more interactive narrative may beused. Camurri et al. have implemented this frame-work within the EyesWeb [8] framework to draw ondraw on lower and higher levels of video-based ges-ture recognition.

3. SOME DESIGN CONSIDERATIONS

Interactive music systems are typically reasonedabout as a mapping from controller or sensor inputparameters to synthesis parameters [11]. This mod-ular approach provides creators with a great dealof freedom as generic well-packaged techniques (e.g.gesture recognition [12]) can be routed to audio-generating parameters as easily as traditional con-trollers such as the MIDI keyboard. But the dis-connection of the innate physical link between ac-tion and sound in electronic instruments has draw-backs, most notably the resulting lack of ‘feel’ [13].Drummond [14, p. 132] presents this as a challengeto ‘create convincing mapping metaphors, balanc-ing responsiveness, control and repeatability withvariability, complexity and the serendipitous.’. How-ever, Jordà [13, p. 322] demands a more holistic ap-proach:

It becomes hard – or even impossible – to

AES 128th Convention, London, UK, 2010 May 22–25

Page 2 of 8


Fig. 1: Heidi Buehler rehearsing before a perfor-mance at Berkeley Art Museum, Berkeley, USA. Onher shoulder’s sits the headpiece of the Serendipti-chord with the trunk extending over and in front ofher head. In her left and right hands are the left andright pods respectively. Accelerometers are embed-ded within the pods, behind her neck and at the endof the trunk. (Photo: Tim Murray-Browne)

design highly sophisticated control inter-faces without a profound prior knowledgeof how the sound or music generators willwork. . . . They can be inventive or adven-turous, but how can coherence be guaran-teed if it cannot be anticipated what thecontrollers are going to control?

With respect to systems responding to full-bodymovement, Antle et al. [15, p. 74] argue for theuse of embodied metaphors that ‘preserve structural

isomorphisms between lived experience and the tar-get domain’ with the suggestion that dancers be in-volved during the design process. By drawing onmodes of action that relate to how our bodies in-teract with the world in everyday life, an intuitivesystem can be built with a natural feel.

Another issue that arises from decoupling the soundsource from the instrument is a lack of physical feed-back, which can result in a sense of restricted con-trol [16]. Both the user and audience need percep-tual feedback allowing them to learn how actionsrelate to consequences, especially when using uncon-ventional or invisible inputs. This issue is of greaterprominence in dance-based instruments as the per-former often has neither visual nor physical feedbackfrom the system. Furthermore, we argue that as wellas understanding the connection between action andsound, the audience must believe that it is the dancerresponsible for the music. This makes it difficultto include scope for subtle or delayed aspects to amapping—the reactive/dialogical trade-off discussedin §2.

Finally, there is the question of how to maintainmusical coherency whilst providing artistic free-dom [17]. In §2 we saw examples of introducingpre-composed music ([2, 4]) as well as compositionaldecisions that avoided the difficult-to-navigate tonaland harmonic spaces ([9, 10]) as potential ways ofavoiding undesirable musical outputs. Camurri etal. [18, p. 196] address this issue by drawing a dis-tinction between autonomy and expressive auton-omy with the latter regarding the amount of free-dom that is provided specifically ‘to take decisionsabout the appropriate expressive content in a givenmoment and about the way to convey it’. Dancehas a huge expressive capability and we would liketo transform as much as possible of this into corre-spondingly expressively music.

4. THE SERENDIPTICHORD

The Serendiptichord takes the form of a headpiecethat sits on the shoulders with a ‘trunk’ that ex-tends over and in front of the head and gently swingswith the movement of the performer (see Figure 1).Two handheld ‘pods’ may be clipped to cords ex-tending from the headpiece or removed and attachedto another part of the body. Three-dimensional ac-celerometers are embedded within each pod, behind


Page 3 of 8


the neck and within the trunk. The output of eachsensor is smoothed and converted into orientationdata.

In order to provide the dancer with a great dealof expressive autonomy whilst still allowing musicalcoherence to be ensured, sound is generated froma bank of composer-created sound objects which intheir simplest form are just samples. Each sound ob-ject is randomly assigned to a specific orientation ofboth the left pod and neck sensors—the noisemak-ers—and is triggered when the orientation of thesesensors approaches that of the sample. This map-ping draws on the embodied metaphor of a percus-sive instrument as the samples are effectively ‘hit’into. It is explained in more detail in §4.1.This intuitive model provides a reactive mappingthat is designed to be easy for both dancer and au-dience to understand. But in order to allow boththe dancer to introduce more structure to their per-formance and the composer to develop their musi-cal ideas it is augmented with the more dialogicalcontrol of intensity. Each sound object is providedwith a continuous intensity value which they com-poser may interpret as they please. With the presentbank of sounds, increasing the intensity of one ob-ject causes it to develop in a unique way, as wellas making it louder and warmer through the use ofdigital signal processing effects.

Shaking the right pod—the intensifier—within theboundaries of a sound object causes a rapid increasein its intensity value, which will then decay to zeroover a period of time. As a composer may choose toimplement a subtle or deferred response to intensifi-cation, this effect is supplemented by passing audiofrom the object through a delay for a few seconds.

In contrast to the other three sensors, output fromthe trunk is directly mapped to the parameters of afrequency shifting effect applied to the master chan-nel. The trunk typically follows the motion of thedancer’s neck making it an effective tool to translateexpressive movement into expressive sound. How-ever, its physical construction causes it to gentlyoscillate after a sudden movement. The resultingvibrato effect is a subtle but essential connection be-tween the physical nature of the instrument and theaudio output. As the trunk is visible to the dancer, italso provides them (and the audience) with instanta-neous perceptual feedback of the system’s sensitivity

Fig. 2: An overview of the mapping used within theSerendiptichord.

to motion. Furthermore, through feeling the trunkfollow their movement the dancer is provided witha constant physical connection between themselvesand the instrument.

An overview of the mapping strategy is shown inFigure 2.

4.1. The percussive mapping model

The percussive mapping model draws on an embod-ied metaphor of striking an object to provide a rela-tionship between body position and musical outputthat is intuitive to both dancer and audience andtranslates a given bodily gesture into a consistentsequence of sounds.

In more detail, sensor inputs from both noisemak-ers are concatenated to produce a position withinthe combined orientation space. Within this spaceeach sound object is assigned a random position.A sample is triggered when the combined input


Page 4 of 8


Fig. 3: Diagram showing a dancer’s trajectory (blueline) through the controller space and within theboundary of a sound object with centre O. The sam-ple is triggered once from entering the boundary andagain from returning back towards the centre hav-ing started moving away. The dotted line shows thestopping boundary.

comes within a (Euclidean) distance d of its respec-tive sound object, with its volume determined bythe velocity of movement towards the sound object.Thus sound objects may be seen as spheres of radiusd within the controller space that the noisemakers‘hit’. When the noisemakers move away from thesound object the sample is stopped when a bound-ary slightly greater than d is crossed to provide aslight sense of ‘stickiness’ to the objects and preventrapid triggering around the boundary. In initial test-ing, however, dancers found it difficult to predictablytrigger a sample twice in a row as they had no wayof explicitly knowing where this boundary was andneeded to escape it and return to retrigger the sound.This was addressed by considering the speed of ap-proach towards or away from the sound object. Byrequiring a minimum threshold of speed towards totrigger a sample and speed away to allow a sample tobe retriggered (Figure 3) the interface became morepredictable and controllable.

One aspect of acoustic instruments that this map-ping strategy naturally emulates is the relationshipbetween the energy put in and how much sound

comes out [19]. For example, a wind instrumentneeds to be blown continuously in order to main-tain a tone, with greater pressure leading to morevolume. When using the percussive mapping modelsound objects are only triggered through movement,with the volume of samples increasing with the speedof motion. However we exaggerate this effect for fur-ther expressive and dramatic effect by cutting thevolume whenever the instrument is completely stillfor more than a second.

5. PERFORMANCES

The Serendiptichord was first shown in performanceat the ACM Creativity & Cognition Conference 2009in Berkeley, USA (Figure 4). Further performanceshave taken place in London at the Kinetica Art Fair2010, an exhibition showcasing new art that incor-porates motion or technology, and the Barbican aspart of the event Swap Meet. When showing in anew place we will typically organise to meet and re-hearse with a dancer for around three hours the daybefore the performance in order to allow them tobecome accustomed to the instrument and its ca-pabilities. During this time we will also consider askeletal narrative around which they may structurean improvised performance. Typically, this will lastbetween eight and 15 minutes and follow a structureof gradually introducing different aspects of the in-strument whilst building up the amount of energyand tension to a chaotic climax.

6. DISCUSSION

The construction of the Serendiptichord was a closemultidisciplinary collaboration with short develop-ment cycles. At every step, we considered howeach individual component worked with the whole:What actions come naturally when wearing the in-strument? How should they sound? What actionshould be done to produce this sound? How can theshape of the instrument accommodate that action?and so on. Through inviting dancers to test theSerendiptichord throughout its development we haveobserved that as development has progressed underthis approach newcomers have tended to be morecomfortable and quicker to explore its capabilitieswithout extensive instruction.

This method of interaction design has consequentlyallowed us to introduce depth to the instrument


Page 5 of 8


Fig. 4: Heidi Buehler performing with the Serendip-tichord at the ACM Creativity & Cognition Confer-ence 2009, Berkeley, USA. Here, the left-hand podhas been attached to the right leg. (Photo: DeirdreMcCarthy)

through ambiguity. Whilst those using the instru-ment quickly develop a knowledge of how it works,its ambiguous shape does not propose a ‘correct’ wayto use it, and we have found that each dancer bringsa different interpretation.

The combination of both reactive and dialogi-cal interaction mechanisms follow a similar ap-proach to [3], with two simple but powerful directmappings—the percussive mapping model and thesway of the trunk—being developed by the morecomplex mapping used to intensify samples. Thiscombination makes it indisputable to the audiencethat the dancer has the expressive autonomy and isresponsible for the sounds, whilst still providing themeans to introduce variety and a narrative struc-ture to the performance. The multilayered mappingapproach is thus well suited to interactive systemsdesigned to be performed to an audience.

The mapping of the sensor within the trunk was de-veloped through early experimenting whilst wearingthe instrument, which showed that its physical con-struction causes it to oscillate gently after a suddenmovement. Furthermore, when the dancer is mov-ing expressively the trunk swings exaggeratedly fol-

lowing their movement in a manner visible to bothdancer and audience. After discussing what possi-ble effect both felt intuitive and complemented otheraspects of the sound generation we chose to mapthe trunk orientation directly to a frequency shift-ing effect. As well as providing an instantaneous andvisible connection between the physical and soniccomponents of the instrument, this reactive map-ping gives the Serendiptichord a unique and charac-teristic sound. It would, however, behave quite dif-ferently if used with another controller, illustratinghow a closely integrated development of controllerand sound generator can allow the development ofpowerful interaction mechanisms that draw on theaesthetics of both components.

Finally, developing the both the audience’s anddancer’s understanding of the relationship betweenmovement and sound was greatly supported by us-ing an instrument with a visually striking physicalform with a strong stage presence. Rather than be-ing a ‘necessary evil’ hidden from the audience, theSerendiptichord is a character within the dancer’sperformance allowing their relationship with it tobe one of interaction rather than reaction.

7. CONCLUSION

In this paper we outlined a number of issues facedwhen designing an interactive music system for usein dance performance. We addressed these when cre-ating the Serendiptichord by following a holistic ap-proach to design, construction and performance withthe physical, software and musical aspects of thesystem being developed simultaneously with regularfeedback from dancers. This resulted in a wearableinstrument who’s physical form naturally suggestshow it may be used to produce a musical output andconsequently lead the dancer towards the serendipi-tous.

Many instruments are designed to be performed infront of an audience, especially those that are dance-driven, and so the visual impact of a piece shouldbe considered simultaneously with its musical inten-tion. Furthermore, the visual and tactile aspectsare closely entangled with both the interaction de-sign and musical output. This not only informs theperformer but also the audience, who must under-stand how the instrument works in order to believe


Page 6 of 8


that the dancer retains expressive autonomy ratherthan the designer.

Our performance made use of a preconceived narra-tive with a large scope for free improvisation overthe top. Other performances have made use of afixed narrative provided by background music [2] ora potentially interactive narrative through dialogi-cal mapping strategies [3]. Further user studies areneeded to determine the effectiveness of these differ-ent approaches both with respect to both audienceand performer. If the music follows a narrative, isthe performer’s experienced enhanced by knowingthey are responsible? Does the audience benefit?And if so, how can a composer provide the performerwith expressive autonomy over the narrative whilstmaintaining their musical vision?

8. ACKNOWLEDGEMENTS

This Serendiptichord was commissioned by the Cen-tre for Digital Music under the Platform Grant (EP-SRC grant EP/E045235/1) for the ACM Creativityand Cognition Conference 2009, produced by Big-Dog Interactive and supported by the InteractionalSound and Music Group. Special thanks to RachelLamb, Niamh O’Connor, Judy Zhang and StaceyGrant for creative assistance and Vesselin Iordanovfor technical assistance. This research has also beensupported by an EPSRC Doctoral Training Account.

9. REFERENCES

[1] A. Albright, Choreographing difference: Thebody and identity in contemporary dance. Wes-leyan University Press, 1997.

[2] K. Ng, “Sensing and mapping for interactiveperformance,” Organised Sound, vol. 7, no. 02,pp. 191–200, 2003.

[3] A. Camurri, B. Mazzarino, and G. Volpe,“Expressive gestural control of sound and vi-sual output in multimodal interactive systems,”in Proceedings of the International ConferenceSound and Music Computing, 2004.

[4] G. Castellano, R. Bresin, A. Camurri, andG. Volpe, “Expressive control of music and vi-sual media by full-body movement,” in Proceed-ings of the 7th international conference on New

interfaces for musical expression, p. 391, ACM,2007.

[5] C. Park, P. H. Chou, and Y. Sun, “A wear-able wireless sensor platform for interactive artperformance,” in Proceedings of the Fourth An-nual IEEE International Conference on Per-vasive Computing and Communications (Per-Com), pp. 13–17, Citeseer, 2006.

[6] R. Aylward and J. Paradiso, “Sensemble: awireless, compact, multi-user sensor system forinteractive dance,” in Proceedings of the 2006conference on New Interfaces for Musical Ex-pression, pp. 134–139, IRCAM—Centre Pom-pidou, 2006.

[7] F. Bevilacqua, L. Naugle, and I. Valverde,“Virtual dance and music environment usingmotion capture,” in Proceedings of the IEEE-Multimedia Technology And Applications Con-ference, Irvine CA, Citeseer, 2001.

[8] A. Camurri, P. Coletta, A. Massari, B. Maz-zarino, M. Peri, M. Ricchetti, A. Ricci, andG. Volpe, “Toward real-time multimodal pro-cessing: EyesWeb 4.0,” in Proc. AISB 2004Convention: Motion, Emotion and Cognition,Leeds, UK, 2004.

[9] R. Morales-Manzanares, E. F. Morales, R. Dan-nenberg, and J. Berger, “Sicib: An interactivemusic composition system using body move-ments,” Computer Music Journal, vol. 25, no. 2,pp. 25–36, 2001.

[10] J. James, T. Ingalls, G. Qian, L. Olsen,D. Whiteley, S. Wong, and T. Rikakis,“Movement-based interactive dance perfor-mance,” in Proceedings of the 14th annual ACMinternational conference on Multimedia, p. 480,ACM, 2006.

[11] E. Miranda, R. Kirk, and M. Wanderley, Newdigital musical instruments: control and inter-action beyond the keyboard. AR Editions, Inc.,2006.

[12] M. Kolsch, Vision based hand gesture inter-faces for wearable computing and virtual envi-ronments. PhD thesis, University of California,Santa Barbara, 2004.


Page 7 of 8


[13] S. Jordà, “Instruments and players: Somethoughts on digital lutherie,” Journal of NewMusic Research, vol. 33, no. 3, pp. 321–341,2004.

[14] J. Drummond, “Understanding interactive sys-tems,” Organised Sound, vol. 14, no. 02,pp. 124–133, 2009.

[15] A. Antle, G. Corness, and M. Droumeva,“What the body knows: Exploring the benefitsof embodied metaphors in hybrid physical digi-tal environments,” Interacting with Computers,vol. 21, no. 1-2, pp. 66–75, 2009.

[16] B. Bongers, “Physical interfaces in the elec-tronic arts,” Trends in Gestural Control of Mu-sic, pp. 41–70, 2000.

[17] G. Widmer, D. Rocchesso, V. Välimäki,C. Erkut, F. Gouyon, D. Pressnitzer, H. Pentti-nen, P. Polotti, and G. Volpe, “Sound and mu-sic computing: Research trends and some keyissues,” Journal of New Music Research, vol. 36,no. 3, pp. 169–184, 2007.

[18] A. Camurri, P. Coletta, M. Ricchetti, andG. Volpe, “Expressiveness and physicality ininteraction,” Journal of New Music Research,vol. 29, no. 3, pp. 187–198, 2000.

[19] A. Hunt, M. Wanderley, and M. Paradis, “Theimportance of parameter mapping in electronicinstrument design,” Journal of New Music Re-search, vol. 32, no. 4, pp. 429–440, 2003.


Page 8 of 8

Documents

Audio Engineering Society Convention Paper 8139a dancer to exert control of an accompanying audio environment. Typically, a computer receives infor-mation of the dancer’s movement