Embed Size (px)
Citation preview
WebHexIso: A Customizable Web-based Hexagonal Isomorphic MusicalKeyboard Interface
Hanlin HuUniversity of Regina
David GerhardUniversity of Regina
ABSTRACT
The research of musical isomorphism has been around forhundreds of years. Based on the concept of musical iso-morphism, designers have created many isomorphic keyboard-based instruments. However, there are two major con-cerns: first, most instruments afford only a single patternper interface. Second, because note actuators on isomor-phic instruments tend to be small, the hand of the playercan block the eye-sight when performing. To overcomethese two limitations and to fill up the vacancy of web-based isomorphic interface, in this paper, a novel customiz-able hexagonal isomorphic musical keyboard interface isintroduced. The creation of this interface allows isomor-phic layouts to be explored without the need to downloadsoftware or purchase a controller. additionally, MIDI de-vices may be connected to the web keyboard to display theisomorphic mapping of note being played on a MIDI de-vice, or to produce control signals for a MIDI synthesizer.
1. INTRODUCTION
Since Euler introduced Tonnetz in 1739 [1], mathemati-cian, composers, computer scientists and instrument de-signers are interested in musical isomorphism, which presentsalgorithms of arrangement of musical notes in 2-dimensionalspace so that musical constructs (such as intervals, chordsand melodies, etc.) can be played with same fingeringshape regardless the beginning note [2].
Based on this concept of musical isomorphism, there havebeen a number of keyboard-based interfaces (instruments)being built over the last hundred years. At the beginning,the keys on the keyboard were designed in the shape of arectangle resembling that of the traditional keyboard. Thisis called “square” or “rectilinear” isomorphism. Since thelimitations of the square isomorphism (e.g. the degenerat-ing of layouts, which is passing by some notes in a equaltemperament with particular layouts [2]), interface design-ers prefer hexagonal shaped keys on the keyboard over thelast decade. They can be categorized to hardware and soft-ware. As for hardware, there are the AXiS keyboards,Opal, Manta, Tummer and Rainboard [3]; As for software,there are Hex Play (PC) [4], Musix Pro (iOS) [3], Hex OSCFull (iOS) [5] and Hexiano (Android) [6] .
Copyright: c©2016 Hanlin Hu et al. This is an open-access article dis-tributed under the terms of the Creative Commons Attribution License 3.0Unported, which permits unrestricted use, distribution, and reproductionin any medium, provided the original author and source are credited.
Figure 1: Euler’s Tonnetz
In addition to the limitation of square isomorphism, thereare two main constraints of isomorphic keyboard design:firstly, most of the interfaces can provide only one par-ticular isomorphic layout, which means the layout is notchangeable or in the other word, the device is not customiz-able, therefore the performers or composers are “locked in”to a single layout. Although learning a single layout maybe desirable in some circumstances, one of the main advan-tages of isomorphic keyboard design is the fact that manydifferent layouts with different harmonic relationships areavailable in the same framework. For this reason, we thinkthat the limitation of a single isomorphic layout on mosthardware instruments is a significant disadvantage whichshould be addressed.
Secondly, when performers or composers play on the phys-ical appearance of an isomorphic keyboard, their handscan easily block the display so that the name or colour ofkeys is not easy to see. For traditional single-layout instru-ments, this is not an issue because performers memorizeand internalize key-actuator positioning, but for reconfig-urable digital instruments, where colour may be the onlyindication of the function or note of an actuator, the prob-lem of actuator occlusion may be considered significant.One possible solution would be to separate the display ofthe reconfigurable keyboard from the actuators. Althoughthis separation may be considered a step back in terms ofusability and control/display integration, it may serve to fa-cilitate exploration of this new class of reconfigurable in-terfaces. Further, although it is possible to plug a MIDIdevice into an iPad and play isomorphic software such asMusix Pro, practically speaking, because many MIDI de-vices draw power over the USB port, it is sometimes im-practical or impossible to connect a MIDI device to an iPador other external display.
In this paper, we present a novel web-based hexagonalisomorphic musical keyboard interface, which is customiz-able, scalable, and MIDI enabled. It can be used as a soft-
ware instrument, a composition device, an educational toolof musical isomorphism, and an assistive screen interfacefor performance.
2. MUSICAL ISOMORPHISMS
The word “isomorphism” has the prefix “iso,” which means“equal,” and an affix term “morph,” which means “shape.”Isomorphism, then, refers to the property of having an iden-tical shape or form. The concept of isomorphism appliedto music notations is that for an isomorphic arrangementof notes, any musical construct (such as an interval, chord,or melody) has the same shape regardless of the root pitchof the construct. The pattern of constructs should be con-sistent in the relationship of its representation, both in po-sition and tuning. Corresponding to transposition invari-ance, tuning invariance 1 is another requirement of musicalisomorphism. Most modern musical instruments (like thepiano and guitar) are not isomorphic. The guitar in stan-dard tuning uses Perfect Fourth intervals between strings,except for the B string which is a Major Third from the Gstring below it. Because of this different interval for onepair of strings, the guitar is not isomorphic.
Isomorphic instruments are musical hardware which canplay the same musical patterns regardless of the startingpitch. Isomorphic arrangements of musical notes intro-duce a number of benefits to performers [7]. The mostnotable of these is that fingerings are identical in all musi-cal keys, making learning and performing easier. Moderninstruments which display isomorphism include stringedinstruments such as violin, viola, cello, and string bass [3].It should be noted in this case that although the relative po-sition of intervals is the same for every note on the finger-board of a violin, the relative size of each note zone maychange, with the notes being smaller as you move closerto the bridge of the instrument. The traditional piano key-board is not isomorphic since it includes seven major notesand five minor notes as a 7-5 pattern.
Thanks to this 7-5 pattern design, performers can eas-ily distinguish in-scale and out-of-scale notes by binarycolours, but the performer must remember which whitenotes and which black notes are in the scale in which theyare performing. Because the piano is not isomorphic, dif-ferent fingerings and patterns are required when perform-ers play intervals and chords in different keys. This is oneof the reasons that the piano is difficult to learn: each mu-sical construct (e.g. the Major scale) must be learned sep-arately for each key (e.g. C Major, G Major, F Major etc.)
The first physical appearance of an isomorphic layoutwas decided by Hungarian pianist Paul von Janko in 1882 [8].The Janko keyboard shown in Fig. 2 was originally de-signed for pianists who have small hands that can causefingering difficulties when stretching to reach the ninth in-terval, or even the octave, on a traditional keyboard. Bysetting every second key into the upper row and shapingall keys identically, the size of the keyboard in the hori-zontal direction shrinks by about half within one octave.After making three duplicates, the performer can play in-tervals or chords by putting the fingers up or down to reach
1 Tuning invariance: where all constructs must have identical geomet-ric shape of the continuum
C D E F# G# A#
C# D# F G A B
C D E F# G# A#
C# D# F G A B
C D E F# G# A#
C# D# F G A B
C D E F# G# A#
C# D# F G A B
Octave 2
Octave 1
...
...
...
...
Figure 2: Janko Keyboard tessellation
C E G
D
F#
A
C D E F# G# A#
C# D# F G A B
C D E F# G# A#
C# D# F G A B
(a) C-Major on Piano
(c) D-Major on Piano
(b) C-Major on Janko
(d) D-Major on Janko
Figure 3: Isomorphism in the Janko keyboard as comparedto polymorphism in the piano keyboard.
the desired notes. Each vertical column of keys to the ad-jacent column are a semitone away, and the horizontal rowof keys to the adjacent row is a whole step away. This de-sign never became popular since performers are not con-vinced of the benefits of this keyboard and they would in-stead have to spend more time learning a new system [9].
This arrangement of notes on the Janko keyboard is iso-morphic because a musical construct has the same shaperegardless of key. Consider a Major triad (Fig. 3). the C-Major triad has the notes C-E-G, while the D-Major triadhas the notes D-F]-A. On the piano keyboard, these tri-ads have different shapes, but on the Janko keyboard (andon any isomorphic keyboard) these triads have the sameshape. In fact, every major triad has the same shape on anisomorphic keyboard.
There are three reasons why hexagonal shaped keys arebetter than rectangular shaped keys:
1. The rectangular shaped keys of an isomorphic key-board does not meet the requirement of transposition in-variance perfectly, because the Euclidean distance betweentwo keys is not identical. For example, in Fig. 2, the dis-tance between C and D in horizontal direction does notequal the distance between C] and D in vertical direction.However, the regular hexagonal shaped keys make the dis-tances identical.
2. It is easy to find three adjacent keys to close into a tri-angle on Janko layout, such as C-D-C]. This relationshiphas been modelled with equilateral triangle by Riemann astriangular Tonnetz in Fig. 4, which is explored from Ton-netz [11]. Since the regular hexagon is the dual of the equi-lateral triangle, for each key, the hexagonal shape is goodto present this relationship.
Figure 4: Tonnetz as regularized and extended by Riemannand others [10]
3. Each hexagon has six adjacent hexagons, while eachsquare only has four adjacent squares. More adjacent notesmeans more harmonic connects and a more compact notearrangement for the same number of notes.
3. WEBHEXISO
WebHexIso is a novel customizable web-based musical iso-morphic interface. There are few existing web-based mu-sical isomorphic interface available online such as “Can-vasKeyboard” [12]. However, the “CanvasBoard” is un-customizable interface with a particular layout — Wicki-Hayden [13].
3.1 Basic Design
Button Layer (Invisible)
Detect Behaviours Activate Animations Interaction with users
Render a particular layout
Run a synthesizer, which is created by Web Audio API
Middle Layer (Visible)
Top Layer (Invisible)
Figure 5: Over-layer design
The basic design of the interface is based on over-layerstrategy. There are three layers as shown in Fig. 5: The but-ton layer, which is invisible to users, is used for running asynthesizer on the background. The synthesizer is createdby Web Audio API [14], which is able to produce manydifferent musical instrument sounds. The middle layer,which is visible to users, is used for rendering a particular
isomorphic layout chosen by users. The top layer, whichis invisible to users, is used for bundle listeners to detectusers’ behaviours (navigation, click and touch). Once lis-teners detect a click or touch behaviour, animation of theclicked tile will be activated and synthesizer will to calledto sound a note.
3.2 Features
(a) Gerhard Layout
(b) Park Layout
Figure 6: Isomorphic layouts rendered in the middle layerof WebHexIso. Root note (C) is red, and notes that wouldnormally be black on a piano are marked in green.
The interface provides options for selecting typical lay-outs (Harmonic table, Wicki-Hayden, Janko, etc.), and ad-ditionally, users can define their own custom layouts bychoosing the musical interval in horizontal and vertical di-rections.
Users can also switch the direction of the layouts hori-zontally (the Zigzag direction of hexagonal grid [15] facesnorth) or vertically (the Armchair direction of hexagonalgrid [15] faces north). Users can choose any note for thetonic, and can choose to colour notes based on any scale,key, or mode. The colour of layouts, the size of the keysand the type of synthesizers are manageable.
3.3 Scalable Multi-touch API and Web MIDI API
Beyond the basic design, by using Multi-touch API withtouch events functions to control the multi-touch behaviour,the interface provides an opportunity to be used as a mo-bile application. The WebHexIso and other existing mobile
apps such as Musix Pro have similar functionality, whenthe web-based interface is opened in a modern browser.
Furthermore, the Web MIDI API [16] allows MIDI musi-cal controllers transfer data by USB. Once a musical con-troller plugged-in while the WebHexIso is open, the Web-HexIso will detect and identify the controller and receiveMIDI notes data from the controller. Fig. 7 shows whenWebHexIso detects the AXiS-49 is plugged-in, the corre-sponding layout is shown on the screen. Moreover, theWeb MIDI API also allows data transfer from WebHex-Iso to MIDI devices, which allows WebHexIso activatesplugged-in slave MIDI device and either play notes on asynthesizer, or send note event and layout patterns to it.
Figure 7: The interface shows harmonic table layout on thescreen when AXiS-49 is plugged-in
3.4 Limitation
Depending on the browser and computer being used, Web-HexIso may have increased latency. On a modern browserand recent computer, the latency should not be noticeable,but on slower systems, if WebHexIso performs slowly tothe point that a noticeable lag is present between the ac-tivation of a key and the sounding of a note, it would bepossible to disable some features (like multitouch) to in-crease performance at the cost of functionality.
4. CONCLUSION AND FUTURE WORK
A novel customizable web-based hexagonal isomorphic mu-sical keyboard interface has been introduced. Users can de-fine or select different isomorphic keyboard by themselves.This interface is online and free so that more people couldhave the chance to access the isomorphic keyboards. Byusing multi-touch API, the web-based interface can imple-ment behaviours as a mobile application. It also could beused as an assisting screen of isomorphic layouts for per-formance when a MIDI controller device, such as AXiS-49, is plugged-in.
In future, more MIDI devices will be recognized by Web-HexIso when their MIDI names are built into the database.Furthermore, a user-interface study of can be conducted.Based on this system, the composers can find more iso-morphic layout patterns, which will benefit performance.
5. REFERENCES
[1] L. Euler, “De Harmoniae Veris Principiis per Specu-lum Musicum Repraesentatis,” in Novi commentariiacademiae scientiarum Petropolitanae, St. Petersburg,1774, pp. 330 – 353.
[2] B. Park and D. Gerhard, “Discrete Isomorphic Com-pleteness and a Unified Isomorphic Layout Format,” inProceedings of the Sound and Music Computing Con-ference, Stockholm, Sweden, 2013.
[3] ——, “Rainboard and Musix: Building Dynamic Iso-morphic Interfaces,” in Proceedings of 13th Interna-tional Conference on New Interfaces for Musical Ex-pression, 05 2013.
[4] A. Milne, A. Xambo, R. Laney, D. Sharp, A. Prechtl,and S. Holland, “Hex Player — a virtual musicalcontroller,” in Proceedings of the 11th InternationalConference on New Interfaces for Musical Expres-sion(NIME), Oslo, Norway, 2011, pp. 244 – 247.
[5] SkyLight and Denryoku, “Hex OSC Full,” http://www.sky-light.jp/hex/, Online.
[6] D. Randolph, J. Haigh, and S. Larroque, “Hexiano,”https://github.com/lrq3000/hexiano/, Online.
[7] V. Goudard, H. Genevois, and L. Feugere, “Onthe Playing of Monodic Pitch in Digital Music In-struments,” in Proceedings of the 40th InternationalComputer Music Conference (ICMC) joint with the11th Sound and Music Computing conference (SMC),Athens, Sep 2014, p. 1418.
[8] P. von Janko, “Neuerung an der unter No 25282 paten-tirten Kalviatur,” German patent 25282-1885.
[9] K. Naragon, “The Janko Keyboard,” typescript, pp.140–142, 1977.
[10] D. Tymoczko, “Geometrical Methods in Recent Mu-sic Theory,” MTO: a Journal of the Society for MusicTheory, vol. 1, 2010.
[11] H. Riemann, “Ideen zu einer “Lehre von den Ton-vorstellungen”,” in Jahrbuch der Bibliothek Peters,1914 – 1915, pp. 21 – 22.
[12] S. Little, “CanvasBoard,” https://github.com/uber5001/CanvasKeyboard, online. Accessed on 8-Jan-2016.
[13] K. Wicki, “Tastatur fur Musikinstrumente,” Swisspatent 13329-1896.
[14] W3C, “Web Audio API,” https://www.w3.org/TR/webaudio/, Online. Draft 8-Dec-2015.
[15] H. Hu, B. Park, and D. Gerhard, “On the Musical Op-potunities of Cylindrical Hexagonal Lattices: MappingFlat Isomorphisms Onto Nanotube Structure,” in Pro-ceedings of the 41th. International Computer MusicConference, Denton, Texas, 2015, pp. 388–391.
[16] W3C, “Web MIDI API,” https://www.w3.org/TR/webmidi/, Online. Draft 17-March-2015.
