PianoText: Transferring MusicalExpertise to Text Entry

Anna Maria FeitMax Planck Institute forInformatics66123 Saarbruecken, Germanyafeit@mpi-inf.mpg.de

Antti OulasvirtaMax Planck Institute forInformatics66123 Saarbruecken, Germanyoantti@mpi-inf.mpg.de

Figure 1: PianoText is a proof-of-concept that exemplifieshow musical expertise can be transferred to an HCI task.PianoText is a text entry method using the piano keyboard,developed with an algorithm that optimizes the mapping ofletters to notes. The algorithm exploits well-practiced motorpatterns in piano playing.

Copyright is held by the author/owner(s).CHI 2013 Extended Abstracts, April 27–May 2, 2013, Paris,France.ACM 978-1-4503-1952-2/13/04.

AbstractWe present PianoText, a text entry method based on apiano keyboard with an optimized mapping between notesand chords of music to letters of the English language.PianoText exemplifies the idea of transferring musicalexpertise to a text entry task by computationallysearching for mappings between frequent motor patternswhile considering their n-gram frequency distributions andrespecting constraints affecting the playability of music. Inthe Interactivity session, audience members with pianoskills can transcribe text with PianoText, and a trainedpianist will show that it allows him to generate text atspeeds close to that of professional QWERTY-typists.

Author KeywordsText entry; transfer of expertise; computational keyboarddesign; the piano keyboard

ACM Classification KeywordsH.5.m [Information interfaces and presentation (e.g.,HCI)]: Miscellaneous.

IntroductionIn the Interactivity session we present PianoText, a textentry method based on a piano keyboard. PianoText is aproof-of-concept showing how to transfer musicalknowledge to HCI. The goal is not to beat existing text

entry methods but to investigate how we can exploit thepiano capabilities and the expertise of piano players in atext entry task.

Figure 2: The distributions ofEnglish and music. Top: lettersvs. notesBottom: letter pairs vs.note pairs

As the piano keyboard has a different layout thantraditional QWERTY keyboards (88 keys instead of 26,etc.), we propose a novel key-letter mapping adapted tothis device. We want to exploit the skill of experiencedpianists in a text entry task, allowing the piano users toachieve a high level performance. For this reason, wedeveloped an algorithm based on computational keyboardoptimization [4, 5], but extend it to a case wherein twodistributions are considered: source (music) and target(language). The algorithm seeks a mapping of the mostfrequent notes in music to letters in the English languageas well as other considerations such as hand-traveldistances, musical patterns, etc. It also allows assigningone letter to several notes, forming clusters of notescorresponding to sequences of letters that are likely tooccur together.

Initial results with a professional pianist were promising,showing a transcription rate of 71 words per minute(wpm) for English sentences without prior training.

From Music to the PianoTextMusical instruments are a part of the human culture fromits beginning and can be used to achieve high-levelperformance. For instance, a trained pianist playing aclassical piece of music can produce 12 notes/sec easily. 1

This would correspond to a typing rate of 144 wpm.Compare this to the approx. 66–113 wpm achieved onlyby professional QWERTY typists [3].

Piano keyboards were already used in the early 19th

1First part of Frederic Chopin’s Mazurka (Op 17 / Bl 77)

century before typewriters with button designs took overthe market [1]. We argue that the piano keyboard andespecially the skilled pianist has a lot of potential as atext entry method: chords, a greater number of keys,well-practiced motor patterns based on the musical scales,etc. However, it never properly matured as a text entrydevice. The most popular designs, such as the Hughestelegraph, used a trivial alphabetical ordering of lettersthat is now known to be suboptimal in view of fingertravel times. Even experienced telegraphers’ performancewas 40 wpm or below [2]. In order to use the PianoText asan efficient text entry device we now define a mappingfrom letters to notes that exploits these advantages. Thismapping is optimized for professional pianists. Thuspotential users of PianoText are trained piano players.

Mapping and Sheet TranslationSimilar to keyboard optimization [4], the space of possiblemappings is too large to be addressed manually, even ifwe limit the analysis to mappings of single notes to singleletters. The piano has 88 keys and the English alphabethas 26 letters (we omit the space character and do notconsider capital letters), for about 1048 possible mappings.

We use the frequency of n-grams (sequence of letters ornotes) as our primary criterion when evaluating thecandidates (Step 1). This is based on the hypothesis thatthe most frequently encountered notes andnote-transitions will also be the fastest to respond to. Ourapproach is based on greedy search with additionalconstraints (Step 2). It maps letters to notes dependingon their frequency, avoiding dissonant intervals andshortening hand-travel times. After an optimal mapping isidentified, a few specific rules guide the translation ofsentences to a sheet of music a pianist can play in atranscription task (Step 3).

Figure 3: The optimized 0-level mapping from single letters of the English language to the keys on the piano keyboard computed bythe presented algorithm.

Step 1: Acquiring n-gram distributions

Figure 4: An extract of then-grams – mapping. Letter pairs,n-grams and words in English aremapped to the chords shown onthe piano.

For optimizing the mapping in Step 2, we need frequencydistributions for music and text. We acquired thedistributions of 1-grams (single notes), 2-grams(note-transitions – i.e., melodic intervals), and 2-notechords (harmonic intervals) from ten sight-readingpractice books. The distribution of letters and bigrams inthe English language was acquired from a corpus ofclassical literature2. In the top column in Figure 2 we cansee the distributions of single notes and single letters(1-grams). The notes roughly form a Gaussiandistribution, but the distribution of letters does notconform to this. The same dissimilarity can be seen in thebottom column, which shows the 2-gram distributions inEnglish (letter pairs) and music (note-transitions). Inaddition to these distributions, we consider all major andminor chords, their inversions, and the correspondingharmonic intervals, as well as the 100 most commonwords in the English language 3.

Step 2: MappingWe address the mapping problem with a greedy algorithmthat favors frequently played notes while avoidingdissonant intervals and minimizing hand-travel distances.This is formulated via the following principles:

2http://www.data-compression.com/english.html3http://oxforddictionaries.com/

– Frequency: Map letters to notes in the order oftheir probability, e.g. map “e”, the most frequentletter in the English language to the most frequentnote in the corpus of music data.

– Interval: Map common letter pairs to frequentnote transitions – i.e., non-dissonant and wellpracticed intervals.

– Distance: Minimize the distance between twonotes corresponding to frequent letter pairs

Algorithm 1 implements these principles and creates a0-level mapping from single notes to single letters likeshown in Figure 3. When all letters are assigned to a

Algorithm 1 Find Mapping

1: procedure FindMapping(ordrdLetters, ordrdNotes)2: for all l in ordrdLetters do3: mapping(l) := pop(ordrdNotes);4: while !(checkInterval) do5: replace assignment with next most frequent note6: end while7: end for8: for all frequent letter pairs do9: if !(checkDistance) then

10: assign additional note11: end if12: end for13: end procedure

unique note, respecting the Interval-principle, themapping is expanded by assignment of additional notes tomore frequent letters. This ensures the Distance-principleand forms clusters of notes corresponding to sequences ofletters that are likely to occur together. These twoprinciples are based on the distribution of the intervals,displayed in the bottom pane of Figure 2. It shows thatintervals in music are composed only up to a limited size.

The 0-level mapping is extended by a chord-mapping, seeFigure 4. This maps additionally the 30 most frequent2-note chords in music to the respective letter pairs inEnglish which covers 27% of the language. Furthermore,it maps the 100 most common words to 2-, 3- and 4-notechords on the basis of the major and minor scales.

The translation rules

Grouping: Create groups of4–8 notes playable by onehand.

Starting Point: Translate thefirst letter in a text with thenote that has the highestfrequency in the music corpus.

Distance: Translate a letterwith the note that minimizesthe distance to the preceding.

Hand-switch: When aninterval or the group sizebecomes too large start a newgroup by choosing a note thatis played with the other hand.

Chords: Certain words andn-grams can be translated intochords. They are not restrictedto one octave, thus theirdistance to the previous notecan be easily minimized.

Step 3: TranslationGiven a fixed mapping, translating English sentences intomusic requires certain rules, since one letter is mostlymapped to several notes and for a sequence of letters wemight have the possibility to translate it to a chord or toa sequence of single notes. The rules described on the lefthelp us to to decide for one translation.

InteractivityAt CHI’13 Interactivity, PianoText will be presented viathe following means:

– Explore PianoText: visitors can try the keyboard,and our software gives them feedback on a display.The notes on the piano are labeled for easieridentification.

– Transcribe text: visitors skilled in piano playing cantranscribe English sentences which will be presentedas sheets of music. We will announce a specialinvitation to those CHI visitors with piano skills tovisit our booth for a fun transcription competition.

– Live demo: we are in the process of training aprofessional pianist to memorize the mapping. Ourpreliminary results showed 71 wpm in transcription,without prior training, and we expect much higherrates to be reachable. Audience members canpropose English sentences and the pianist will typethem. Feedback for WPM will be shown on adisplay.

Figure 5: An English sentence translated with the computedmapping and the given translation rules.

References[1] M. Adler. Antique Typewriters, from Creed to

QWERTY. Schiffer Pub., 1997.[2] B. Bowers. Electrical engineering 100 years ago.

British Journal of Audiology, 13(S2):1–4, 1979.[3] D. Gentner. Expertise in typewriting. Technical

report, DTIC Document, 1984.[4] L. Light and P. Anderson. Typewriter keyboards via

simulated annealing. AI Expert, 1993.[5] I. MacKenzie and S. Zhang. The design and

evaluation of a high-performance soft keyboard. InProc. of the SIGCHI conference on Human factors incomputing systems, pages 25–31. ACM, 1999.