15 Nov. 07 1

Visualization of Music

Donald ByrdSchool of Informatics & Jacobs School of Music

Indiana University

Updated 15 Nov. 2007

Copyright © 2003-07, Donald Byrd

15 Nov. 07 2

Basic Representations of Music & Audio (1)

Audio (e.g., CD, MP3): like speech

Time-stamped Events (e.g., MIDI file): like unformatted text

Music Notation: like text with complex formatting

Digital Audio

Time-stamped Events

Music Notation

15 Nov. 07 3

Basic Representations of Music & Audio (2)

Notation: very complex structure

Audio: no (explicit) structure

Events/MIDI: simple structure

15 Nov. 07 4

Basic and Specific Representations vs. Encodings

Audio Time-stamped Events Music Notation

CMN Mensural not.Gamelan not.

SMF

Csound score

NotelistMusicXML

FinaleETFexpMIDI File

Time-stamped MIDITime-stamped expMIDI

Csound score

Waveform

Red Book (CD)

Tablature

.WAV

Basic and Specific Representations (above the line)

Encodings (below the line)

15 Nov. 07 5

A Music Collection: Visualization via “Islands of Music”, etc.

Cooper, Matthew; Foote, Jonathan; Pampalk, Elias; & Tzanetakis, George (2006). Visualization in Audio-Based Music Information Retrieval. Computer Music Journal 30(2), pp. 42—62.

15 Nov. 07 6

Chopin Preludes in R: All at once; the C major alone

15 Nov. 07 7

Types of Music Visualizations• What music is visualized?

– Abstractions of music, no particular works– Collections of pieces– Individual pieces

• How is visualization organized?• By time (horizontal structure)• Other

• Performances or works?• How dynamic?• Dimensionality

– 2D (on paper or screen)– 3D (simulated on screen, virtual reality, etc.)

15 Nov. 07 8

Types of Visualizations: Static or Changing?

• Changing (“animation”) resulting from scrolling is superficial– Effectively “semi-dynamic”

• Truly dynamic visualizations are less common

15 Nov. 07 9

Visualization Pitfalls

• Disorientation (navigation confusion)– Common with…

• Simulated 3D: most people aren’t good at• Multiple visualizations

– …but can occur w/ any poorly-designed interface

• Distracting glitziness• Unresponsive• Misleading

– Cf. Edward Tufte’s books

• The more parameters, the harder to make everything clear– Music has loads of parameters!

15 Nov. 07 10

The Obvious Static Visualization

The first few measures of Variation 8 of the “Twinkle” Variations

15 Nov. 07 11

Static Multiple Visualizations of Music via Text: Standard MIDI File (file dump)

• 0: 4D54 6864 0000 0006 0001 0003 01E0 4D54 MThd.........‡MT • 16: 726B 0000 0014 00FF 5103 0B70 C000 FF58 rk......Q..p¿..X • 32: 0402 0218 0896 34FF 2F00 4D54 726B 0000 .....ñ4./.MTrk.. • 48: 0055 00FF 0305 5069 616E 6F00 9048 3881 .U....Piano.êH8Å • 64: 6480 4840 0C90 4A38 8164 804A 400C 904B dÄH@.êJ8ÅdÄJ@.êK • 80: 3881 6480 4B40 0C90 4D38 8164 804D 400C 8ÅdÄK@.êM8ÅdÄM@. • 96: 904F 3883 4880 4F40 1890 4F38 8360 9050 êO8ÉHÄO@.êO8É`êP • 112: 3883 4880 4F40 1890 4D38 8330 8050 4018 8ÉHÄO@.êM8É0ÄP@. • 128: 804D 400D FF2F 004D 5472 6B00 0000 3200 ÄM@../.MTrk...2. • 144: FF03 0550 6961 6E6F 8F00 9041 2B81 6480 ...Pianoè.êA+ÅdÄ • 160: 4140 0C90 4330 8164 8043 400C 9044 3181 A@.êC0ÅdÄC@.êD1Å • 176: 6480 4440 0C90 4647 8164 8046 4001 FF2F dÄD@.êFGÅdÄF@../ • 192: 00 .

15 Nov. 07 12

The Obvious Static Visualization: A Complex Case

Complications on staff 2:• Editorial additions (small notes)• Instruments sharing notes only some of the time• Mixed durations in double stops• Multiple voices (divisi notation)

• Rapidly gets worse with more than 2!

15 Nov. 07 13

Static Visualizations Aligned by Time: Hendrix Experience, L.L.C.

15 Nov. 07 14

Static Visualizations Aligned by Time: Audacity & Don’s Audio Segmentation demo

15 Nov. 07 15

Static Visualizations Aligned by Time: Sonic Visualiser

15 Nov. 07 16

Static Visualization over Time: Variations2 Timeliner

15 Nov. 07 17

Static Visualization over Time: Chopin Mazurkas Project (Harmony)

(from http://www.mazurka.org.uk/info/keyscape/)

15 Nov. 07 18

Semi-Dynamic Visualization over Time: Malinowski’s Music Animation Machine

(from http://www.musanim.com/)

15 Nov. 07 19

Dynamic Visualization over Time: Langner & Goebl’s “Performance worm”

15 Nov. 07 20

Why is Visualization Relevant to Music Informatics?

• Affects music representation

• Brings up HCI (Human/Computer Interaction) issues

• A powerful tool for finding patterns in information