A Preliminary Evaluation of Music Visualization

A Preliminary Evaluation of

Music VisualizationDanny Rado

The “clavilux” [above] was invented by Thomas Wilfred in 1924

¿Is Music Visualization an area of research?  PlanMotivationEveryday music visualizationsResearchThe very best of music visualizationsDiscussionIdentify some key issues for the fieldDetermine why the field is underdeveloped

Motivation :: Karaoke

Karaoke sounds bad because it is difficult:• The text gives little music information • User must rely on memory

Motivation :: PersonalAI project:• LongSong: Automatic Song Extension 

 Songs can be extended at theme repetitionsIdentifying theme structure by sound is difficult

The identification is simple with a 2D visualization:

Martin Wattenburg, http://www.bewitched.com/match/music.html

Motivation :: Music InformaticsMusic information that requires visualization:• Interaction of instruments• Tonality, movement of tonal center• Rhythm• Relationship between notes

o Consonance, dissonance, triadic relationships•  Harmonic Analysis• Phrase Structure• Note spelling

Bach, Minuet G Major

Itunes Visualization

Effects are generated on sound eventsThese visualizations give less information than the standard music video

Windows Media Player Visualization

Frequency bars are the most informative visualizationsStill not portraying any depth of music visualization

Winamp Visualization

Stick Figures plug-in allows users to create drawings and animate them to a few frequenciesGives semantic meaning to music information making it easier to identify and follow

Media Player Visualization Review• Equivalent to music video screen savers• Simply hypnotic, psychedelic imagery

 Despite the useful framework for plug-ins, the creative effort is effectively wasted• Frequencies simply have little inherent information

The Standard Music Video

Not a visualization• Limited information is obscured by performance, creativity,

and celebrity

Shows the instruments being played Serves as a baseline to measure the success of music visualizations

The Important Music Video

Hints at the structure of the music Graphics are repeated on musical theme repetition

Music Format• Standard music formats do not have enough information

 MIDI format has far more information Instead of audio information, MIDI stores composition information:• Instruments• Note times• Note intensities• Changes in pitch• Changes in rhythm 

 MIDI creates the opportunity to visualize music

MIDI Text [Midi Disassembler]1. MThd | Format=1 | # of Tracks=8 | Division=384

Track #0 ******************************************    Time       Event   1: 1:  0 |Tempo     | BPM=140   | micros\quarter=428570        |Time Sig     |  4/4      | MIDI-clocks\click=24 | 32nds\quarter=8 106: 1:  0 |End of track|

Track #1 ******************************************    Time       Event   1: 1:  0 |Track Name  | len=9     |        0x31 0x73 0x74 0x20 0x56 0x6F 0x69 0x63 0x65       <1st Voice>   9: 1:  0 |Program     | chan=14   | pgm #= 54 Choir Oohs        |Controller  | chan=14   | contr=Volume H | value=100     96 |Controller  | chan=14   | contr=Effects | value= 40      3:  0 |On Note     | chan=14   | pitch=F 3     | vol=127    336 |Off Note     | chan=14   | pitch=f 3     | vol=127      4:  0 |On Note     | chan=14   | pitch=G 3     | vol=127    336 |Off Note     | chan=14   | pitch=g 3     | vol=127  10: 1:  0 |On Note     | chan=14   | pitch=A 3     | vol=127    336 |Off Note     | chan=14   | pitch=a 3     | vol=127      2:  0 |On Note     | chan=14   | pitch=G 3     | vol=127      4:332 |Off Note     | chan=14   | pitch=g 3     | vol=127

2. [Skipping ahead]3. Track #5 ******************************************

    Time       Event   1: 1:  0 |Track Name  | len=7     |        0x54 0x72 0x75 0x6D 0x70 0x65 0x74               <Trumpet>   2: 1:  0 |Controller  | chan= 4   | contr=Effects | value= 40        |Program     | chan= 4   | pgm #= 57 Trumpet        |Controller  | chan= 4   | contr=Volume H | value=100      1 |On Note     | chan= 4   | pitch=F 2     | vol=62     

UIUC Research: Isochords Isochords: Visualizing Structure in Music,•  Helps classify music structure with information about:

o chord qualityo interval qualityo chord progression

• http://social.cs.uiuc.edu/projects/isochords-movies/Here%20Comes%20the%20Sun.mov

 University of Illinois,Urbana Champaign

UL + CUL Research: i-Maestro

3D Augmented Mirror: A Multimodal Interface for StringInstrument Learning and Teaching with Gesture Support•  Prevent or repair bad performance posture

University of Leeds + City University of London

USC Research: MuSA.RT

MuSA.RT: Music on the Spiral Array . Real-Time•  a multimodal interactive system for music analysis and visualization•  3D model, shows tonal structures such as pitches, chords and keys.

Applications: Animusic

Manually Animated Music Visualizations•  Invent virtual instruments that often resemble real ones•  Visualizations present some limited music information

Applications: MidiJam

Automatically Animated Music Visualizations• Shows interaction between real instruments and animates

their performance

Applications: Music Animation MachineProgram has many display types:Piano Roll [right], Pitch Class Wheel, Interval Wheel, Interval + Pitch, Shapes,Tonality Staff, Tonality Compass, Triads Lattice, Parts Sequence:• Lines• Balls• Wedges

Applications: Music Animation MachineProgram has many display types:Piano Roll, Pitch Class Wheel, Interval Wheel, Interval + Pitch, Shapes,Tonality Staff, Tonality Compass, Triads Lattice, Parts Sequence:• Lines• Balls [right]• Wedges

Conclusion:Despite the common perception that Music Visualization is not an area of research, significant work has been done.

Beethoven, Fur Elise

Discussion:• Does the Music Visualization research seem substantial?• Why are there so few publications in the field?

o ACM Portal has less than 35• Can we identify some key issues for the field?