View
217
Download
0
Embed Size (px)
Citation preview
Music image processing
Tim Bell
Department of Computer Science and Software Engineering
University of Canterbury, Christchurch, New Zealand
Mu
sic
imag
e
pro
cess
ing
With…
o David Bainbridge (Waikato)
o Richard Lobb
o Dave Blizzard (Portland, OR)
o Richard Green
o John McPherson
o Karen Lin
o Annabel Church
o Simon Glass
Mu
sic
imag
e
pro
cess
ing
Overview
o OMR
o Digital music stand
o Page turning and image size
o Fast capture of music
o Music classification
Mu
sic
imag
e
pro
cess
ing
The vision...
o All music available on the web
o as score, recording and MIDI
o search by name, composer, lyrics, phrase, similarity, genre, novelty…
o remunerate those responsible
o culturally sensitive access
Mu
sic
imag
e
pro
cess
ing
Barlow and Morgenstern 1949
Mu
sic
imag
e
pro
cess
ing
Barlow and Morgenstern 1949
Mu
sic
imag
e
pro
cess
ing
Barlow and Morgenstern 1949
Mu
sic
imag
e
pro
cess
ing
Activities with music
o Composing
o Arranging
o Performance
o Teaching
o Musicology
o Recording
o Accompanying
o Transcribing...
Mu
sic
imag
e
pro
cess
ing
Digital music problems
o Cost of hardware and software
o Viewing music on small screen
o Loss of inspiration and creativity
o Loss of efficiency
o Learning curve
o Software compatibility
Mu
sic
imag
e
pro
cess
ing
Problems with paper
o Pen or pencil?
o Indexed retrieval
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
memorisedoral traditionoriginal ideas
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
live performancemp3, wav, cd
video?
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
sheet musicscreen
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
sheet musicscreen
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
sheet musicscreen
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
sheet musicscreen
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
sheet musicscreen
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
sheet musicscreen
D (lick #1) Dmin#5 Look at me now, will I ever learn?
D (lick #2) Dmin#5 GI don’t know how but I suddenly lose control.
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Live performance
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Live performance
read
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Live performance
play(interpret)
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Transcription
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Transcription
listen
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Transcription
write
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
BMP, GIF, JPEG
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
scanner,camera
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
print,display
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
MIDINIFF
MUSICXMLGUIDO
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Render(Sibelius, Lime, Guido, Tex etc.)
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
OMR
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Synthesis(audio rendering)
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Audio analysis(monophonic,polyphonic)
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Data entry
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Weak links
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Labour intensivelinks
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
QBH
Operations on music
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
ComposeArrange/orchestrate
RehearseReact
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Library (personal, shared)Music stand (rehearse, perform)
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Intermediate formArchive
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Suitable for:transpositionpart splitting
reductionsearching
theme detectionaccompaniment
performance following
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
listeningrecording studio
analysisthumbnailbacking
Mu
sic
imag
e
pro
cess
ing
Optical Music Recognition
Mu
sic
imag
e
pro
cess
ing
Mu
sic
imag
e
pro
cess
ing
Mu
sic
imag
e
pro
cess
ing
Wabot-2
o 1980-1984
o Read simple score
o Heavy processing requirements
Mu
sic
imag
e
pro
cess
ing
Mu
sic
imag
e
pro
cess
ing
Staff line removal/identification
o Horizontal projection
o Vertical slices
o Wobble/track
o Chords
o Template
Mu
sic
imag
e
pro
cess
ing
Mu
sic
imag
e
pro
cess
ing
Horizontal projection
Mu
sic
imag
e
pro
cess
ing
Piece at an angle
o Rotate until correct
Mu
sic
imag
e
pro
cess
ing
Vertical slices
Mu
sic
imag
e
pro
cess
ing
Vertical slices
Mu
sic
imag
e
pro
cess
ing
Wobble/track
Mu
sic
imag
e
pro
cess
ing
Object location
o Fragmentation
o Superimposed
o Touching objects
Mu
sic
imag
e
pro
cess
ing
Identifying objects
o Flood fill
o Template matching
o Hough transform
Mu
sic
imag
e
pro
cess
ing
Constructing musical features
o Grammars
o Decision tree
o Rules
Mu
sic
imag
e
pro
cess
ing
Musical semantics
o Treble clef determines pitches
o Accidentals change pitch
o Time signature changes note lengths
Mu
sic
imag
e
pro
cess
ing
Commercial systems
o Sharpeye
o Vivaldi
o Neuratron PhotoScore
o … and more
Mu
sic
imag
e
pro
cess
ing
Is 96% recognition good enough?
o One mistake in 24 notes
o No interpretation if playing music
o Time to set up, train and correct greater than typing?
Mu
sic
imag
e
pro
cess
ing
Not just notes…
Mu
sic
imag
e
pro
cess
ing
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Optical musicrecognitionin practice!
Mu
sic
imag
e
pro
cess
ing
The “gulf of interpretation”
o Classical tempo and dynamics
o Jazz improvisation
o Rock style (e.g. syncopation, articulation)
o Figured bass
o Cadenzas
o MIDI vs. Orchestra
Mu
sic
imag
e
pro
cess
ing
Interpretation
Mu
sic
imag
e
pro
cess
ing
Interpretation
Mu
sic
imag
e
pro
cess
ing
Humanmemory
Visualdisplay
Digitalimage
Digitalsemantic
Audio
Pen-based musicdata entry
Mu
sic
imag
e
pro
cess
ing
Mu
sic
imag
e
pro
cess
ing
Mu
sic
imag
e
pro
cess
ing
Coloured staveline removal
o Scanned as RGB
o Convert to HSV and CMYK
o V indicates colour
o K indicates pencil, black pen
Mu
sic
imag
e
pro
cess
ing
Mu
sic
imag
e
pro
cess
ing
Mis-classified images
Mu
sic
imag
e
pro
cess
ing
Discussion?