Music Processing

Roger B. Dannenberg

OverviewMusic RepresentationMIDI and SynthesizersSynthesis TechniquesMusic Understanding

Music Representation

Acoustic Level: sound, samples, spectra

Performance Information: timing, parameters

Notation Information: parts, clefs, stem direction

Compositional Structure: notes, chords, symbolic structure

Performance InformationMIDI bandwidth is 3KB/s, or 180KB/min

More typical: 3KB/minute, 180KB/hour•Complete Scott Joplin: 1MB•Output of 50 Composers (400 days of music):

500MB (1 CD-ROM)Synthesis of acoustic instruments is a problem

Music NotationCompact, symbolic representation

Does not capture performance information

Expressive “performance” not fully automated

Compositional StructureExample: Nyquist (free software!)

(defun melody1 ()(seq (stretch q (note a4) (note b4) (note cs5) (note d5))))

(defun counterpoint () …)

(defun composition () (sim (melody1) (counterpoint)))

(play (transpose 4 (composition)))

MIDI: Musical Instrument Digital Interface Musical Performance Information:•Piano Keyboard key presses and releases

•“instrument” selection (by number)

•sustain pedal, switches

•continuous controls: volume pedal, pitch bend, aftertouch

•very compact (human gesture < 100Hz bandwidth)

MIDI (cont’d)

Point-to-point connections:•MIDI IN, OUT, THRU

•Channels

No time stamps•(almost) everything happens in real time

Asynchronous serial, 8-bit bytes+start+stop bits, 31.25K baud = 1MHz/32

MIDI Message Formats 8 ch key# velKey Up

9 ch key# velKey Down

Program Change

Polyphonic Aftertouch

System Exclusive

A ch presskey#

C ch index#

B ch ctrl# valueControl Change

Channel Aftertouch D ch press

E ch lo 7 hi 7Pitch Bend

F 0

F E

… DATA …

Standard MIDI FilesKey point: Must encode timing informationKey point: Must encode timing information

<track data> =1 or more <track event>,<track event> = <delta time> <event>,<event> = midi data or <meta event>, <meta event> = FF<type><length><data>

<track data> =1 or more <track event>,<track event> = <delta time> <event>,<event> = midi data or <meta event>, <meta event> = FF<type><length><data>

Delta times use variable length encoding, omit for zero.

Interleave time differences with MIDI data...

Music Synthesis IntroductionPrimary issue is control•No control Digital Audio (start, stop, ...)•Complete control Digital Audio (S[0], S[1], S[2], ... )•Parametric control Synthesis

Music Synthesis Introduction (cont’d)What parameters?•pitch•loudness•timbre (e.g. which instrument)•articulation, expression, vibrato, etc.•spatial effects (e.g. reverberation)

Why synthesize?•high-level representation provides precision of specification and supports interactivity

Additive Synthesis

amplitude A[i] and frequency [i] specified for each partial (sinusoidal component)

potentially 2n more control samples than signal samples!

S i A j sin j i t j0

n 1

Additive Synthesis (cont’d)often use piece-wise linear control envelopes to save space

still difficult to control because of so many parameters

and parameters do not match perceptual attributes

Table-Lookup Oscillators If signal is periodic, store one period Control parameters: pitch, amplitude,

waveform

Phase

+

Frequency

Amplitude

x

Efficient, but ... Spectrum is static

Efficient, but ... Spectrum is static

(Note that phase and frequency are fixed point or floating point numbers)

(Note that phase and frequency are fixed point or floating point numbers)

FM Synthesis Usually use sinusoids “carrier” and

“modulator” are both at audio frequencies

If frequencies are simple ratio (R), output spectrum is periodic

Output varies from sinusoid to complex signal as MOD increases

A F

A F

AMPL

out = AMPL·sin(2·FREQ·t + MOD sin(2R·FREQ·t))

+

FREQMOD

FM Synthesis (cont’d) Interesting sounds, Time-varying spectra, and ...Low computation requirementsOften uses more than 2 oscillators

… but …Hard to recreate a specific waveform

No successful analysis procedure

Samplers store waveforms for playback

Sounds are “looped” to extend duration

Spectrum is static (as in table-lookup), so:•different samples are used for different pitches•simple effects are added: filter, vibrato, amplitude envelope•attack portion, where spectrum changes fastest, added to front

Sample-based Synthesis

Attack Loop Loop again ...

Physical ModelsAdditive, FM, and sampling:more-or-less perception-based.

Physical Modeling is source-based: compute the wave equation, simulate attached reeds, bows, etc.

Example:Reed Bore Bell

Physical Models (cont’d)Difficult to control, and ...Can be very computationally intensive

… but ...Produce “characteristic” acoustic sounds.

Music Understanding IntroductionScore Following, Computer Accompaniment

Interactive PerformanceStyle RecognitionConclusions