NYAI #4: Unsupervised Learning (Soumith Chintala) & Music Through ML (Brian McFee)

Understanding music through machine

learningBrian McFee

@functiontelechy

Why analyze music with computers?

Recommender systems

Musicology

Music education

Visualization, interactive display

Creative applications

What do we mean by music?Score, MIDI

Audio recordings

Time-varying prediction tasksStructural segmentation

Chord recognition

Beat / meter tracking

Instrument detection

Transcription

Structure analysiswith Dan Ellis @Columbia.edu, @Google

[M., Ellis, ISMIR2014]

Structure analysis

1. Similarity vs. dissimilarity

2. Break a song into pieces(detect boundaries)

3. Re-connect the pieces(detect similar segments)

Our strategy: a song is a graph1.Construct a graph over beats

2.Partition the graph to recover structure

3.Vary the partition size to expose multi-level structure

https://commons.wikimedia.org/wiki/File:Random_walk_25000_not_animated.svg

Building the graph [1/3]: The local graph1.Add a vertex for each beat

2.Add local edges between adjacent beats (t, t ± 1)

3.Weight edges by MFCC similarity (gaussian kernel)

1 2 3

1 2 3

1 2 3

1.Link k-nearest neighbors (in CQT space)

Building the graph [2/3]: The repetition graph

31 2 5 64 87

1.Link k-nearest neighbors (in CQT space)

2.Weight edges by feature similarity(gaussian kernel)

Building the graph [2/3]: The repetition graph

31 2 5 64 87

Building the graph [3/3]: The combination1.Take a weighted combination of local and repetition

A = μ *Local + (1-μ) *Repetition

Building the graph [3/3]: The combination1.Take a weighted combination of local and repetition

A = μ *Local + (1-μ) *Repetition

2.Optimize μ for a balanced random walk: P[Local move] ≅ P[Repetition move]

∀ i: μ ∑j Local[i,j] ≅ (1-μ) ∑j Repetition[i,j]

Building the graph [3/3]: The combination1.Take a weighted combination of local and repetition

A = μ *Local + (1-μ) *Repetition

2.Optimize μ for a balanced random walk: P[Local move] ≅ P[Repetition move]

3.μ has a closed-form (least-squares) optimum

∀ i: μ ∑j Local[i,j] ≅ (1-μ) ∑j Repetition[i,j]

Example: The Beatles - Come Together

Tim

e →

Time →

Partitioning via spectral clusteringAffinity matrix A

degree matrix Dii = ∑j Aij

Normalized Laplacian L = I - D-1A

Bottom eigenvectors encode component membership for each beat

Ly = λy

Cluster the eigenvectors Y of L to reveal structureTime →

Y9

Y8

Y7

Y6

Y5

Y4

Y3

Y2

Y1

Y0

Example: The Beatles - Come Together

Low-rank reconstructions expose structure

L ≅ Y[:, :m] · Y[:, :m]T

Multi-level segmentation

1.Construct the n-by-n graph

A

2.Compute Laplacian

eigenvectors Y

3.for m in [2, 3, …]

Partitions[m] := kmeans(Y[:, :m], k=m)

4.Return Partitions

Interactive visualization demo

Quantitative evaluation

Data sets:Beatles_TUT (174 tracks)

SALAMI small (735 tracks) and functions

Choosing the number of components

Maximum label entropy (with duration constraints)

Oracle: Best m per track, per metric (simulates interactive display)

Baseline: [Serrà, Müller, Grosche &

Arcos, 2012]

Boundary detectionPairwise frame labeling

Results: Beatles

F 0.5s F 3.0s F-Pairwise

Automatic m 0.312 +- 0.15 0.579 +- 0.16 0.628 +- 0.13Oracle 0.414 +- 0.14 0.684 +- 0.13 0.694 +- 0.12SMGA* 0.293 +- 0.13 0.699 +- 0.16 0.715 +- 0.15

Results: SALAMI small

F 0.5s F 3.0s F-Pairwise

Automatic m 0.192 +- 0.11 0.344 +- 0.15 0.448 +- 0.16Oracle 0.292 +- 0.15 0.525 +- 0.19 0.561 +- 0.16SMGA 0.173 +- 0.08 0.518 +- 0.12 0.493 +- 0.16

Results: SALAMI functions

F 0.5s F 3.0s F-Pairwise

Automatic m 0.304 +- 0.13 0.455 +- 0.16 0.546 +- 0.14Oracle 0.406 +- 0.13 0.579 +- 0.15 0.652 +- 0.13SMGA 0.224 +- 0.11 0.550 +- 0.18 0.553 +- 0.15

Part 1: wrap-upThe graph representation provides a simple framework for

segmentation

Algorithm is available in MSAF by Oriol Nieto @NYU @Pandorapip install msaf

Follow-up work: hierarchical structure evaluationBoundaries [M., Nieto, Bello, ISMIR2015]

Labelings [M., Nieto, Bello, in preparation]

Links to software

Music structure analysis framework

pip install msaf

JAMSpip install jams

Librosapip install librosa

conda install -c conda-forge librosa

Thanks!brian.mcfee@nyu.edu

@functiontelechy

https://bmcfee.github.io/