Slide 1

Detection and Segmentation of Bird Song in Noisy Environments Lawrence Neal, UHC Honors Thesis

Slide 2

Bioacoustics Project Bird Species Identifiable by species Presence/Absence, activity data is useful Bird activity may shift in response to climate change, ecological factors

Slide 3

Bioacoustics Project

Slide 4

Automated Recording Song Meter automated recorders Collected May-August beginning 2009

Slide 5

Audio Data Analysis Involves several steps: Extracting Bird Sound from Audio Identifying Bird Species Mapping species data back to sites

Slide 6

Audio Data Analysis Involves several steps: Extracting Bird Sound from Audio (Segmentation) Identifying Bird Species Mapping species data back to sites

Slide 7

Segmentation Time-Domain Segmentation Separates audio into multiple clips Energy Thresholding, Onset/Offset Detection Has been applied to bird song Harma 2003, Fagerlund 2004, Lee 2008

Slide 8

Segmentation Time-Domain Segmentation

Slide 9

Segmentation Cannot separate overlapping sounds

Slide 10

Segmentation Time-Frequency Segmentation Segment regions of the 2D spectrogram

Slide 11

Segmentation Spectrogram Segmentation Similar to image segmentation

Slide 12

Spectrograms Two-dimensional representation of sound Audio amplitude at each (time, frequency) Generated by short-time Fourier Transform Male voice saying 'nineteenth century'. Violin playing (note harmonics)

Slide 13

Spectrograms Tradeoffs in parameters Larger STFT size Higher freq. resolution

Slide 14

Spectrograms Tradeoffs in parameters Shorter step size Higher time resolution

Slide 15

Spectrogram Segments Each segment is a continuous region Defined by a binary mask over the spectrogram

Slide 16

Spectrogram Segments Can be converted back to audio with inverse STFT, or left as 2D segments

Slide 17

Segmentation Methods Per-Pixel Random Forest Trains on one feature vector per pixel Outputs probability per-pixel Superpixel Merger Method First splits spectrogram into superpixels Trains on one feature vector per superpixel Second classifier trains per superpixel pair Outputs connected sets of superpixels

Slide 18

Random Forest Supervised Classifier Trains on human-provided data with labels Feature Vector of values, each with yes/no label Learns to mimic the humans labels Based on decision trees: Tree is traversed with feature vector X Each interior node is a decision of the type: If (X d < ) go left; else go right Each leaf node contains a class label In this case, two classes: Bird Sound and Negative

Slide 19

Random Forest Constructed by recursive procedure Check if all remaining examples are the same If so, finish with a leaf node Select a random subset of features For each one, find the optimal split (highest Gini) Choose the (feature, split) pair for maximum Gini coefficient and create new interior node Split the examples and recursively create two child nodes Classification is a vote among all trees

Slide 20

Per-Pixel Training Hand-Drawn mask over spectrogram Pixels are randomly sampled

Slide 21

Per-Pixel Training Feature vector includes: Pixel Frequency Window Variance All window pixel values

Slide 22

Per-Pixel Output Probability Mask over the spectrogram Threshold is applied to extract segments

Slide 23

Per-Pixel Output

Slide 24

Slide 25

Slide 26

Per-Pixel Limitations Scope is limited to window size High threshold causes oversegmentation Low threshold causes undersegmentation Slow- must classify for each pixel

Slide 27

Superpixel Method Begins with an initial pre-segmentation Modification of Simple Linear Iterative Clustering (SLIC) image segmentation Uses computed features that describe regions of the spectrogram Segments are sets of superpixels

Slide 28

Superpixel Clustering Based on SLIC method: Each pixel is assigned a 5-valued vector (X,Y, L, a, b) for position and color Locally-constrained K-Means Clustering Each centroid searches only a radius of 2S S = sqrt(N/K) Creates a set of regularly-sized regions Some regions boundaries follow the edges of larger objects in the image

Slide 29

Superpixel Clustering Over-segments an image Edges of clusters are along image edges But, doesnt work for spectrograms

Slide 30

Superpixel Clustering Spectrograms lack edges Also, only one channel of color Instead of (x,y,L,a,b), we use a new vector: (x, y, B, V, G x, G y, P x, P y )

Slide 31

Superpixel Clustering X,Y values Time and frequency values in the spectrogram B, V Pixel values after Gaussian blur, variance of pixel values G x,G y Horizontal/Vertical Sobel Gradient values P x, P y Time and Frequency values of nearest peak (weighted by Gaussian kernel)

Slide 32

Superpixel Clustering

Slide 33

Foreground/Background Classifier Random Forest trained using the same manual spectrogram labels as per-pixel Each superpixel is labeled positive (foreground) if more than 10% of its area overlaps with a positive-labeled region Feature vector describes superpixel: Mean and variance of pixel values, blurred pixel values, peak frequencies Histogram of Oriented Gradients

Slide 34

Foreground/Background Classifier

Slide 35

Superpixel Merger Classifier Random Forest trained to classify pairs of adjacent superpixels Positive classification: Merge together Negative classification: Split apart After background pixels are discarded, all remaining edges between superpixels are classified All edges above a threshold are merged

Slide 36

Superpixel Merger Classifier

Slide 37

Superpixel Method Output

Slide 38

Slide 39

Slide 40

Slide 41

Slide 42

Slide 43

Evaluation Datasets HJ Andrews dataset, 625 recordings Each 15 seconds long Drawn 2 each from 24 hours Set A dataset, 166 recordings All from early and mid morning Paired by year, 2009/2010

Slide 44

Differences in Training Data

Slide 45

Results

Slide 46

Results

Slide 47

Results

Slide 48

Results

Slide 49

Future Work Superpixel Method is promising Faster than per-pixel classification Could use more sophisticated merger technique

Slide 50

Bibliography A. Harma, Automatic identication of bird species based on sinusoidal modeling of syllables, in IEEE International Conference on Acoustics Speech and Signal Processing, April 2003, pp. 545 548. Chang-Hsing Lee, Chin-Chuan Han, and Ching-Chien Chuang, Automatic classication of bird species from their sounds using two-dimensional cepstral coefcients, IEEE Transactions on Audio, Speech, and Language Processing, vol. 16, no. 8, pp. 1541 1550, 2008. Leo Breiman, Random forests, Machine Learning, pp. 532, January 2001. Fagerlund, Seppo. Automatic Recognition of Bird Species by Their Sounds. Masters Thesis, HELSINKI UNIVERSITY OF TECHNOLOGY, Laboratory of Acoustics and Audio Signal Processing. Nov. 8, 2004