BiRD-EMANNBird Recognition and Detection via Embedded Systems, Microphone Arrays, and Neural Networks

Milton Collins, Luis Garrido, Eric Jiang, Patrick Karol {mec327, lmg275, ewj12, pok1} @scarletmail.rutgers.edu

Advisors: Prof. Athina Petropulu, Vikram Padman*, Manoj Viswambharan* * from Harris Corporation

- To record Nocturnal Flight Calls (NFCs) of migratory birds for use in ornithological research.

- Create a Convolutional Neural Network (CNN) which identifies bird species based on the spectrograms of their NFCs.

- Organize the species correctly identified by the CNN and build a SQL Database for pairing flight conditions, migration direction, as well as time and place of identification.

- Automatic Start/Stop Recording using the Power Spectral Density of the sample buffer frames.

- Network communication via TCP/IP for sending audio captured by FPGA to a remote Host.

- Auto correlating recordings between various HMAs to triangulate the location of a sound source.

- Expand database of calls for more species.

- Use Hemispherical Microphone Array (HMA) to capture NFCs. Optimize sampling points on the HMA to avoid spatial aliasing. Implement beamforming algorithms using MATLAB to extract sound Direction-of-Arrival (DoA).

- With a Zybo Z7-10 running PetaLinux, read in and store samples in PL fabric, then use PetaLinux to move data to DDR and finally to external memory.

- CNN trained on database of warbler NFC spectrograms. 10 species chosen, model created using Keras.

- Create a SQL database using MySQL and Amazon’s Relational Database Services. The cloud based instance is connected to the system via Python code.

- We would like to thank Prof. Petropulu, our advisors from Harris, Vikram Padman and Manoj Viswambharan, Prof. Phil Southard, Prof. Sophocles Orfanidis, Yi Han, Dean Hana Godrich, and Rutgers Makerspace.

References [1] “Keras: The Python Deep Learning Library.” Keras Documentation, keras.io [2] Rafaely, B. (2015) Fundamentals of Spherical Array Processing. Berlin, DE: Springer International [3] Politis, A (2016) Microphone array processing for parametric spatial audio techniques. Dept. of Signal Processing and Acoustics, Aalto University, Finland [4] Crockett, Louise H., et al. The Zynq Book: Embedded Processing with the ARM Cortex-A9 on the Xilinx Zynq-7000 All Programmable SoC. Strathclyde Academic Media, 2014.

- CNN Accuracy reaching 70.58% with the training set and data augmentation. This can be improved by accumulating larger data sets.

- Theoretical estimate of 11 kHz spatial bandwidth for localization and estimates of ideal microphone locations on HMA.

- Database configured to share captured information and data with ornithologists. Collected bird flight data will aid in research to further identify habits in environmental migration patterns.

Embedded Zybo Solution

Motivation

Methodology

Results

Future Work

Acknowledgements

CNN and HMA Processing on Host

Capturing the Audio Source

- Our warbler makes its NFC, which is then picked up by the 8 microphone elements on our HMA. Note, the bottom half of the sphere shows the microphone reflections allowed due to the acoustic image principle.

- Microphone signals sampled using the PMOD AD1 ADC and get sent to the Zybo Z7-10

- NFC Spectrogram is created and then fed into the CNN, architecture on the right, in Python using Keras.

- Vivado Block Design, using a FIFO and DMA to access samples stored on FPGA fabric. Then the samples get turned into .wav files for processing on Host.

- Below is a graph of the orthogonality of our proposed HMA. Some aliasing occurs due to small number of microphones.

- Entity-Relationship Diagram for storing bird identification, location of microphone, calls being made, and the bird’s position.

Photo: Milton Collins