Embed Size (px)
Citation preview
Making Machines that Make Music
Srihari Sriraman nilenso
should we listen to some now?
Why I do thisI Sing, I do computers
Bleeding edge research Dreamy ambitions
Interesting By-products
What this talk is about
Melody Modelling Synthesis
Generation
Melody Modelling
SynthesisGeneration
Melody ˈmɛlədi
noun a sequence of single notes that is musically satisfying; a tune.
Carnatic Music
Carnatic Music
Kalyani, Extempore MS Gopalakrishnan, Violin
Khamas, Thillana Abhishek Raghuram
Carnatic Music
South Indian classical music Ragas, Gamakams
Vocal Tradition Rich in compositions
Extempore / Manodharma
Tanpura
Veena Mridangam
The foundations of musical abstractions
in Carnatic music
ShruthiTonic note
Choice of artist Swarams are relative to this
LayaRhythm concepts
Similar to time signatures A rather mature system
Sa Ri Ga Ma Pa Da SaNi
Swarams
Sa Ri Ga Ma Pa Da SaNi
S R G M P D SN
R1 R2 R3
G1 G2 G3 M1 M2
D1 D2 D3
N1 N2 N3
Notation
Pronunciation
Variations
SwaramsThe 12 semitones
Elements of a raga Simples are sung
Prescriptive notation
Rāgā
Kaapi, Extempore TM Krishna
RagaHas a name
Rule to ascend Rule to descend
Not necessarily symmetric Not necessarily linear Grouped into families
RagaHas a name
Rule to ascend Rule to descend
Not necessarily symmetric Not necessarily linear Grouped into families
Demo
of the fundamental abstractions.
Tools & Libraries
Fuzzy searchFor indic languages
Needs to be fast Primary stitching mechanism Helps with multi-source data
A quick recapPlay the scale of a raga
Fuzzy find a raga Play a phrase
Play a phrase in the context of a raga Play some prescriptive notation
But..
..that doesn’t sound like Carnatic music, does it?
Synthesis
Enter Melographs
Me
Machine
another phrase
Me
Machine
Prescriptive vs Descriptive
Gamakams
Sphuritam
Orikai
Jaaru
Kampitam
Sphuritam Nokku Ravai
Kandippu Ullasitam Etra-jaru
Iraka-jaru Odukkal
Orikai Vali
Kampitam
Gamakams in SSP
Gamakams in SSP
Gaayaka
| S, N D | N S R G |
((P S,,)) , ((S , S>>> S)) -((D. S. D)) ((S , S>> S))- S R ((G<< G , ,))
Subramanian, 2009 Database of phrases
Automatic Gamakam feature – guided
Modelling Gamakams
Me
Machine
Back to this…
PASR
Srikumar 2013 Pitch, Attack, Sustain, Release
Vector specifies the PASR vars for each prescriptive note
Me
Machine
Rendering PASR…
Rendering PASR…
Generation
Random | Within a raga
Random | Within a raga
Get data
Kosha An Open Carnatic Music Database
http://github.com/ssrihari/kosha
Study data
Melographs
Melographs
kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg.wav.pitch.frequencies-pitch-histogram
kalyANi-Kunnakudi-R-Vaidyanathan-nidhi_cAla_sukhamA-tyAgarAja49.mpeg.wav.pitch.frequencies-pitch-histogram
Pitch Histograms
Pitch Histograms
kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg
Pitch HistogramsKalyani - Vocal Kalyani - Violin
Mohana - MandolinMohana - Vocal
Revati - Vocal Revati - Instrumental
Extract Music Information
Midi Histogram
Normalised Midi Histogram
Tonic note identification
Bellur, A., V. Ishwar, X. Serra, and H. A. Murthy (2012) A knowledge based signal processing approach to tonic identification in indian classical music.
Bellur, A., and H. A. Murthy (2013) Automatic tonic identification in classical music using melodic characteristics and tuning of the drone.
Srihari, S. (2016) * Pick the most frequent note, it mostly just works.
* not really, no
Tonic note identification
Tonic note identification
Swaram HistogramKalyani
S, R2, G3, M2, P, D2, N3, S. S., N3, D2, P, M2, G3, R2, S
Kalyani S, R2, G3, M2, P, D2, N3, S. S., N3, D2, P, M2, G3, R2, S
Revati S, R1, M1, P, N2, S. S., N2, P, M1, R1, S
Mohana S, R2, G3, P, D2, S. S., D2, P, G3, R2, S
Generation with weighted probabilities
In comparison with random
Random
Single Swaram
Weighted
Melody insights #1Tonic note is prominent
Sa, and Pa have higher and sharper peaks Other note peaks are blunt
Probabilities of all swarams in a raga are not the same Probabilities across octaves are not the same
Two swaram probabilities
Two swaram probabilities
Prominence of adjacency Encoded rules of Arohanam and Avarohanam
Two swaram probabilities
Single swaram vs Two swarams
Two Swaram
Weighted
Single Swaram
Weighted
Melody insights #2Swarams close to each other are more melodious The rules of Arohanam, Avarohanam are encoded
We begin to see gamakams Sometimes, the in-between is worse than either extreme
Three swaram probabilities and more
A simple markov chain
First Order Matrix
https://en.wikipedia.org/wiki/Markov_chain#Music
Markov Chains in Music
https://github.com/rm-hull/markov-chains
Second Order Matrix
Markov Chains in Music
Melody insights #3Generic markov chains don’t really work
LSTMs also don’t work, probably
By-products
Automatic Transcription
Automatic Transcription
(:..n1 :..n1 :..m1 :..m1 :..d1 :..d1 :..d3 :.g1 :.m1 :.m1 :.m1 :.r3 :.r1 :..n3 :..d3 :..p :..g3 :..m1 :..m2 :..g2 :..m1 :..g3 :.r1 :.s :.s :..r1 :..p :.s :..n3 :.s :.s :.g1 :.s :..d3 :..n1 :..n1 :..n1 :..r1 :.s :.s :.g1 :.r3 :.g1 :.r1 :..d3 :..d3 :..d3 :.g1 :.r1 :.s :.g1 :..r2 :..r1 :.s :.r3 :..n3 :..d3 :..d3 :..n1 :..n1 :..n1 :..n1 :..n1 :..p :.s :..n1 :..g2 :..n3 :.r1 :.g3 :.g3 :.m1 :.r3 :.m1 :.g3 :.g3 :.p :.m1 :.m1 :..m1 :..g3 :.m1 :..r2 :..r2 :..n3 :.s :.s :.g1 :.g3 :.m2 :.p :.d2 :.m2 :.m1 :.r3 :.r3 :.g1 :.g1 :.g1 :.r1 :..n1 :.r3 :.g3 :.s :.s :.r1 :.g1 :.r1 :..n3 :..n1 :..d3
:..d3 :..n1 :..d3 :..n1 :..n1 :..n1 :..r1 :.s :..n1)
Raga Identification
Goodness of fit test
:base mohanam-base :samples mohanam-files (12.39 3.84 11.14 6.46 9.88 7.02 9.41 12.61 13.22 1.58)
:base mohanam-base :samples kalyani-files (10.95 28.66 25.61 15.26 27.32 21.53 16.42 18.58 24.80 23.80)
:base mohanam-base :samples revati-files (46.56 57.19 65.69 55.21 38.61 78.10 56.27 42.99 70.92 58.39)
Raga Identification
Revati sample vs
Revati base
Mohana sample vs
Revati base
What nextModel insights as melodic abstractions
Use synthesis models with generative music Experiment with Rhythm Synthesise Human Voice
Deep learning (Recurrent variational auto encoders)
Is this music though?
Behag
Dasarapada Abhishek Raghuram
Making Machines that Make Music
Srihari Sriraman nilenso
