ListentoYourData:TurningChemicalDynamicsSimulationsintoMusicAustinAtsango1 (atsango),SorenHolm1 (sorenh),andK.GraceJohnson1 (kgjohn)1DepartmentofChemistry,StanfordUniversity

Abstract

References

Datasets

DiscussionModels

FutureWork

GMM

Results

LSTMRNN

SoftmaxRegression

Music

Chemicaldynamics

Mostpianopieceshavemelodiesinpitchrange50-90

0

10

20

30

40

50

60

70

80

90

100

Softmax GMM LSTM

%correctlyiden

tified

Turning Chemical Dynamics Simulations into Music

Figure 12. Normalized confusion matrix for the LSTM

likely predicted a note very close in pitch to the true note.

5.2. Survey

Our goals in this project were to 1) generate music, and2) have that music reflect a given trajectory. To analyzethe success of our models in these two goals, we created asurvey asking participants first to decide whether a givenaudio clip was composed by a human or generated by acomputer. Results from the Turing test portion of the surveycan be seen in Table 1.

Table 1. Turing test for generated music: results of a survey with40 participants showing % responding the sample was composedby a human.

GENERATED

LSTM FULL 56%LSTM SUBSET 64%

GENERATED BASED ON TRAJECTORY

SOFTMAX REGRESSION 13%GMM 45%LSTM SUBSET 42%

REAL MUSIC (CONTROL) 57%

First, we generated music directly from the LSTM withouttrajectory information to see if we could simply generaterealistic music (first two rows of Table 1). Both LSTMsare the same architecture, but one was trained on randomsamples from the full musical data set, while the other wastrained on a subset: all pieces by the composer Clementi,whose melodies are regular and all occur in the right handof the piano (i.e. they are conducive to the format of oursimplified MIDI data). We see the samples generated fromthe subset LSTM pass the Turing test more successfullythan the LSTM trained on the full data set. This could bean issue of not having a model complex enough to handlethe variability in melodies, but also could be a by-productof our automated music simplification process, which does

not perfectly select for a melody out of a musical piece.Because our goal is to generate believable music, we usedthe subset LSTM to generate music based on trajectories.

Comparing the audio clips generated based on the a trajec-tory, the softmax regression model produced samples thatwere least believable as human-composed, followed by theLSTM, then the GMM. The GMM is the highest likely be-cause the snippets from which it was generated were part ofa human-composed piece (a Clementi melody).

The LSTM-generated samples based on the trajectory areless believable as human composed than the samples notbased on the trajectory. However, they still pass the Turingtest 42% of the time, a moderate success especially consider-ing the real music only passes 57% of the time. This bringsup an issue for future consideration: rhythm, tempo, andchords must be very important to the musicality of a sample,as when we remove them in the controls, the samples do notsafely pass the Turing test.

Figure 13. Trajectory and resulting LSTM-generated sample.

To assess the second goal, we asked participants to listen toa given audio clip and attempt to match it to the trajectoryfrom which it was generated. Similar questions were usedby Middleton to analyze the sonification of protein foldingdata1 Participants would see a plot like the black line inFigure 13, and hear an audio clip corresponding to the redline. Figure 13 shows a generated sample from the LSTM. Inthe survey, 88% of participants matched the LSTM sampleto the correct trajectory, 53% for the GMM, and only 25%for the softmax regression. According to this survey, ourLSTM model was the most successful in achieving Goal 2.

6. Conclusions and Future Work

Of all models tested, the LSTM RNN was most successfulat generating music that reflected trends in a given dynamicstrajectory. Softmax regression produced samples with thesame note repeated, which were neither musical nor reflec-tive of trajectory data. The GMM approach had roughly thesame success as the LSTM, but cannot truly be consideredmusic generation, as it sampled snippets from composedpieces. It should be noted that to more fully analyze the

5

Future efforts include curating a larger dataset withdistinctive melodies and exploring other generativemodels such as GANs or GRUs. The control of theTuring test shows that reducing a piece to simplypitch and time removes much of the musicality. Wewould also want to extend the model to train notjust on pitch, but also on rhythm, chords, and otherexpressive information, then explore methods ofinterpreting the trajectory data with theseadditional features.

We explored training predictive models with severalarchitectures and on several subsets of the musicdata. We found the best training and validationaccuracy using a subset of the full dataset: thepieces composed by Clementi.

Of all models tested, the LSTM RNN was mostsuccessful at generating music that reflected trendsin a given dynamics trajectory. Softmax regressionproduced samples with the same note repeated,which were neither musical nor reflective oftrajectory data. The GMM approach had roughly thesame success as the LSTM, but cannot truly beconsidered music generation, as it sampled snippetsfrom composed pieces. To more fully analyze thesuccess of the models in achieving both goalsoutlined, we would need a survey with a muchlarger sample size both in number participants andnumber of audio clips.

Turning Chemical Dynamics Simulations into Music

Figure 2. MIDI file representation in GarageBand

this usually contains the melody of the song. Once we havethe single stream, we replace chords with notes as follows:if several notes are played at the same time in a chord, wetake the note with the highest pitch. This leaves us with arelatively simple melody sequence as shown in Figure 3:

Figure 3. Simplified version of MIDI file

The notes are then mapped to integer values and representedas a 1D numpy array. The values range from 0 to 127. Theinteger values were normalized to fall between 0 and 1. Asa final step, these arrays have to be converted into examplesthat can be fed into a learning model. As such, the 1D musi-cal pieces are split into sequences of 50 consecutive notes.This was achieved by a convolutional approach, where a 1Dwindow of length 50 was slid across a musical piece, and a50-note sequence was captured at each step. These 50-notesequences served as the features while the note that cameimmediately after a sequence served as the correspondinglabel. This process yielded 276677 examples for the clas-sical piano set and 33500 examples for the Final Fantasyset. During training, we used an 80:20 training/validationsplit. The goal of our model would then be to predict thenext note given a sequence of 50 notes. The length of oursequence (50) was chosen as a compromise between accu-racy (one would expect the model to predict better if given alonger note sequence) and computational cost (longer notesequences require more resources to train). Because thenumber of notes is finite, this was treated as a classificationproblem, and the labels were one-hot encoded.

Chemical dynamics data was generated in the researchgroup of one of the authors,2 and consists of a set of tra-jectories with potential energies of the ground and excitedstate at every timestep of the simulation. Trajectory data isnormalized such that the minimum and maximum valuesmatch the lowest and highest pitches seen in most songs:50 and 90 on the pitch scale. While the simulations recordevents on the femtosecond timescale, we simply represent

each timestep as one note. Figure 4 shows an example of anormalized ground state trajectory.

Figure 4. Chemical dynamics simulation of ground state potentialenergy of stilbene.

4. Methods

First we discuss classification with softmax regression andan LSTM RNN (supervised), then we discuss unsupervisedclassification of trajectory data with a GMM.

4.1. Softmax Regression

Because this is a classification problem, we used softmaxregression as a baseline. In this model, the response variabley is assumed to be drawn from a multinomial distributionwhere each possible yi value has a corresponding probability�i. The goal, then, is to maximize the log-likelihood:

l(✓) =nX

i=1

logkY

l=1

exp(✓>l x

(i))Pk

j=1 exp(✓>j x

(i))

!1{y(i)=l}

Where p(y = i|x) = �i is given by:

p(y = i|x) = �i =exp(✓>i x)Pkj=1 exp(✓

>j x)

4.2. LSTM-RNN

Simple Recurrent Neural Networks (RNNs) are useful formodeling sequences and are therefore a natural fit for mu-sic generation. This is because they are able to retain a”memory” of previous inputs by looping the output from aprevious step (ht�1) back into the network with the currentstep’s input xt.

The current output ht is then evaluated based on both xt andht�1. Simple RNNs perform very well in the short term buthave problems modeling long-term dependencies in data.

Long Short-Term Memory (LSTMs) are much better suitedfor handling long-term dependencies. The basic structure isshown in Figure 5. LSTMs work by maintaining a memoryct and using gates to determine what information to store ordiscard. To this effect, LSTMs have three gates: the input,forget, and output gates.

2

Turning Chemical Dynamics Simulations into Music

Figure 5. LSTM architecture12

The LSTM first decides what information to discard byusing the forget gate. This gate multiplies the LSTM statect�1 by the sigmoid-activated vector shown below. Valuesof 0 mean ”completely forget” while 1 means ”completelykeep”

ft = �(Wf · [ht�1, xt] + bf ),

where xt is the current input and ht�1 is the hidden state.

The input gate then decides what new information to addto ct�1. This gate contains both a sigmoid layer to decidewhat values to update and a tanh layer to create new valuesfor the LSTM state. These two layers are multiplied beforebeing added to the LSTM.

it =�(Wi · [ht�1, xt] + bi)

Ct =tanh(Wc · [ht�1, xt] + bC)

Finally, the output gate then decides what value ht is goingto be output. The output value depends on the hidden stateht�1, the input xt, and the current LSTM state ct. It isevaluated as follows:

ot = �(Wo[ht�1, xt] + bo)

ht = ot ⇥ tanh(Ct)

4.3. Music Generation

With the neural networks trained to generate music in acertain style, we needed a way to bias the music generationtowards fitting the trajectory data. This required that wefound a way to influence the prediction of the network atprediction time. It is the case that the RMS distance betweentwo consecutive notes in our dataset is only 3.80 in the pitchspace of 0 to 127. It is likely that the trained model willhave learned this relationship, and this we can exploit to

push the predicted note towards the trajectory by giving thetrajectory value as the last element of the input sequence.Ideally, the network would come up with a compromise notethat is close to the trajectory but still musical in accordancewith its training.

Figure 6. Scheme for generating music based On trajectory

To start the music generation, we picked a random trainingexample from the validation set with the last note exchangedfor the first point in the trajectory data. The sequence wasthen fed into the trained model to generate the first note. Themusic generation was propagated as illustrated in Figure 6,by concatenating the last 49 predicted notes with the nextdata-point in the trajectory and then predicting the next notewith the model.

4.4. Gaussian Mixture Model

We would like to compare music generated with the abovepredictive models to something more straightforward andnaive. We would also like to characterize motifs within thetrajectories. For this we use a Gaussian Mixture Model, andcluster snippets from all trajectories using the Expectation-Maximization (EM) algorithm. This model involves maxi-mizing the log-likelihood of the data by alternately updatingQ(z) = p(z|x; ✓) and ✓. This has the effect of updatingand then maximizing the lower bound for the log-likelihoodgiven by:

ELBO(Q, ✓) =X

i

X

z(i)

Qi(z(i)) log

p(x(i), z

(i); ✓)

Qi(z(i))

5. Results and Discussion

5.1. Music Generation

5.1.1. SOFTMAX REGRESSION

As mentioned above, our baseline approach was to modelthe note sequences as 50-dimensional vector inputs and traina softmax regression to predict the next note given a vector

3

Turning Chemical Dynamics Simulations into Music

Figure 5. LSTM architecture12

The LSTM first decides what information to discard byusing the forget gate. This gate multiplies the LSTM statect�1 by the sigmoid-activated vector shown below. Valuesof 0 mean ”completely forget” while 1 means ”completelykeep”

ft = �(Wf · [ht�1, xt] + bf ),

where xt is the current input and ht�1 is the hidden state.

The input gate then decides what new information to addto ct�1. This gate contains both a sigmoid layer to decidewhat values to update and a tanh layer to create new valuesfor the LSTM state. These two layers are multiplied beforebeing added to the LSTM.

it =�(Wi · [ht�1, xt] + bi)

Ct =tanh(Wc · [ht�1, xt] + bC)

Finally, the output gate then decides what value ht is goingto be output. The output value depends on the hidden stateht�1, the input xt, and the current LSTM state ct. It isevaluated as follows:

ot = �(Wo[ht�1, xt] + bo)

ht = ot ⇥ tanh(Ct)

4.3. Music Generation

With the neural networks trained to generate music in acertain style, we needed a way to bias the music generationtowards fitting the trajectory data. This required that wefound a way to influence the prediction of the network atprediction time. It is the case that the RMS distance betweentwo consecutive notes in our dataset is only 3.80 in the pitchspace of 0 to 127. It is likely that the trained model willhave learned this relationship, and this we can exploit to

push the predicted note towards the trajectory by giving thetrajectory value as the last element of the input sequence.Ideally, the network would come up with a compromise notethat is close to the trajectory but still musical in accordancewith its training.

Figure 6. Scheme for generating music based On trajectory

To start the music generation, we picked a random trainingexample from the validation set with the last note exchangedfor the first point in the trajectory data. The sequence wasthen fed into the trained model to generate the first note. Themusic generation was propagated as illustrated in Figure 6,by concatenating the last 49 predicted notes with the nextdata-point in the trajectory and then predicting the next notewith the model.

4.4. Gaussian Mixture Model

We would like to compare music generated with the abovepredictive models to something more straightforward andnaive. We would also like to characterize motifs within thetrajectories. For this we use a Gaussian Mixture Model, andcluster snippets from all trajectories using the Expectation-Maximization (EM) algorithm. This model involves maxi-mizing the log-likelihood of the data by alternately updatingQ(z) = p(z|x; ✓) and ✓. This has the effect of updatingand then maximizing the lower bound for the log-likelihoodgiven by:

ELBO(Q, ✓) =X

i

X

z(i)

Qi(z(i)) log

p(x(i), z

(i); ✓)

Qi(z(i))

5. Results and Discussion

5.1. Music Generation

5.1.1. SOFTMAX REGRESSION

As mentioned above, our baseline approach was to modelthe note sequences as 50-dimensional vector inputs and traina softmax regression to predict the next note given a vector

3

Turning Chemical Dynamics Simulations into Music

Figure 5. LSTM architecture12

The LSTM first decides what information to discard byusing the forget gate. This gate multiplies the LSTM statect�1 by the sigmoid-activated vector shown below. Valuesof 0 mean ”completely forget” while 1 means ”completelykeep”

ft = �(Wf · [ht�1, xt] + bf ),

where xt is the current input and ht�1 is the hidden state.

The input gate then decides what new information to addto ct�1. This gate contains both a sigmoid layer to decidewhat values to update and a tanh layer to create new valuesfor the LSTM state. These two layers are multiplied beforebeing added to the LSTM.

it =�(Wi · [ht�1, xt] + bi)

Ct =tanh(Wc · [ht�1, xt] + bC)

Finally, the output gate then decides what value ht is goingto be output. The output value depends on the hidden stateht�1, the input xt, and the current LSTM state ct. It isevaluated as follows:

ot = �(Wo[ht�1, xt] + bo)

ht = ot ⇥ tanh(Ct)

4.3. Music Generation

With the neural networks trained to generate music in acertain style, we needed a way to bias the music generationtowards fitting the trajectory data. This required that wefound a way to influence the prediction of the network atprediction time. It is the case that the RMS distance betweentwo consecutive notes in our dataset is only 3.80 in the pitchspace of 0 to 127. It is likely that the trained model willhave learned this relationship, and this we can exploit to

push the predicted note towards the trajectory by giving thetrajectory value as the last element of the input sequence.Ideally, the network would come up with a compromise notethat is close to the trajectory but still musical in accordancewith its training.

Figure 6. Scheme for generating music based On trajectory

To start the music generation, we picked a random trainingexample from the validation set with the last note exchangedfor the first point in the trajectory data. The sequence wasthen fed into the trained model to generate the first note. Themusic generation was propagated as illustrated in Figure 6,by concatenating the last 49 predicted notes with the nextdata-point in the trajectory and then predicting the next notewith the model.

4.4. Gaussian Mixture Model

We would like to compare music generated with the abovepredictive models to something more straightforward andnaive. We would also like to characterize motifs within thetrajectories. For this we use a Gaussian Mixture Model, andcluster snippets from all trajectories using the Expectation-Maximization (EM) algorithm. This model involves maxi-mizing the log-likelihood of the data by alternately updatingQ(z) = p(z|x; ✓) and ✓. This has the effect of updatingand then maximizing the lower bound for the log-likelihoodgiven by:

ELBO(Q, ✓) =X

i

X

z(i)

Qi(z(i)) log

p(x(i), z

(i); ✓)

Qi(z(i))

5. Results and Discussion

5.1. Music Generation

5.1.1. SOFTMAX REGRESSION

As mentioned above, our baseline approach was to modelthe note sequences as 50-dimensional vector inputs and traina softmax regression to predict the next note given a vector

3

Surveyresultswith40participantsshowing%respondingthesamplewascomposedbyahuman.

Our goal is to translate simulation data into a musicalform in order to present a different way to interact withdata. Specifically, the goals are 1) to generate music, i.e.melodies that are indistinguishable from those composedby humans, and 2) to have those melodies reflect trendsin the underlying data.

We take two approaches: 1) We use a supervised model(either softmax regression or an LSTM RNN trained oncomposed melodies) to predict the next note in a song,biased by the trajectory values. 2) We cluster snippets of atrajectory using a Gaussian Mixture Model (GMM) withthe EM algorithm to discover motifs within a trajectory,then match these motifs to similar ones from a composedmelody.

We evaluate the success of these approaches with asurvey designed to assess the two goals of the project.

Musicgenerationfromtrajectory:

Goal1:Turingtest Goal2:matchinggeneratedmusictotrajectory

Percentageofparticipantsmatchingcorrecttrajectory

[1]Weir,H.,Williams,M.,Parrish,R.,andMartinez,T.J.Nonadiabatic dynamicsofphotoexcited cis-Stilbeneusingabinitiomultiplespawning.Inprep.(2019).[2]Classicalpianomidipage.Retrievedfromhttp://www.piano-midi.de/[3]Skli,Sigurur.HowtoGenerateMusicUsingaLSTMNeuralNetworkinKeras.DataScience,7Dec.2017.Retrievedfromtowardsdatascience.com[4]Holzner,A.LSTMCellsinPytorch.Retrievedfromhttps://medium.com/@andre.holzner/lstm-cells-in-pytorch-fab924a78b1c

• MIDIdataformat• 312classicalpianopieces• 93pianopiecesfromFinalFantasyvideogame• Simplifiedusingmusic21andmido packagesin

Pythontorepresentaspitch(withvalue0to127)vstime

• Quantumdynamicssimulationsofstilbenedecayingfromexcitedtogroundstate

• 200trajectoriesofpotentialenergyvs.time(femtoseconds)

• Potentialenergynormalizedto50-90pitchrange(Z)-Stilbene

• useaGaussianMixtureModelwiththeEMalgorithmtoclustersnippetsofalltrajectoriesbasedondistanceandgradient

• Matchsnippetstomotifsinagivenmusicalpiece

Architecture:• LSTMwith256hiddenunits• LSTMwith38hiddenunits• Denselayerwithsoftmax

activation

• Convolutionovereachmusicalpiece• One-hotencoding• Supervised:predictnextnotebasedonprevious50

notes

PopulationoftrajectorysnippetsforEMwith16GMMmeans

Predictivemodels