Acceleration and Quantitation of LocalizedCorrelated Spectroscopy using Deep Learning: APilot Simulation StudyZohaib Iqbal1, Dan Nguyen1, M. Albert Thomas2, and Steve Jiang1,*

1Medical Artificial Intelligence and Automation Laboratory, Department of Radiation Oncology, University of TexasSouthwestern Medical Center, Dallas, TX, USA 753902Department of Radiological Sciences, University of California Los Angles, Los Angeles, CA, USA*Corresponding author contact: steve.jiang@utsouthwestern.edu

ABSTRACT

Nuclear magnetic resonance spectroscopy (MRS) allows for the determination of atomic structures and concentrations ofdifferent chemicals in a biochemical sample of interest. MRS is used in vivo clinically to aid in the diagnosis of severalpathologies that affect metabolic pathways in the body. Typically, this experiment produces a one dimensional (1D) 1H spectrumcontaining several peaks that are well associated with biochemicals, or metabolites. However, since many of these peaksoverlap, distinguishing chemicals with similar atomic structures becomes much more challenging. One technique capableof overcoming this issue is the localized correlated spectroscopy (L-COSY) experiment, which acquires a second spectraldimension and spreads overlapping signal across this second dimension. Unfortunately, the acquisition of a two dimensional(2D) spectroscopy experiment is extremely time consuming. Furthermore, quantitation of a 2D spectrum is more complex.Recently, artificial intelligence has emerged in the field of medicine as a powerful force capable of diagnosing disease, aidingin treatment, and even predicting treatment outcome. In this study, we utilize deep learning to: 1) accelerate the L-COSYexperiment and 2) quantify L-COSY spectra. All training and testing samples were produced using simulated metabolite spectrafor chemicals found in the human body. We demonstrate that our deep learning model greatly outperforms compressed sensingbased reconstruction of L-COSY spectra at higher acceleration factors. Specifically, at four-fold acceleration, our method hasless than 5% normalized mean squared error, whereas compressed sensing yields 20% normalized mean squared error. Wealso show that at low SNR (25% noise compared to maximum signal), our deep learning model has less than 8% normalizedmean squared error for quantitation of L-COSY spectra. These pilot simulation results appear promising and may help improvethe efficiency and accuracy of L-COSY experiments in the future.

IntroductionMagnetic resonance imaging (MRI) is a popular imaging modality capable of providing valuable anatomical and functionalinformation in vivo. By utilizing a strong magnetic field and radio-frequency (RF) waves, MRI successfully images hydrogenatoms in their local chemical environment, allowing for useful soft tissue contrast. One technique that allows for the metabolicinvestigation of different tissues is the magnetic resonance spectroscopy (MRS) method. In particular, single-voxel 1H MRSis capable of providing biochemical information from a volume of interest (VOI) in the human body1. MRS provides a1H spectrum rich with peaks representative of various chemicals. Furthermore, this spectrum can be quantified by using aspectral fitting algorithm2–6 to yield chemical, or metabolite, concentrations. MRS, and more specifically the point resolvedspectroscopy (PRESS) experiment, has been used to explore pathologies affecting the brain7, prostate8, liver9, breast10, aswell as other sites, and is often used in combination with other imaging studies to discern how metabolic alterations in tissuescorrelate with anatomical abnormalities.

Unfortunately, one-dimensional (1D) spectroscopy techniques such as PRESS have a disadvantage when it comes toquantifying overlapping metabolite spectral signals. Since many metabolites are found in the body at very low concentrations,separating these signals from more dominant spectral peaks becomes very challenging. For this reason, several approaches havebeen developed to better quantify these lower concentrated metabolites, including J-editing techniques11–14 and two-dimensional(2D) spectral acquisitions15–19. In particular, 2D MRS offers the advantage of quantifying all metabolite signals in a single scanat the expense of increasing acquisition time. A typical 2D MRS experiment includes a time increment, t1, in the pulse sequenceto acquire data from the indirect temporal dimension. Combined with the acquisition of the direct temporal dimension, t2, a 2Dspectrum, S(F2,F1), can be acquired by Fourier transforming the 2D temporal data, s(t2,t1).

arX

iv:1

806.

1106

8v1

[ph

ysic

s.m

ed-p

h] 2

8 Ju

n 20

18

Figure 1. Two proposed implementations for the D-UNet architecture are shown. A) A non-uniformly sampled L-COSY experiment isreconstructed into the fully sampled spectrum. While under-sampling can be performed in both the t2 and t1 dimensions, this study analyzesthe reconstruction of L-COSY spectra acquired using non-uniform sampling along only the t1 dimension. B) Several metabolite spectra areidentified from a fully sampled L-COSY spectrum using a D-UNet model. The intensities of the metabolite spectra directly correlate toconcentration values, and therefore this study also investigates the potential application of deep learning to quantify L-COSY spectra. In total,17 metabolites were quantified in this simulation study.

One popular 2D MRS technique is the localized correlated spectroscopy (L-COSY) experiment18. This experimentacquires data by using a 90◦-180◦-t1-90◦-t2 sequence and yields several cross-peaks which can be used to identify and quantifyoverlapping resonances. However, there are two main limitations of the L-COSY technique. First, due to the t1 incrementnecessary to obtain the indirect dimension, the L-COSY scan time is very long. Second, because of the nature of an additionaldimension, spectral fitting becomes more complex and therefore less ideal quantitation techniques such as peak integrals areoften used. Several methods have been proposed to overcome these two challenges to improve L-COSY, including non-uniformsampling with reconstruction20 and 2D spectral fitting using prior-knowledge21, 22.

Recently, deep learning and artificial intelligence have become more prominent in the medical field and radiology23–26.These methods are often used for segmenting medical images, aiding with diagnosis, and verifying image quality. One populardeep learning architecture is the UNet26, which is a fully convolutional network27 capable of image-to-image domain mapping.While UNet is often used for segmentation purposes, our group has recently demonstrated that a novel UNet architecture, thedensely connected U-Net (D-UNet)28–30, is capable of reconstructing super-resolution spectroscopic images. In this study, wedemonstrate that the D-UNet architecture can be used to: 1) reconstruct non-uniformly sampled (NUS) L-COSY acquisitionsand 2) quantify fully sampled L-COSY spectra accurately. The D-UNet models were trained and evaluated using simulatedL-COSY data. The first type of D-UNet model was trained to reconstruct NUS L-COSY. This reconstruction method wasquantitatively compared to compressed sensing (`1-norm) reconstruction31. The second type of D-UNet model was trained toquantify seventeen metabolites from a simulated fully sampled L-COSY spectrum. All reconstruction results were compared tothe actual simulations to evaluate the errors of the reconstructions both qualitatively and quantitatively.

Methods

As shown in Figure 1, the goal of this study was to perform two distinct tasks using the D-UNet architecture: 1) reconstructNUS L-COSY spectra and 2) quantify L-COSY spectra. While each task used different data for training the models and testingthe results, the initial simulation process to synthesize L-COSY spectra was identical for both applications.

2/17

SimulationGAMMA simulation32 was used to simulate seventeen different metabolites found in the human brain using the 90◦-180◦-t1-90◦-t2 L-COSY sequence18. These metabolites included aspartate (Asp), choline (Ch), creatine at 3ppm (Cr3.0), creatine at3.9ppm (Cr3.9), γ-butyric acid (GABA), glucose (Glc), glutamine (Gln), glutamate (Glu), glutathione (GSH), lactate (Lac),myo-Inositol (mI), N-acetyl aspartate (NAA), N-acetyl-asparate-g (NAAG), phosphocholine (PCh), phosphoethanolamine (PE),taurine (Tau), and threonine (Thr). Chemical shift values for the biochemicals were found in the literature33. The metaboliteswere simulated using the following experimental parameters: TE=30ms, t2 points = 2048, t1 points = 100, spectral bandwidthalong the direct dimension (SBW2) = 2000Hz, and spectral bandwidth along the indirect dimension (SBW1) = 1250Hz. Themagnetic field strength was chosen to be the field strength of a Siemen’s 3T scanner (Erlangen, Germany).

Then, L-COSY spectra were randomly generated by modifying the original metabolite simulations, also referred to as thebasis set. Each metabolite in the basis set (Bm) was first line broadened in both the direct and indirect temporal dimensionsusing an exponential filter and a random phase was applied to the basis metabolite signal as well:

Blb,m = Bme−r1,me−r2,me−iφr (1)

Above, Blb,m is the new line-broadened metabolite, φr is a random angle between 0 and 2π , e−r1,m is an exponential filterapplied to the t1 domain, and e−r2,m is an exponential filter applied to the t2 domain. Each metabolite was allowed to haveseparate line-broadening terms. The factors e−r2,m and e−r1,m resulted in effective line-broadenings of 5-25Hz and 0-15Hz,respectively, and were implemented in this fashion to mimic the range of common T2 values in vivo.

Next, the individual metabolites were combined linearly using random concentration values to produce an initial L-COSYspectrum, sinit :

sinit = ∑m

r3,mBlb,m (2)

In equation 2, r3,m is a random concentration value between 0 and 10, and is representative of the concentration value inmmol. The final L-COSY spectrum, s f , was created by adding noise to sinit . The noise level could vary drastically from 0% to25% of the maximum metabolite signal.

Non-uniform Sampling and ReconstructionNon-uniform sampling was performed on the final s f matrix along the t1 dimension utilizing an exponential probability densityfunction34–36. This NUS scheme emphasized sampling earlier t1 points more due to the fact that these points have less T2 decay(more signal). The last t1 point was sampled for all of the NUS schemes. The three sampling masks used in this study aredisplayed in Figure 2. A t1 point was sampled if the value in the mask was 1, and it was not sampled if the value in the maskwas 0. The number of points sampled for each mask were 75, 50, and 25 resulting in a scan acceleration factor of 1.3x, 2x, and4x, respectively.

Aside from the D-UNet reconstruction of NUS data described below, data were also reconstructed using compressed sensingreconstruction31. The `1-norm minimization reconstruction was performed by solving the following optimization problem:

minimizeu

||u||1

subject to ||MFu− f ||22 ≤ σ2(3)

Equation 3 is the general formulation for compressed sensing reconstruction. u is the reconstructed data in the (F2,F1)spectral domain, M is the sampling mask along the t1 domain, F is the 2D Fourier transformation, f is the NUS data in the(t2,t1) temporal domain, and σ2 is the estimate of the noise variance. The noise variance was estimated from a noisy region ofthe spectrum, as previously described35, 37–39.

Reconstructing NUS L-COSY with D-UNetThe densely connected UNet architecture utilized in this study was very similar to a previously reported model29, and thegeneral architecture can be seen in figure 3. This model utilized the generic UNet architecture, which operates by learningimportant global and local features using a variety of convolutional layers. The first half of the UNet continuously usesconvolutional and max pooling layers, and these layers help reduce the input matrix size. By reducing the size, the networklearns the primary global features of the input images. The second half of the UNet uses deconvolutional and up-pooling layers,which restore the matrix size. This process helps learn local features that are vital to restoring the images on a finer scale. The

3/17

Figure 2. A ground truth simulated L-COSY spectrum is shown (top). Sampling schemes were applied to the simulated spectrum using thesampling masks shown in the 1st column. These masks sampled 25, 50, and 75 t1 points out of a total 100 t1 points to yield 4x, 2x, and 1.3xacceleration factors, respectively. The 2nd column shows the under-sampled spectra in the (F2,F1) domain and the 3rd column shows thespectra reconstructed using a D-UNet model. Errors for each reconstruction are displayed as difference maps in the final column.

4/17

Figure 3. The densely connected U-Net architecture from a previous publication29 is displayed. The densely connected flavor of thismodel allows for important features to be carried over throughout the entire training process.

architecture also leveraged densely connected convolutional layers, which aid in carrying important features throughout thelearning process. All convolutional layers used a kernel size = 3 x 3, stride = 1, and a rectified linear unit (ReLU) activationfunction40.

The D-UNet model used for reconstructing the NUS L-COSY data was designed to take an NUS L-COSY spectrum asinput and produce a reconstructed L-COSY spectrum as output. The NUS L-COSY data was produced by multiplying s f by thesampling mask in the (F2,t1) domain and then transforming this matrix back into the (F2,F1) domain. The output was simplythe s f matrix without noise in the (F2,F1) domain. Both the input and output matrix sizes were 512 x 32, and corresponded tospectral ranges of 0.5-4.5ppm in the direct spectral dimension (F2), and 1.2-4.3ppm in the indirect spectral dimension (F1).Additionally, the inputs and outputs were inserted as three different channels into the network with each channel representingthe real, imaginary, and magnitude information of the spectrum. Finally, all inputs and outputs were normalized to be inbetween values of 0 and 1, and were normalized based on the maximum value of the magnitude images. The loss function wasthe mean squared error (MSE) between the reconstructed L-COSY (Recon) and the actual simulated L-COSY (Actual), whichwas defined as:

MSE = ∑F2

∑F1

(Recon−Actual)2

512∗32(4)

The Adam optimizer41 was used with a learning rate set to 1e−3. Three D-UNet models with identical architecture weretrained to reconstruct spectra sampled using the masks shown in figure 2. A total of 40,000 simulated NUS L-COSY spectrawere simulated for each sampling scheme, and 100 spectra were used to evaluate the results as an independent test set. Thebatch size for the training was 10 samples per batch.

Quantitation of L-COSY with D-UNetThe quantitation of fully sampled L-COSY data was performed in a similar manner to the method described above. The inputto the quantitation D-UNet was the L-COSY spectrum as a 512 x 32 matrix with three channels representing the magnitude,real, and imaginary components of the spectrum. The input was scaled from 0 to 100 based on the maximum of the magnitudespectrum. The output of the network was a 512 x 32 matrix representative of each metabolite basis set. Therefore, since 17metabolites were quantified, the output had 17 channels representing the magnitude spectrum for each metabolite. All othertraining parameters were identical to those described above. A total of 21,000 simulated L-COSY spectra were used for training,and 100 spectra were used for testing the results independently.

5/17

Figure 4. A qualitative comparison between the D-UNet and the compressed sensing (`1-norm) reconstructions is shown. The fullysampled L-COSY spectrum displayed in figure 2 was sampled using 25 t1 points (4x acceleration). The spectrum was then reconstructedusing the trained deep learning model and optimization described in equation 3. Errors between the two reconstructions are displayed asdifferences between the actual spectrum and the reconstructions.

EvaluationAll of the results were compared to the actual simulated spectra by utilizing the MSE metric from equation 4. For thenon-uniformly sampled spectral reconstruction, the MSE was calculated over all 100 test spectra and compared to the MSEof the `1-norm reconstruction from equation 3 for all acceleration factors. Normalized MSE was also used, and errors werenormalized based on the maximum signal intensity of the spectrum. In addition to MSE, the quantitation with D-UNet alsoinvestigated the effect of noise on the quantitative results. Specifically, ten different noise levels were evaluated on the same100 spectra to determine how the signal-to-noise ratio (SNR) affects the model results and overall stability. These noise levelsranged from 0% to 25% of the maximum signal intensity.

Results

NUS L-COSY reconstructionThe NUS L-COSY spectra reconstructed using the D-UNet architecture can be seen in figure 2. The non-uniform samplingproduces several F1 ridging artifacts present in the spectral domain, which are ultimately removed by using the D-UNet models.For training, the MSE loss function achieved a loss of approximately 3e−5 for each of the models. The errors as the differencebetween the Actual and Recon spectra are also shown for each acceleration factor.

A qualitative comparison between the D-UNet reconstruction and `1-norm minimization methods are shown in figure4 for the 4x reconstructions. While the D-UNet reconstruction displays minimal errors surrounding the major peaks, thecompressed sensing results show large errors. Due to the iterative reconstruction, several false cross-peaks also appear in the`1-norm reconstructed spectra, which are not present in the D-UNet reconstruction. Also, a quantitative comparison betweenthe two reconstruction methods is provided in table 1. At lower acceleration factors where more points are sampled, `1-norm

6/17

Acceleration D-UNet `1

1.3x 0.0388 0.01922x 0.0170 0.06814x 0.0327 0.208

Table 1. Total mean squared error (MSE) over 100 testing spectra for each acceleration factor. A D-UNet model was trained to learnreconstruction for each sampling factor, and the results were compared to `1-norm reconstruction as described in equation 3. Since themaximum signal is 1 for all spectra analyzed, the normalized MSE as a percentage for the D-UNet is 3.88%, 1.70%, and 3.27% foracceleration factors of 1.3x, 2x, and 4x, respectively. Similarly, the MSE as a percentage for the `1-norm reconstruction is 1.92%, 6.81%, and20.8% for acceleration factors of 1.3x, 2x, and 4x, respectively.

minimization performs better than the D-UNet reconstruction. However, at higher acceleration factors where less points aresampled, the D-UNet mean error remains under 5%, whereas the `1-norm minimization reconstruction error is larger than 20%.Once again, these values were calculated over 100 testing L-COSY data that were simulated independently of the training set.

L-COSY quantitationThe capabilities of the D-UNet to identify metabolites from a given L-COSY spectrum are demonstrated in figure 5. Fromthe given L-COSY spectrum, 9 metabolite reconstructions are shown and compared alongside the simulated ground truthspectra: NAA, PCh, Cr3.0, mI, Gln, Glu, GABA, GSH, and Asp. In the example spectrum displayed, NAA was simulated at aconcentration level of approximately 8 mmol. For GSH, which was simulated closer to 1 mmol, the reconstruction results stillhave similar intensity values to the simulated ground truth. While only 9 metabolite reconstructions are shown, it is importantto note that all 17 metabolites in the basis set are reconstructed and could be visualized.

Of course, SNR can play a large role on the performance of any quantitation algorithm, and therefore errors resultingfrom high noise were investigated. Figure 6 displays the effect of noise levels on the calculated mean squared error for all 17metabolite spectral reconstructions. As expected, degrading SNR results in larger MSE values for quantitation. In addition, anexample spectrum is shown at two different noise levels: noise level 2 (5% noise) and noise level 8 (20% noise). It is clear thatcross-peak intensities vary largely with noise, due to the fact that cross-peaks are low signal peaks for the L-COSY experiment.

The linear relationships between the actual and predicted measurements for all 100 test spectra and 17 metabolites werealso analyzed. Figure 7 shows the linear relationships for 16 metabolites quantified from the test spectra with a noise level of5% of the maximum signal intensity. Linear fits are shown on the correlation plots between the simulated ground truth (Actual)and the reconstructed (Recon) concentrated values. In order to produce the concentration results for Recon, the maximumintensity was used from the individually reconstructed metabolite spectra from the D-UNet quantitation model.

Finally, table 2 compares the concentration values of the 17 metabolites at different SNR values. Ideally, if the quantitationwas perfect, the slope would be one and the standard error would be zero. For many metabolites at noise level=2, the slope anderror are close to ideal values. However, at noise level=8, slopes start to deviate largely from the ideal values and error alsoincreases. The r2 metric displayed is the coefficient of determination and is the variance of the fit. Overall, the r2 values showthat variance is low for the quantitative correlations at both noise levels, as demonstrated by r2 >0.8.

DiscussionFrom the results, it is clear that the D-UNet architecture is capable of both reconstructing non-uniformly sampled L-COSY dataand quantifying L-COSY spectra after appropriate training. Figures 2 and 5 show this qualitatively whereas tables 1 and 2 showthis quantitatively. While deep learning has very recently been used for quantitation of 1D MRS42, to our knowledge this is thefirst application of deep learning for reconstructing and quantifying L-COSY MRS. For reconstruction at high accelerationfactors, the D-UNet method greatly outperforms a standard compressed sensing method. Spectral quality plays a large rolein determining the outcome of the quantitation method, and poor spectral quality results in higher errors, as seen in figure 6.Even though the model architecture for both applications is identical, the two models learn separate properties of the L-COSYspectrum.

The first model, which reconstructs NUS L-COSY data, learns to remove the artifacts produced from the application ofa particular non-uniform sampling mask. Due to the non-uniform t1 sampling, various ridging artifacts are present in the

7/17

Figure 5. The results for the quantitation D-UNet are displayed for an example fully sampled L-COSY spectrum. From the input spectrum(top), the deep learning model reconstructs each metabolite’s magnitude spectrum individually (Recon). For comparison, the actual simulatedmagnitude spectra (Actual) are plotted alongside the reconstructed spectra with the same intensity windows. While only 9 metabolites aredisplayed, the D-UNet model produces 17 metabolite spectra. The concentrations for these spectra are proportional to the signal intensities,as is standard for most fitting algorithms.

8/17

Figure 6. The mean squared error is displayed as a function of noise level for the quantitation D-UNet results (left). These results wereproduced by analyzing MSE for 100 identical spectra at 10 different noise levels ranging from 0% - 25% noise relative to the maximumsignal intensity. Two example spectra are shown displying 5% noise (middle) and 20% noise (right). Qualitatively, it is clear that cross-peaksignal amplitude is greatly altered due to the added noise in the noise level = 8 spectrum.

Figure 7. The relationship between the actual metabolite concentration on the x-axis (Actual) and the reconstructed metaboliteconcentration on the y-axis (Recon) is shown for 16 metabolites for 100 test spectra. The spectra contained approximately 5% noise signalrelative to the maximum signal intensity (noise level = 2). Overall, most metabolites displayed an expected linear relationship even at lowerconcentration values. Quantitative values for these results are tabulated in table 2.

9/17

Noise Level = 2 Noise Level = 8Metabolite Slope r2 Std. Error Slope r2 Std. Error

Asp 0.939 0.997 0.00734 0.683 0.946 0.0235Ch 0.948 0.974 0.0224 0.747 0.842 0.0484

Cr3.0 0.970 0.998 0.00667 0.849 0.949 0.0285Cr3.9 1.02 0.974 0.0241 0.561 0.740 0.0516

GABA 0.915 0.989 0.0136 0.591 0.854 0.0363Glc 0.921 0.992 0.0120 0.804 0.927 0.0328Gln 1.04 0.997 0.00846 0.723 0.913 0.0327Glu 1.01 0.995 0.0101 0.873 0.933 0.0342GSH 0.936 0.940 0.0344 0.696 0.599 0.0941Lac 1.03 0.974 0.0242 0.617 0.806 0.0458mI 0.922 0.996 0.00832 0.833 0.974 0.0196

NAA 1.00 0.997 0.00759 0.853 0.958 0.0257NAAG 1.02 0.985 0.0183 0.824 0.870 0.0472

PCh 0.857 0.944 0.0303 0.639 0.842 0.0413PE 0.903 0.995 0.00932 0.767 0.959 0.0229Tau 0.860 0.984 0.0159 0.629 0.887 0.0331Thr 1.06 0.979 0.0222 0.782 0.819 0.0554

Table 2. A quantitative comparison between the quantitation results for two different noise levels is shown. A perfect quantitationalgorithm would produce the following results: slope = 1, r2 = 1, and standard error (Std. Error) = 0. From the results, it is clear that higherSNR spectra produce more accurate quantitative results.

F1 domain20. Depending on the sampling pattern, the artifacts will be mostly constant for each metabolite, but will still bea function of the metabolite concentration, line-broadening factor, and noise level. By providing enough example data, thenetwork essentially learns how to identify the ridging artifacts and remove them appropriately for a given sampling mask andbasis set. This is best illustrated in figure 2, where it is clear that ridging is removed in the reconstructed spectra for eachacceleration factor.

On the other hand, the second model that quantifies metabolite concentrations from L-COSY spectra learns a differentproperty of the L-COSY images. After adding all of the metabolites together to form a composite spectrum, several signalsoverlap and are hard to disentangle. Optimization problems are able to handle this issue by fitting overlapping peaks usingseveral parameters, often including appropriate prior-knowledge21, 22, 43. Unfortunately, these algorithms take a very long timeto calculate these parameters and often yield sub-par results if the quality of the L-COSY spectrum is low (high noise, lowSNR, signal contamination, etc.). The quantitation model learns how to disentangle overlapping signals through analysis of themagnitude, real, and imaginary components of the input spectrum. By training on thousands of data, the model learns whichsignals best represent each metabolite even if the signal is buried in another peak and noise. Furthermore, the calculation isextremely fast and is on the order of seconds for a single spectrum. In terms of accuracy, even lower concentrated metabolitessimulated at less than 1 mmol are accurate (error less than 3%) for most metabolites, as shown in figure 7. While these pilotresults look promising for reconstruction and quantitation of L-COSY spectra, the current implementation of this method hasseveral weaknesses.

First, the D-UNet model requires prior-knowledge for all metabolites present in the tissue for training as well as how these

10/17

signals are affected by a particular non-uniform sampling pattern. Compressed sensing reconstructions do not require anyspectral prior-knowledge, and therefore are more versatile for different sampling masks. This is not necessarily a weakness if:1) the sampling mask used for acquisition matches the D-UNet sampling mask used for training and 2) all metabolites in thetissue are known a priori. For most experiments, 1) is easily satisfied. For healthy tissues and well documented pathologies, 2)is not an issue. However, 2) may become an issue for pathologies that are not well understood and involve unknown chemicalchanges. This problem may be alleviated by including prior-knowledge for all metabolites appearing in the analysis of exvivo tissue samples of this pathology if available. For example, mass spectrometry of ex vivo tissues played a pivotal role inidentifying 2-hydroxyglutarate (2HG) in certain glioma patients as a metabolite of interest44. Additional prior-knowledge canalways be included into the training process to account for macromolecule signals or other signals that may be present in thespectrum retrospectively if necessary.

Another weakness of the current methodology is that water and fat contamination were not added to the training spectra.Due to water suppression pulses45, spectral distortions around the water region may affect metabolite quantitation. For 2Dexperiments, total removal of water signal while retaining metabolite signal is more challenging, and may affect the amplitudesof correlated cross-peaks close to water. This problem can be overcome through more advanced training, however the effectsof water suppression and removal through common methods such as singular value decomposition (SVD) have to be wellunderstood in order to be modeled correctly. Contaminating fat signal may affect quantitation of metabolites such as lactate andNAA, depending on severity. These fat signals can also be incorporated into the training process, however it is important toutilize the correct fat species.

The final weakness of the current methodology is the broadening model used to produce the training and testing data.Currently, only an exponential line-broadening term was used for this pilot study. While exponential line-broadening maybe a great first approximation for peak shapes, gaussian, lorentzian and even voigt lineshapes may be present in the finalexperimental peaks43. Due to the increased number of parameters introduced with these added lineshapes, the training data sizewould need to be much larger for adequate training of the model. In addition, the number of features present in the model mayneed to be increased in order to handle the complexity of the additional broadening parameters.

Even with these weaknesses, the methodology presented here can easily be applied to other 2D MRS experimentsand to iterative MRS experiments in general. The J-resolved spectroscopy (JPRESS) experiment is another useful 2D MRStechnique16, 17, and we have pilot results showing that these models apply for this type of experiment (Supplemental Information).Other 2D experiments include the nuclear overhauser effect spectroscopy (NOESY), total correlation spectroscopy (TOCSY),as well as others. Iterative MRS experiments include diffusion weighted spectroscopy46–48, J-editing spectroscopy, and anymulti-TE spectroscopy49. In addition, this methodology could be refined for the application of super-resolution spectroscopy,including covariance spectroscopy50, 51. However, super-resolution may be unnecessary if accurate quantitative results canalready be obtained from low resolution spectra.

Simulation results are certainly powerful for evaluating the feasibility of potential applications, and this study demonstratesthat the D-UNet is capable of reconstructing NUS L-COSY data and quantifying L-COSY spectra. However, these methodsneed to be further validated in vitro and in vivo. Also, these methods have to be compared to state of the art techniques for eachapplication. For reconstruction, the D-UNet model should ideally be compared to compressed sensing, maximum entropy39, 52,or other reconstruction methods. It is important to note that while many reconstruction methods require certain samplingschemes (random, non-uniform, etc.), the D-UNet is capable of reconstructing any sampling pattern with the correct trainingapproach. While an exponential sampling scheme was used in this study, a skewed-squared sine-bell sampling scheme maybe better to implement in the future39. For quantitation, it is important to compare the deep learning method to other 2D invivo fitting algorithms to assess accuracy and reproducibility22. After further validation, these models may easily be combinedtogether to create a single deep learning model capable of simultaneously reconstructing and quantifying L-COSY spectra.With further improvements, this method will hopefully have the same acquisition duration as a 1D single-voxel scan (3-5minutes), which will make this method extremely useful clinically for discerning overlapping metabolite signals.

ConclusionWe present a deep learning approach capable of reconstructing non-uniformly sampled L-COSY spectra and quantifying fullysampled L-COSY spectra. Overall, the results demonstrate accurate reconstruction and quantitation with normalized meansquared error less than 5% for most SNR levels. This technique was evaluated using simulated data, and further studies willvalidate this method for in vitro and in vivo measurements, and compare this method to state of the art techniques.

11/17

References1. Bottomley, P. A. Spatial localization in nmr spectroscopy in vivo. Annals of the New York Academy of Sciences 508,

333–348 (1987).

2. Provencher, S. W. Estimation of metabolite concentrations from localized in vivo proton nmr spectra. Magnetic Resonancein Medicine 30, 672–679 (1993).

3. Ratiney, H. et al. Time-domain semi-parametric estimation based on a metabolite basis set. NMR in Biomedicine 18, 1–13(2005).

4. Naressi, A. et al. Java-based graphical user interface for the mrui quantitation package. Magnetic Resonance Materials inPhysics, Biology and Medicine 12, 141–152 (2001).

5. Vanhamme, L., van den Boogaart, A. & Van Huffel, S. Improved method for accurate and efficient quantification of mrsdata with use of prior knowledge. Journal of Magnetic Resonance 129, 35–43 (1997).

6. Wilson, M., Reynolds, G., Kauppinen, R. A., Arvanitis, T. N. & Peet, A. C. A constrained least-squares approach to theautomated quantitation of in vivo 1h magnetic resonance spectroscopy data. Magnetic Resonance in Medicine 65, 1–12(2011).

7. Soares, D. & Law, M. Magnetic resonance spectroscopy of the brain: review of metabolites and clinical applications.Clinical radiology 64, 12–21 (2009).

8. Costello, L., Franklin, R. & Narayan, P. Citrate in the diagnosis of prostate cancer. The prostate 38, 237 (1999).

9. Fischbach, F. & Bruhn, H. Assessment of in vivo 1h magnetic resonance spectroscopy in the liver: a review. LiverInternational 28, 297–307 (2008).

10. Bolan, P. J., Nelson, M. T., Yee, D. & Garwood, M. Imaging in breast cancer: magnetic resonance spectroscopy. BreastCancer Res 7, 149–152 (2005).

11. Mescher, M., Merkle, H., Kirsch, J., Garwood, M. & Gruetter, R. Simultaneous in vivo spectral editing and watersuppression. NMR in Biomedicine 11, 266–272 (1998).

12. Edden, R. A., Puts, N. A. & Barker, P. B. Macromolecule-suppressed gaba-edited magnetic resonance spectroscopy at 3t.Magnetic resonance in medicine 68, 657–661 (2012).

13. Chan, K. L., Puts, N. A., Schär, M., Barker, P. B. & Edden, R. A. Hermes: Hadamard encoding and reconstruction ofmega-edited spectroscopy. Magnetic resonance in medicine 76, 11–19 (2016).

14. Chan, K. L. et al. Echo time optimization for j-difference editing of glutathione at 3t. Magnetic resonance in medicine 77,498–504 (2017).

15. Aue, W., Bartholdi, E. & Ernst, R. R. Two-dimensional spectroscopy. application to nuclear magnetic resonance. TheJournal of Chemical Physics 64, 2229–2246 (1976).

16. Ryner, L. N., Sorenson, J. A. & Thomas, M. A. Localized 2d j-resolved 1 h mr spectroscopy: strong coupling effects invitro and in vivo. Magnetic Resonance Imaging 13, 853–869 (1995).

17. Kreis, R. & Boesch, C. Spatially localized, one-and two-dimensional nmr spectroscopy andin vivoapplication to humanmuscle. Journal of Magnetic Resonance, Series B 113, 103–118 (1996).

18. Thomas, M. A. et al. Localized two-dimensional shift correlated mr spectroscopy of human brain. Magnetic Resonance inMedicine 46, 58–67 (2001).

19. Dreher, W. & Leibfritz, D. Detection of homonuclear decoupled in vivo proton nmr spectra using constant time chemicalshift encoding: Ct-press. Magnetic resonance imaging 17, 141–150 (1999).

20. Schmieder, P., Stern, A. S., Wagner, G. & Hoch, J. C. Application of nonlinear sampling schemes to cosy-type spectra.Journal of biomolecular NMR 3, 569–576 (1993).

21. Schulte, R. F. & Boesiger, P. Profit: two-dimensional prior-knowledge fitting of j-resolved spectra. NMR in Biomedicine19, 255–263 (2006).

22. Martel, D., Koon, K. T. V., Le Fur, Y. & Ratiney, H. Localized 2d cosy sequences: Method and experimental evaluation fora whole metabolite quantification approach. Journal of Magnetic Resonance 260, 98–108 (2015).

23. LeCun, Y. et al. Backpropagation applied to handwritten zip code recognition. Neural computation 1, 541–551 (1989).

24. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436 (2015).

12/17

25. Goodfellow, I., Bengio, Y., Courville, A. & Bengio, Y. Deep learning, vol. 1 (MIT press Cambridge, 2016).26. Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation. In International

Conference on Medical image computing and computer-assisted intervention, 234–241 (Springer, 2015).

27. Long, J., Shelhamer, E. & Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEEconference on computer vision and pattern recognition, 3431–3440 (2015).

28. Huang, G., Liu, Z., Weinberger, K. Q. & van der Maaten, L. Densely connected convolutional networks. In Proceedings ofthe IEEE conference on computer vision and pattern recognition, vol. 1, 3 (2017).

29. Iqbal, Z., Nguyen, D., Hangel, G., Bogner, W. & Jiang, S. Super-resolution 1h magnetic resonance spectroscopic imagingutilizing deep learning. arXiv preprint arXiv:1802.07909 (2018).

30. Nguyen, D. et al. Three-dimensional radiotherapy dose prediction on head and neck cancer patients with a hierarchicallydensely connected u-net deep learning architecture. arXiv preprint arXiv:1805.10397 (2018).

31. Lustig, M., Donoho, D. & Pauly, J. M. Sparse mri: The application of compressed sensing for rapid mr imaging. MagneticResonance in Medicine 58, 1182–1195 (2007).

32. Smith, S., Levante, T., Meier, B. H. & Ernst, R. R. Computer simulations in magnetic resonance. an object-orientedprogramming approach. Journal of Magnetic Resonance, Series A 106, 75–105 (1994).

33. Govindaraju, V., Young, K. & Maudsley, A. A. Proton nmr chemical shifts and coupling constants for brain metabolites.NMR in Biomedicine 13, 129–153 (2000).

34. Macura, S. & Brown, L. R. Improved sensitivity and resolution in two-dimensional homonuclear j-resolved nmr spec-troscopy of macromolecules. Journal of Magnetic Resonance 53, 529–535 (1983).

35. Wilson, N. E., Iqbal, Z., Burns, B. L., Keller, M. & Thomas, M. A. Accelerated five-dimensional echo planar j-resolvedspectroscopic imaging: Implementation and pilot validation in human brain. Magnetic Resonance in Medicine 75, 42–51(2016).

36. Iqbal, Z., Wilson, N. E. & Thomas, M. A. 3d spatially encoded and accelerated te-averaged echo planar spectroscopicimaging in healthy human brain. NMR in Biomedicine 29, 329–339 (2016).

37. Wilson, N. E., Burns, B. L., Iqbal, Z. & Thomas, M. A. Correlated spectroscopic imaging of calf muscle in three spatialdimensions using group sparse reconstruction of undersampled single and multichannel data. Magnetic resonance inmedicine 74, 1199–1208 (2015).

38. Burns, B. L., Wilson, N. E. & Thomas, M. A. Group sparse reconstruction of multi-dimensional spectroscopic imaging inhuman brain in vivo. Algorithms 7, 276–294 (2014).

39. Burns, B., Wilson, N. E., Furuyama, J. K. & Thomas, M. A. Non-uniformly under-sampled multi-dimensional spectroscopicimaging in vivo: maximum entropy versus compressed sensing reconstruction. NMR in Biomedicine 27, 191–201 (2014).

40. Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advancesin neural information processing systems, 1097–1105 (2012).

41. Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).42. Hatami, N., Sdika, M. & Ratiney, H. Magnetic resonance spectroscopy quantification using deep learning. arXiv preprint

arXiv:1806.07237 (2018).

43. Fuchs, A., Boesiger, P., Schulte, R. F. & Henning, A. Profit revisited. Magnetic Resonance in Medicine 71, 458–468(2014).

44. Dang, L. et al. Cancer-associated idh1 mutations produce 2-hydroxyglutarate. Nature 462, 739 (2009).45. Ogg, R. J., Kingsley, R. & Taylor, J. S. Wet, a t 1-and b 1-insensitive water-suppression method for in vivo localized 1 h

nmr spectroscopy. Journal of Magnetic Resonance, Series B 104, 1–10 (1994).46. Cao, P. & Wu, E. X. In vivo diffusion mrs investigation of non-water molecules in biological tissues. NMR in biomedicine

30 (2017).47. Nicolay, K., Braun, K. P., Graaf, R. A. d., Dijkhuizen, R. M. & Kruiskamp, M. J. Diffusion nmr spectroscopy. NMR in

Biomedicine 14, 94–111 (2001).48. Ronen, I. & Valette, J. Diffusion-weighted magnetic resonance spectroscopy. eMagRes (2015).49. Hurd, R. et al. Measurement of brain glutamate using te-averaged press at 3t. Magnetic Resonance in Medicine 51,

435–440 (2004).

13/17

50. Brüschweiler, R. & Zhang, F. Covariance nuclear magnetic resonance spectroscopy. The Journal of chemical physics 120,5253–5260 (2004).

51. Iqbal, Z., Verma, G., Kumar, A. & Thomas, M. A. Covariance j-resolved spectroscopy: Theory and application in vivo.NMR in Biomedicine 30 (2017).

52. Mobli, M., Stern, A. S. & Hoch, J. C. Spectral reconstruction methods in fast nmr: reduced dimensionality, randomsampling and maximum entropy. Journal of Magnetic Resonance 182, 96–105 (2006).

14/17

Supplementary FiguresThe above methodology was also performed for JPRESS experiments. Some example results are shown below, and these resultsdemonstrate the capabilities of the D-UNet for JPRESS acceleration and JPRESS quantitation.

15/17

Figure 8. A JPRESS spectrum was simulated, and was under-sampled using the same 4x NUS mask shown in figure 2. The actual JPRESSspectrum is shown (Full JPRESS) and is qualitatively compared against the JPRESS spectrum reconstructed using a trained D-UNet model(Recon JPRESS). The difference between the two spectra is also shown (bottom).

16/17

Figure 9. A ground truth simulated JPRESS spectrum is displayed. Below the full spectrum are the actual and reconstructed individualmetabolite spectra for Cr3.0, NAA, PCh, mI, Gln, Glu, Lac, and GSH. The reconstruction was performed in an identical manner to theL-COSY quantitation described above.

17/17

References