Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Neurocomputing 397 (2020) 94–107
Contents lists available at ScienceDirect
Neurocomputing
journal homepage: www.elsevier.com/locate/neucom
A dual-domain deep lattice network for rapid MRI reconstruction
Liyan Sun
a , Yawen Wu
a , Binglin Shu
a , Xinghao Ding
a , ∗, Congbo Cai b , Yue Huang
a , John Paisley
c
a Fujian Key Laboratory of Sensing and Computing for Smart City, Xiamen University, Fujian, China b School of Electronic Science and Engineering, Xiamen University, China c Department of Electrical Engineering, Columbia University, New York, NY, USA
a r t i c l e i n f o
Article history:
Received 17 March 2019
Revised 18 August 2019
Accepted 17 January 2020
Available online 22 January 2020
Communicated by Dr. Shenghua Gao
Keywords:
Compressed sensing
Magnetic resonance imaging
Dual domain
Deep neural network
a b s t r a c t
Compressed sensing is utilized with the aims of reconstructing an MRI using a fraction of measurements
to accelerate magnetic resonance imaging called compressed sensing magnetic resonance imaging (CS-
MRI). Conventional optimization-based CS-MRI methods use random under-sampling patterns and model
the MRI data in the image domain as the classic CS-MRI paradigm. Instead, we design a uniform under-
sampling strategy and explore the potential of modeling the MRI data directly in the measured Fourier
domain. We propose a dual-domain deep lattice network (DD-DLN) for CS-MRI with variable density uni-
form under-sampling. We train the networks to learn the mapping between both image and frequency
domains. We observe the dual networks have complementary advantages, which motivates their combi-
nation via a lattice structure. Experiments show that the proposed DD-DLN model provides promising
performance in CS-MRI under the designed variable density uniform under-sampling.
© 2020 Elsevier B.V. All rights reserved.
p
f
x
w
s
a
m
t
fi
s
[
t
a
i
i
s
i
1. Introduction
Magnetic resonance imaging (MRI) is an important technique
in the field of medical imaging. Despite its high resolution in
soft tissues and low radiation, slow data acquisition is a major
limitation of MRI [1] . The raw measurements in MRI are Fourier
“k-space” coefficients, and the diagnostic MR image is then ob-
tained by an inverse 2D Fast Fourier Transform (FFT), which is
also the case where parallel imaging is utilized that multi-channel
k-space data are obtained to produce image domain result. One
technique for accelerating MRI is compressed sensing [2,3] , which
has attracted much attention since it can be combined with other
accelerating methods, e.g., parallel imaging techniques such as
SENSE [4,5] . According to compressed sensing theory, the MRI
scan can obtain much fewer k-space measurements than the
classic Nyquist sampling theorem requires [1] while still allow-
ing for very accurate reconstructions. With compressed sensing
techniques being approved by US Food and Drug Administration
(FDA) to two main MRI vendors: GE and Siemens in year 2017,
more MRI data are expected to be generated using compressed
sensing [6] .
∗ Corresponding author.
E-mail address: [email protected] (X. Ding).
k
i
m
https://doi.org/10.1016/j.neucom.2020.01.063
0925-2312/© 2020 Elsevier B.V. All rights reserved.
Under the subsampling environment, the problem of com-
ressed sensing for magnetic resonance imaging (CS-MRI) can be
ormulated as
= arg min
x
λ
2
‖
F u x − y ‖
2 2 + ρ( x ) , (1)
here x ∈ C N × 1 denotes the vectorized MR image to be recon-
tructed, F u ∈ C M × N denot es the under-sampled Fourier matrix
nd y ∈ C M × 1 ( M < N ) denotes the vectorized k-space measure-
ents with unsampled positions removed. The term ρ( x ) is used
o regularize the ill-posed problem and the first term is the data
delity ensuring consistency on the Fourier coefficients of recon-
truction at the measured locations.
The classic CS-MRI problem contains three key ingredients
1] : (1) The MRI should have a sparse representation in some
ransform domain, such as wavelets or a learned dictionary basis,
s imposed by the regularization ρ . (2) The aliasing artifacts
ntroduced by under-sampling in k-space should be incoherent,
ndicating that the sampling mask should be as random as pos-
ible, so that the under-sampled Fourier operator F u is highly
ncoherent. (3) The reconstruction is performed using the partial
-space measurements based on certain sparse regularization.
Following these three prerequisites, research in CS-MRI falls
nto proposing effective sparse regularization functions ρ( · ) for
odeling the MRI and algorithms for their efficient optimization.
L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107 95
Fig. 1. Different under-sampling patterns and the corresponding under-sampled k-space data and zero-filling MRI.
L
c
t
i
a
a
d
[
D
s
t
l
s
F
t
M
M
i
o
T
s
f
S
r
m
d
V
C
i
w
w
t
t
a
1
a
M
A
s
a
T
s
t
z
fi
i
M
n
s
p
M
c
ustig et al. [1] proposed the pioneering SparseMRI, where the
lassic objective function is formed by a data fidelity with to-
al variation minimization [7,8] and � 1 wavelet regularization us-
ng conjugated gradient method for optimization. Some work has
imed at optimizing this objective function more efficiently, such
s TVCMRI [9] , RecPF [10] and FCSA [11] . Others have focused on
esigning adaptive basis transforms using wavelets such as PBDW
12] , PANO [13] and GBRWT [14] , or dictionary learning such as
LMRI [15] , BPTV [16] and TLMRI [17] . These methods can offer
parser representations and yield better reconstruction accuracy at
he expense of higher computational burden. Other nonlocal regu-
arizations include NLR [18] and BM3D-MRI [19] . All these can be
ummarized as optimization-based CS-MRI methods.
Recently, deep learning has been utilized into CS-MRI field.
or example, Wang et al. [20] use the vanilla CNN to learning
he mapping from the zero-filling MR images to the full-sampled
R images. Lee et al. [21] proposed the modified U-Net for CS-
RI using the idea of residual learning. The Generative Adversar-
al Networks (GAN) models [22,23] are also utilized in the field
f compressed sensing MRI to generate high-quality MR images.
he above deep models directly learn the mapping from under-
ampled MRI data to full-sampled ones without considering un-
olding the classic inverse optimization in MRI recovery. Notably
chlemper et al. [24] proposed the deep cascade convolutional neu-
al network (DC-CNN) which achieves the state-of-the-art perfor-
ance by virtue of such unfolding structure. Similar unfolding
eep neural networks were also proposed in ADMM-Net [25] and
ariational Network (VN) [26] . Compared with optimization-based
S-MRI methods, deep CS-MRI is more computationally efficient
n application because only a forward pass is required with
ell-learned network parameters of the unfolded structure [27] ,
hile optimization-based methods require additional iterations
hat are often very time-consuming [24] .
From the previous work on CS-MRI, two key observations arise:
he desire for random under-sampling patterns and the need for
ppropriate regularization when reconstructing the MRI.
.1. Variable density random sampling
Variable density random under-sampling patterns have been
dopted due to the constraint of the second prerequisite of CS-
RI, a result of the Restricted Isometry Property (RIP) theorem [3] .
s a result, artifacts should appear to be random noise because of
uch random sampling. We show some illustrations in Fig. 1 with
fully-sampled k-space data and fully-sampled MR image given.
he under-sampling is simulated by applying the mask to the fully-
ampled k-space measurements. In the mask, the sampled posi-
ions are denoted by ones (white) and zeros (black) otherwise. The
ero-filling MRI is obtained via the inverse 2D FFT of the zero-
lling k-space data. We show a 2D 25% random sampling mask
n Fig. 1 , the corresponding under-sampled k-space and zero-filling
RI are also shown. We observe artifacts appear like random
oise [1] .
To see the effect of replacing the random sampling by uniform
ampling, we apply the 2D 25% consistent density uniform sam-
ling and obtain the corresponding k-space data and zero-filling
RI. The MRI suffers noticeably from inadequate sampling in the
entral low-frequency regions. We also fully-sample a small rect-
96 L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107
Fig. 2. The PSF functions of proposed uniform sampling masks and compared non-uniform ones.
s
d
t
f
I
i
p
I
m
G
t
u
s
s
w
k
o
I
m
i
c
m
m
d
i
1
t
s
p
m
s
1
angular region in the low-frequency center with the same uniform
sampling mask to obtain a variable density uniform sampling mask
with 26% under-sampling ratio 1 and show the k-space measure-
ments and zero-filling MRI. The artifacts are now more regular
and structured. Deep learning can excel in this structured setting
to relax the requirement for randomness in sampling. The auto-
mated transform by manifold approximation (AUTOMAP) [28] and
transfer learning [29] were primarily tested on the misaligned
and has shown potential of deep learning models in handling this
structural artifacts.
Throughout the paper, our proposed deep models are primarily
validated on such uniform under-sampling trajectories based on
two major motivations. In the standard CS-MRI formulation, the
incoherence of the under-sampling pattern is highly desired. In
the pioneering work SparseMRI [1] , the degree of coherence of
an under-sampling mask is measured by its point spread function
(PSF). The PSF is simply defined as P SF ( i, j ) = F H u F u ( i, j ) . The PSF
evaluates how a true underlying pixel leaks its energy to other pix-
els in the zero-filled MRI reconstruction. The PSF of a full-sampled
MRI is the identity and off-diagonal entries vanish. The values on
diagonal entries should be non-zero, however, non-zero or even
large off-diagonal entries indicate severe coherence. Here we plot
the PSF functions of our proposed 1D and 2D variable density
uniform under-sampling patterns versus the 1D and 2D vari-
able density non-uniform under-sampling in Fig. 2 . A number of
non-zeros entries in PSF maps indicate the coherence brought by
such regular sampling schemes. According to the Candes [2], the
number of measurements needed for accurate reconstruction is in
direct proportion to the degree of coherence of sensing matrix and
the inverse proportion to the underlying sparsity of a signal on a
transform domain. Previously the deep learning model has proved
their superior performance in restoring the random-noise-like
artifacts in incoherent sampling since the deep neural networks
could provide sparser representation. However, the strong seman-
tic representation ability of deep neural network is insufficiently
discussed in regarding to relax the requirement for incoherent
samplings according to Lustig et al. [1] . One purpose of designing
1 The variable density here denote the dense uniform sampling in low frequency
regions and sparse uniform sampling in high frequency regions.
p
b
i
s
uch coherence masks is to investigate how the properties of
eep neural network could help when exploring novel sampling
rajectories.
Another major motivation for such coherent sampling comes
rom the concerns of the potential medical image application.
n the field of parallel MRI imaging, the General Autocalibrat-
ng Partially Parallel Acquisition (GRAPPA) is a popular partial
arallel acquisition technique for MR imaging acceleration [30] .
n the GRAPPA model, multiple coils collect a set of k-space
easurements. The 1D Cartesian phase encoding is adopted in
RAPPA since it is suitable in practical MRI. The sampling pat-
ern of each GRAPPA coil is identical to the 1D variable density
niform sampling mask we developed in Fig. 11 . The full-sample
ub-regions in our proposed mask are referred as autocalibration
ignals (ACS). The ACS regions are leveraged to infer the GRAPPA
eights. Then the weights are utilized to interpolate the missing
-space data. The each individual coil image is recovered based
n the reconstructed k-space. The coil images are then combined.
n comparison, the dense sampling in low-frequency regions is
ainly motivated by capturing the majority of signal energy. Also,
n contrast to the linear GRAPPA weights obtained in situ from the
urrent data, the nonlinear network weights (parameters) in our
odel are inferred in massive-data-driven manner. The GRAPPA
odel is well studied in clinical MRI practice. However, the ran-
omness required in conventional sparsity-based compressed sens-
ng MRI is highly undesired in GRAPPA. Right now, the proposed
D variable density uniform sampling can be easily extended
o parallel GRAPPA MRI and meet the requirement for regular
ampling.
Despite the benefits of exploring the uniform sampling
atterns, we still evaluate our models on irregular sampling
asks to make our model more convincing in later experiment
ection.
.2. Frequency acquisition and image modeling
In most CS-MRI algorithms [12,15,17,24] , the reconstruction ap-
roach can be formulated as an iterative denoising that is solved
y alternating between denoising in the image domain and fill-
ng/correction in the k-space domain, as illustrated in Fig. 3 . More
pecifically, the zero-filling MRI is denoised using a particular regu-
L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107 97
Fig. 3. The standard diagram in designing CS-MRI algorithm.
l
t
c
i
v
s
a
i
b
[
d
i
n
t
fi
m
a
n
1
1
d
i
d
[
a
t
p
p
c
a
1
u
f
M
n
u
p
i
A
w
d
n
o
2
2
A
m
t
d
b
o
i
(
l
c
i
c
r
o
t
[
s
C
t
o
t
b
a
b
�
s
t
c
F
w
m
p
a
u
f
i
L
arization method and then is projected into k-space by the Fourier
ransform where the missing data is filled in. The filled-in Fourier
oefficients are transformed back into the image domain via the
nverse FFT, where the image is again denoised.
From this paradigm, CS-MRI is essentially k-space completion
ia finding a “noise-free” image that agrees with the measured k-
pace values [15] . The missing values of the unsampled positions
re estimated via a model in image domain, where visually mean-
ngful patterns exist. Rather, there may be some further advantages
y modeling k-space directly, since under the uncertainty principle
31] a trade-off in resolution between the spatial and frequency
omains exists. We therefore investigate the advantage of model-
ng in both the image and frequency domains with deep neural
etworks.
Based on the observations, we propose a dual-domain deep lat-
ice network (DD-DLN) for CS-MRI. To our knowledge, this is the
rst work to conduct the research on uniform under-sampling to
easure the MRI data and fuse the information from both im-
ge and frequency domain under the framework of deep neural
etworks.
.3. Related work
.3.1. Dual-domain models
Dual domain models—e.g., working in image and frequency
omains simultaneously—have attracted attention in the field of
mage processing. For example, Knaus and Zwicker developed
ual-domain filtering for image denoising [32,33] . Pierazzo et al.
34,35] further extended this work. Hirani et al. [36,37] proposed
modified Projection Onto Convex Sets (POCS) model to combine
he spatial and frequency information, and Wang et al. [38] pro-
osed the Deep Dual Domain D
3 model for fast restoration of com-
ressed JPEG images, where the deep sparse coding is used to en-
ode the sparsity nature of natural images in both image domain
nd DCT domain.
.3.2. Frequency interpolation
In the signal processing community the k-space interpolation is
sed in the non-uniform FFT (NUFFT) [39,40] to obtain Fourier in-
ormation of a finite length signal at any frequency. In the field of
RI, the structured low-rank matrix completion methods like an-
ihilating filter-based low-rank Hankel matrix approach (ALOHA) is
tilized to convert the CS-MRI models into frequency interpolation
roblem [41–43] . Some previous work attempt to interpolate miss-
ng k-space data in data-driven manner with residual learning [44] .
k-space convolutional neural network and image restoration net-
ork were developed in the KIKI-Net [45] . However, the proposed
ual-domain deep neural networks alternatives in a cascaded man-
er, meaning the sharing between the complementary information
f the dual domain is not fully utilized.
. Method
.1. Structural artifact removal
In Fig. 1 , we design a variable density under-sampling mask.
s discussed in the previous section, we expect the deep network
odel can distinguish structural artifacts from normal image struc-
ures brought on by uniform under-sampling, and so we design a
eep model for this end and compare it with other optimization-
ased CS-MRI models.
An intuitive and popular network design is shown in top row
f Fig. 4 , called deep residual network (ResNet) [46] . The network
n this case is formed by stacking complex-valued residual units
due to the Fourier transform). Each unit contains 4 convolutional
ayers with filters of size 3 × 3. A leaky ReLU with 0.2 slope rate is
hosen as the activation function, except for the last layer, which
s the identity. Details about the complex convolutional operations
an be found in [47] . The residual learning strategy in the complex
esidual unit has been shown to facilitate network training because
f better gradients during back propagation [46,48] .
Inspired by the classic CS-MRI paradigm in Fig. 3 , which is also
he motivation for the state-of-the-art deep CS-MRI model DC-CNN
24] , we propose an image domain reconstruction network (IDRN)
hown in Fig. 4 . The IDRN model shares a identical structure to DC-
NN except the complex convolution is utilized in IDRN, while a
wo-channel scheme is adopted in DC-CNN. In our experiments we
bserve the complex convolution leads to fewer network parame-
ers with comparable model performance. The network is formed
y cascading blocks. Each block contains a complex residual unit
nd a k-space correction. If the model consists of N such inference
locks, we call the model IDRN- N B.
We represent the subset of k-space that has been sampled as
and define y zf ∈ C N × 1 the k-space measurements where un-
ampled positions are filled with zeros. The input and output of
he corrected k-space (KC) is denoted a x in and x out . The k-space
orrection for the k th entries of the x in in IDRN can be written as
x out ( k ) k / ∈ � = F x in ( k ) , F x out ( k ) k ∈ � = y z f ( k ) , (2)
here F is the Fourier transform. For IDRN, similar to other CS-MRI
ethods that model the image domain, the output from the com-
lex residual unit is transformed back into the frequency domain
nd the missing Fourier coefficients are filled in by these new val-
es. The k-space is then transformed back into image space and
ed into the next residual unit. The loss function for training IDRN
s
IDRN ( θ ) =
1
∥∥x f s − f θ(x z f
)∥∥2
2 , (3)
2
98 L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107
Fig. 4. The network architecture of the ResNet, IDRN and FDRN. KC stands for k-space correction.
Fig. 5. Deep neural networks (IDRN with 5 blocks) can denoise the and remove artifacts better than optimization-based approaches (e.g., PANO).
i
s
t
w
t
t
s
d
r
t
n
F
I
r
v
f
q
q
i
f
c
d
c
r
r
where x fs is the fully-sampled MRI and x zf is the corresponding
zero-filling MRI. f θ ( · ) denotes the function mapping of IDRN with
network parameters θ .
We compare the state-of-the-art optimization-based CS-MRI
method PANO [13] with the proposed IDRN-5B model using the
variable density uniform sampling mask in Fig. 1 . We show the re-
construction on a test MRI data in Fig. 5 . We observe the PANO re-
construction can preserve image structures but struggles to remove
the structural artifacts. IDRN-5B model on the other hand can re-
move these artifacts without interfering with the image structures.
2.1.1. Frequency domain reconstruction network
From Fourier analysis, the 2D discrete Fourier transform on an
image f of the size M × N is denoted as
F [ k, l ] =
1 √
MN
N−1 ∑
n =0
M−1 ∑
m =0
f [ m, n ] e − j2 π( mk M + nl
N )
f [ m, n ] =
1 √
MN
N−1 ∑
l=0
M−1 ∑
k =0
F [ k, l ] e j2 π( mk M + nl
N )
( 0 ≤ m, k ≤ M − 1 , 0 ≤ n, l ≤ N − 1 )
. (4)
Where the first equation is Fourier analysis and the second Fourier
synthesis. According to the Fourier analysis equation, each k-space
coefficient is obtained by a dot product with all pixels across the
entire spatial domain, meaning it contains information from the
entire image. Conversely, the entire frequency domain contributes
nformation to every pixel of the image according to the Fourier
ynthesis equation. This has inspired popular signal processing
ools such as the short-time Fourier transform (STFT) [49] and
avelets [50] to analyze a signal in multi-scale fashion.
In the conventional optimization-based CS-MRI, the approach is
o model statistical characteristics of the MRI, such as sparsity, in
he image domain. Although CS-MRI is essentially the process of k-
pace completion, few models that work directly in the frequency
omain exists. In this work we consider learning a mapping di-
ectly in the frequency domain with a deep network. We name
he architecture for this approach a frequency domain reconstruction
etwork (FDRN), as shown in Fig. 4 . Similarly, we call the model
DRN- N B if it is formed by cascading N inference blocks.
The FDRN is constituted by cascaded inference blocks similar to
DRN, but with two simple differences. First, the complex-valued
esidual unit works on the frequency domain. In the traditional
ariable density sampling scheme, the dense sampling in the low
requency regions and sparse or even no sampling in the high fre-
uency regions, is not suitable for such direct convolution in fre-
uency domain. This phenomenon was also not studied thoroughly
n [44] . While for the variable density uniform sampling, the uni-
orm sampling guarantees that data appears in each step of the
onvolution, which is especially unlikely with variable density ran-
om sampling when the filters are 3 × 3. Second, using k-space
orrection with FDRN, the measured samples in k-space directly
eplace the corresponding positions in the output of the complex
esidual unit in the same block. Unlike IDRN, no image/frequency
L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107 99
Fig. 6. The network architecture of the dual-domain deep lattice network.
d
d
a
o
y
T
w
F
L
w
s
f
2
a
w
f
s
i
b
n
t
t
c
n
f
I
i
L
T
F
t
t
D
s
e
i
I
F
r
t ∣∣r
a
l
u
i
F
c
f
a
a
m
b
a
T
v
t
i
H
fi
E
t
b
S
r
r
fi
3
F
s
omain transform is necessary for k-space correction (hence the
ifferent color of KC in Fig. 4 ).
In FDRN, the input and output of the k-space correction step
re denoted y in and y out . The k-space correction for the k th entries
f the y in in FDRN can be written as
out ( k ) k / ∈ � = y in ( k ) , y out ( k ) k ∈ � = y z f ( k ) . (5)
he output of the last block in FDRN is the completed k-space,
hich we then transform into the image domain with an inverse
FT. The loss function for training FDRN model is
F DRN ( θ ) =
1
2
∥∥y f s − g θ(y z f
)∥∥2
2 , (6)
here y fs is the fully-sampled k-space data and y zf is the corre-
ponding zero-filling k-space measurements. g θ ( · ) denotes the
unction mapping of FDRN with network parameters θ .
.2. A dual-domain deep lattice network
Based on the proposed IDRN and FDRN models, we propose
dual-domain deep lattice network (DD-DLN), shown in Fig. 6 ,
hich fuses information from IDRN and FDRN in a single objective
ramework. Here the two subnetworks interact in the form of the
hown lattice structure. (In digital filter design, the term “lattice”
s often referred to as a cross-connection structure [51] , which we
orrow here.) Now, for a certain block in the IDRN or FDRN sub-
etwork, the output is not only fed into the next block, but also
ransformed into frequency or image domain and concatenated to
he other subnetwork as indicated by arrows in Fig. 6 . The con-
atenation operation helps to fuse information from both domains.
As before, we call the DD-DLN model DD-DLN- N B if each sub-
etwork contains N blocks. DD-DLN produces a reconstructed MRI
rom each branch (or subnetwork), which we call DD-DLN- N B-
DRN and DD-DLN- N B-FDRN. The loss function for training DD-DLN
s
DD −DLN =
1
2
( L IDRN + L F DRN ) . (7)
his is our final framework, but we also experiment with IDRN and
DRN separately in the next section to evaluate the advantage of
he network shown in Fig. 6 .
To further evaluate our conjecture that information sharing be-
ween the image and frequency domains is feasible, we train the
D-DLN-5B model on the variable density uniform sampling mask
hown in Fig. 1 and give intermediate reconstruction results of
ach block for both IDRN-5B branch and FDRN-5B branch, shown
n the first and second row of Fig. 7 respectively. We observe the
DRN-5B branch has better imaging quality, but to check whether
DRN-5B branch provides a more accurate reconstruction in some
egions we look at the differences between their absolute errors
o the ground truth. We denote the absolute error of IDRN e IDRN =x IDRN − x f s
∣∣ and FDRN e F DRN =
∣∣x F DRN − x f s
∣∣, where x IDRN and x FDRN
epresents the output MR images from the corresponding IDRN-5B
nd FDRN-5B models. We take the difference between the abso-
ute error of IDRN-5B and FDRN-5B and keep the non-negative val-
es, yielding the difference map m di f f = ( e IDRN − e F DRN ) + . The pos-
tive parts indicate the regions of more accurate reconstruction of
DRN-5B than IDRN-5B.
In the last row of Fig. 7 we show the difference maps with the
olormaps in display range [0 0.1], i.e., positive parts of this dif-
erence, corresponding to locations where FDRN branch was more
ccurate than IDRN branch. These difference maps are generated
cross different blocks. We observe in the shallow blocks, the FDRN
odel provides better reconstruction qualities on the edges of the
rain (edges and outlines), while the IDRN model achieves higher
ccuracies on the small structures (low-contrast) in the MR image.
his phenomenon can be attributed to the facts that the deep con-
olutional neural network in image domain has difficulty in cap-
uring large contextual information in shallow layers due to lim-
ted receptive fields especially with small kernel size, e.g., 3 × 3.
owever, for deep models in frequency domain, the large receptive
elds can be achieved in shallow layers according to our analysis of
q. 4 . We also observe in the last block, the IDRN branch achieves
he completely superior performance over FDRN branch, which can
e proved by the fact the difference map is nearly zeros in Fig. 7 .
ince the IDRN produces better reconstruction than the FDRN and
epresents the state-of-the-art performance in under-sampled MRI
econstruction, we reasonably regard the DD-DLN- N B-IDRN as the
nal output.
. Results
In this section we experiment with the proposed deep IDRN,
DRN and fused DD-DLN models on different uniform under-
ampling patterns with different under-sampling ratios to evaluate
100 L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107
Fig. 7. The performance of DD-DLN-5B-IDRN and DD-DLN-5B-FDRN. The intermediate reconstructions from each block in both IDRN-5B branch and FDRN-5B branch are
shown in the first row and second row, respectively. We show post-processed absolute differences of reconstruction error in the third row. Brightness indicates FDRN branch
is more accurate.
b
d
3
r
1
W
1
T
t
3
d
T
t
m
P
a
h
[
S
their effectiveness. We also compare with several state-of-the-art
non-deep CS-MRI techniques: transform learning MRI (TLMRI) [17] ,
graph-based redundant wavelet transform (GBRWT) [14] , patch-
based nonlocal operator (PANO) [13] and BM3D-MRI [19] . Since
IDRN shares a similar network structure with DC-CNN [24] , it can
be taken as representing the state-of-the-art in CS-MRI, which we
will see is improved upon by DD-DLN. We also compare with the
unfolding deep architecture ADMM-Net [26] .
3.1. Dataset
In the experiment section, we present experimental results
using complex-valued MRI datasets. The datasets are collected in
the second affiliated hospital of Xiamen Medical College. The T1
weighted MRI dataset (size 256 × 256) is acquired on 40 volun-
teers with total 3800 MR images at Siemens 3.0T scanner with
12 coils using the fast low angle shot (FLASH) sequence (TR/TE
= 55/3.6 ms, 220 mm
2 field of view, 1.5 mm slice thickness).
Following the simulation setting in [13] , the SENSE reconstruction
is introduced to compose the gold standard full k-space, which
is used to emulate the single-channel MRI. We randomly select
80% MR images as training set and 20% MR images as testing set.
Informed consent was obtained from the imaging subject in com-
pliance with the Institutional Review Board policy. We apply nor-
malization on the full-sampled MR images. It is worth noting the
experiment data used in the work is complex-valued MR images,
which conforms to the fact the actual measurements of MRI are
complex-valued.
Under-sampled k-space measurements are manually obtained
via 1D Cartesian and 2D random sampling mask. Different under-
sampling ratios are adopted in the experiments. The under-
sampling ratio here is defined as the ratio between the num-
er of partially sampled data and the number of full-sampled
ata.
.2. Implementation details
We use ADAM for network optimization with an initial learning
ate of 0.0 0 01. The training runs for 20,0 0 0 iterations and at every
00 iterations we decrease the learning rate by a multiple of 0.99.
e implement the deep models on Tensorflow using NVIDIA GTX
080 with 8G memory and Intel Xeon CPU E5-2683 at 2.00 GHz.
he deep learning models and conventional non-deep models are
ested on the same device for fair comparison in testing phase.
.3. Evaluation metric
We use peak-signal-to-ratio (PSNR) and structural similarity in-
ex (SSIM) as the objective metric to compare model performance.
he PSNR is a widely-used metric to measure the distance between
he reconstructed MRI x recon and full-sampled MRI x fs , which is for-
ulated as
SNR = 20 log 10
(
1 ∥∥x recon − x f s
∥∥2
2
)
. (8)
Although the PSNR index can assess the absolute reconstruction
ccuracy, the reconstruction quality of structural information can
ardly to be well evaluated. Therefore we use another index SSIM
52] to compare the models. The SSIM is formulated as
SIM =
(2 μ f s μ recon + C 1
)(2 σ f s,recon + C 2
)(μ 2
f s + μ 2
recon + C 1 )(
σ 2 f s
+ σ 2 recon + C 1
) , (9)
L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107 101
Table 1
Reconstruction time of different methods. Note the PANO, TLMRI, GBRWT and BM3D-MRI methods are non-deep models without the need for GPU computation.
PANO TLMRI GBRWT BM3D-MRI FDRN-5B IDRN-5B DD-DLN-5B
Runtime (seconds) 14.82 s 100.8 s 138.1 s 14.94 s 0.26 s 0.3 s 0.42 s
Fig. 8. The training and validation losses of IDRN-5B, FDRN-5B and DDDLN-5B are
shown.
w
s
σ
i
n
3
t
o
r
t
t
m
M
m
t
p
c
d
t
3
t
F
I
t
d
[
i
a
o
r
t
o
r
a
a
p
h
F
w
t
s
m
e
q
b
i
a
5
o
b
F
p
5
L
f
u
v
c
p
p
4
4
m
t
A
p
4
s
r
m
t
F
here μfs and μrecon represent the mean value of the recon-
tructed full-sampled and reconstructed MR images, and σ fs ,
recons and σ fs,recons denote the standard deviation of the two
mages and their covariance. C 1 and C 2 are two constants for
umeric stability.
.4. Running time
The compressed sensing MRI models regularized with conven-
ional sparse or non-local prior requires time-consuming iterative
ptimization. However, for deep-based CS-MRI models, the forward
econstruction is rapid without needs of iterative optimization if
he network is well-trained. We show the averaged reconstruc-
ion time on the test dataset with uniform 2D 25% under-sampling
ask in Table 1 . We observe the PANO and BM3D-MRI reconstruct
RI data faster than TLMRI and GBRWT, but still the non-deep
odels takes more than 10 s for reconstruction. While the state-of-
he-art deep model IDRN-5B model (similar to DC-CNN [24] ), the
roposed frequency learning model FDRN-5B have the fastest re-
onstruction speed around 0.3 seconds. Our proposed dual domain
eep lattice network takes 0.1 s longer than IDRN-5B and FDRN-5B
o reconstruct a MRI data in average.
.5. Evaluation
We compare the training and validation errors in Fig. 8 using
he 26% 2D variable density uniform sampling mask as shown in
ig. 1 , showing a stable more lower error of DD-DLN model over
DRN and FDRN model.
In Fig. 9 , we compare the DD-DLN-5B model with state-of-
he-art optimization-based CS-MRI methods including several non-
eep models and the ADMM-Net with the unfolding deep structure
o
26] , as well as with IDRN-5B and FDRN-5B on a test MRI data us-
ng 2D variable density uniform sampling. The residual error maps
re also provided in the range 0–0.3. These results correspond to
ne testing image, but represents the common observation. Pa-
ameters of the compared optimization-based CS-MRI methods are
uned to give their best performance using source codes provided
n the respective authors’ websites.
We observe that optimization-based methods have difficulty in
emoving structural artifacts. GBRWT achieves good performance
t this task, but still with obvious artifacts. For TLMRI, the artifacts
re better removed at the expense of image blurring. FDRN-5B
roduces worse reconstruction compared with IDRN-5B, but both
andle artifacts well. DD-DLN-5B improves further on independent
DRN and IDRN results, as seen by the errors of their subnetworks
ithin this joint framework. As a quantitative evaluation, we show
he PSNR and SSIM averaged over the test MRI in Fig. 10 . The re-
ults are consistent with the subjective reconstruction quality.
Because of the lattice structure, the DD-DLN-5B model has
ore network parameters than the IDRN-5B and FDRN-5B mod-
ls. Thus a natural question arises: Is the improved reconstruction
uality of DDDLN model attributed to the enlarged model capacity
rought by more network parameters rather than the dual domain
nformation? We expand the model capacity of IDRN-5B model to
dapt it to the near identical number of parameters of DD-DLN-
B. We name the enlarged IDRN-5B as IDRN-5B-L with the number
f parameters 390k, while the DD-DLN-5B models have the num-
er of parameters of 386k. The comparison results are shown in
ig. 10 . We observe the larger model capacity of IDRN-5B-L out-
erforms the IDRN-5B with some margins, however, the DD-DLN-
B-IDRN still achieves better reconstruction quality over IDRN-5B-
, demonstrating the effectiveness of the dual domain information
usion idea.
We also test DD-DLN using the 1D Cartesian variable density
niform mask shown in Fig. 11 . The residual error maps are pro-
ided in the same range as the 2D case. Also we fully-sample the
entral low-frequency regions to retain most of the energy. The ex-
erimental results in Fig. 11 again show the effectiveness of the
roposed model, and robustness to various sampling patterns.
. Discussions
.1. Discussion on number of blocks
We show how the performances of IDRN, FDRN and DD-DLN
odels varies with number of blocks in Fig. 12 . We observe
he DD-DLN-IDRN model steadily outperforms other three models.
s the network architectures go deep, the DD-DLN-FDRN model
erforms well and approaches the DD-DLN-IDRN model.
.2. Discussion on different under-sampling ratios
We also compare our proposed models with other compressed
ensing MRI methods on uniform 2D masks with under-sampling
atios of 26%, 28%, 30%, 32% and 40%. The higher sampling ratios
eans larger full-sampling central regions in our design. We show
he 26% mask in Fig. 9 and the 28%, 30%, 32% and 40% masks in
ig. 13 .
As shown in Table 2 , the DD-DLN-5B-IDRN model outper-forms
ther models on all sampling patterns. We observe the DD-DLN-
102 L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107
Fig. 9. The reconstruction results with 26% 2D variable density uniform sampling masks and their corresponding error images.
Table 2
Model comparison conditioned on 2D variable density uniform masks with different under-sampling ratios.
Under-sampling Ratios % 26 28 30 32 40
Evaluation Index PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM
PANO [13] 26.08 0.774 27.41 0.859 28.62 0.882 30.49 0.909 33.32 0.947
TLMRI [17] 25.81 0.785 26.74 0.761 27.33 0.773 28.24 0.796 30.17 0.854
GBRWT [14] 25.71 0.834 29.07 0.894 30.68 0.910 32.57 0.929 35.39 0.957
BM3D-MRI [19] 27.93 0.748 29.87 0.816 30.88 0.846 32.27 0.880 35.38 0.936
ADMM-Net [26] 27.72 0.738 29.86 0.850 31.59 0.905 32.67 0.915 37.96 0.971
FDRN-5B 28.20 0.846 30.77 0.901 31.67 0.917 33.18 0.940 36.06 0.968
IDRN-5B [24] 32.14 0.949 35.50 0.973 36.65 0.977 38.41 0.983 41.18 0.989
DD-DLN-5B-FDRN 32.56 0.949 35.45 0.973 36.71 0.978 38.23 0.983 41.05 0.989
DD-DLN-5B-IDRN 32.80 0.955 35.73 0.974 36.99 0.979 38.50 0.983 41.22 0.989
4
W
m
p
F
s
a
5B-FDRN achieves comparable performance with IDRN-5B model.
Interestingly, with the higher under-sampling ratio, the gap be-
tween the model performance of IDRN-5B and DD-DLN-5B-IDRN
narrows.
The performance of GBRWT boosts rapidly with high under-
sampling ratio because more accurate graph representation can be
achieved with less structural artifacts. However, for BM3D-MRI, the
structural artifacts impose an adverse influence on the searching
of similar patches in non-local prior, resulting in unsatisfactory
reconstructions. a
d
.3. Discussion on conventional non-uniform masks
We validate our model on a 30% 2D random sampling mask.
e employ the same experimental setting as reported in the
anuscript with the only difference being the different sampling
atterns. We show an example MRI construction comparison in
ig. 14 . Along with the reconstructions we also show the corre-
ponding residual error maps and objective metrics such as PSNR
nd SSIM. We observe the DD-DLN-5B-IDRN preserves the fine im-
ge details better compared with the baseline IDRN-5B on this ran-
om sampling mask. Both deep models outperform the non-deep
L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107 103
Fig. 10. The PSNR and SSIM averaged on test MRI data.
Table 3
Averaged PSNR and SSIM values on the test dataset on the 30% 2D Random under-
sampling pattern.
Metrics TLMRI PANO IDRN-5B DD-DLN-5B-IDRN
PSNR dB 35.87 36.27 38.78 39.12
SSIM 0.897 0.923 0.946 0.951
m
w
i
p
d
n
d
4
o
i
n
r
a
s
b
n
the artifacts.
odels, which is already well established in previous deep CS-MRI
orks.
We also show the PSNR and SSIM values averaged on the test-
ng datasets in Table 3 . We observe the results random sampling
rovides higher reconstruction accuracy reasonably due to the high
egree of incoherence. Even though, our dual domain deep lattice
etwork also offers gains over the state-of-the-art image domain
eep model.
.4. Further analysis
For the IDRN subnetwork of DD-DLN-5B, we show the residuals
f the complex-valued residual unit within in each cascaded
nference block in Fig. 15 . The residuals become smaller as the
etwork goes deeper, because the deeper blocks produce better
econstructions.
In the lower blocks, the residuals should ideally consist of neg-
tive artifacts. We observe that the magnitudes of the residuals
how a similar structure in the artifacts as the block goes deeper,
ut it is less pronounced, meaning the deeper components in the
etwork are better able to distinguish the image structure within
104 L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107
Fig. 11. The reconstruction results with 26% 1D variable density uniform sampling masks and their corresponding error images.
Fig. 12. The comparison of model performance of FDRN, IDRN and DD-DLN conditioned on the number of cascaded blocks.
L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107 105
Fig. 13. The uniform 2D masks with under-sampling ratios of 26%, 28%, 30%, 32% and 40%.
Fig. 14. The comparison of models on the 30% 2D random under-sampling pattern.
Fig. 15. The magnitudes of the residuals for each block of the DD-DLN-5B-IDRN subnetwork.
5
b
c
a
b
c
s
w
m
D
c
i
A
S
6
F
G
f
C
a
2
R
. Conclusion
The variable density random under-sampling pattern required
y RIP and appropriate image regularizations are two common fo-
uses in performing CS-MRI inversion. In this paper, we proposed
dual-domain deep lattice network to fuse the information from
oth image and frequency domains using a sampling strategy that
aptures most of the low resolution information. The experiments
how that the proposed deep model has promising performance
ith the uniform sampling strategy compared with non-deep
ethods.
eclaration of Competing Interests
The authors declare that they have no known competing finan-
ial interests or personal relationships that could have appeared to
nfluence the work reported in this paper.
cknowledgment
This work was supported in part by the National Natural
cience Foundation of China under Grants 61571382 , 81671766 ,
1571005 , 81671674 , 61671309 and U1605252 , in part by the
undamental Research Funds for the Central Universities under
rant 20720160075 , 20720180059 , in part by the CCF-Tencent open
und, and the Natural Science Foundation of Fujian Province of
hina (No. 2017J01126 ). L. Sun conducted portions of this work
t Columbia University under China Scholarship Council grant (No.
01806310090 ).
eferences
[1] M. Lustig , D. Donoho , J.M. Pauly , Sparse MRI: the application of compressedsensing for rapid MR imaging, Magn. Reson. Med. 58 (6) (2007) 1182–1195 .
[2] D.L. Donoho , Compressed sensing, IEEE Trans. Inf. Theory 52 (4) (2006)1289–1306 .
[3] E.J. Candes , The restricted isometry property and its implications for com-
pressed sensing, C.R. Math. 346 (9–10) (2008) 589–592 . [4] K.P. Pruessmann , M. Weiger , M.B. Scheidegger , P. Boesiger , et al. , SENSE: sensi-
tivity encoding for fast MRI, Magn. Reson. Med. 42 (5) (1999) 952–962 . [5] D. Liang , B. Liu , J. Wang , L. Ying , Accelerating SENSE using compressed sensing,
Magn. Reson. Med. 62 (6) (2009) 1574–1584 . [6] J.A. Fessler, Medical image reconstruction: a brief overview of past milestones
and future directions. arXiv preprint arXiv: 1707.05927 . [7] V.P. Gopi , P. Palanisamy , K.A. Wahid , P. Babyn , D. Cooper , Multiple regulariza-
tion based MRI reconstruction, Signal Process. 103 (2014) 103–113 .
[8] P. Zhuang , X. Zhu , X. Ding , MRI reconstruction with an edge-preserving filter-ing prior, Signal Process. 155 (2019) 346–357 .
[9] S. Ma , W. Yin , Y. Zhang , A. Chakraborty , An efficient algorithm for compressedMR imaging using total variation and wavelets, in: Computer Vision and Pat-
tern Recognition, IEEE, 2008, pp. 1–8 .
106 L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107
[10] J. Yang , Y. Zhang , W. Yin , A fast alternating direction method for TVL1-L2 signalreconstruction from partial fourier data, IEEE J. Sel. Top. Signal Process. 4 (2)
(2010) 288–297 . [11] J. Huang , S. Zhang , D. Metaxas , Efficient MR image reconstruction for com-
pressed MR imaging, Med. Image Anal. 15 (5) (2011) 670–679 . [12] X. Qu , D. Guo , B. Ning , Y. Hou , Y. Lin , S. Cai , Z. Chen , Undersampled MRI re-
construction with patch-based directional wavelets, Magn. Reson. Imaging 30(7) (2012) 964–977 .
[13] X. Qu , Y. Hou , L. Fan , D. Guo , J. Zhong , Z. Chen , Magnetic resonance image re-
construction from undersampled measurements using a patch-based nonlocaloperator, Med. Image Anal. 18 (6) (2014) 843 .
[14] Z. Lai , X. Qu , Y. Liu , D. Guo , J. Ye , Z. Zhan , Z. Chen , Image reconstructionof compressed sensing MRI using graph-based redundant wavelet transform.,
Med. Image Anal. 27 (2016) 93 . [15] S. Ravishankar , Y. Bresler , MR Image reconstruction from highly undersampled
k-space data by dictionary learning, IEEE Trans. Med. Imaging 30 (5) (2011)
1028–1041 . [16] Y. Huang , J. Paisley , Q. Lin , X. Ding , X. Fu , X.-P. Zhang , Bayesian nonparametric
dictionary learning for compressed sensing MRI, IEEE Trans. Image Process. 23(12) (2014) 5007–5019 .
[17] S. Ravishankar , Y. Bresler , Efficient blind compressed sensing using sparsify-ing transforms with convergence guarantees and application to magnetic res-
onance imaging, SIAM J. Imaging Sci. 8 (4) (2015) 2519–2557 .
[18] W. Dong , G. Shi , X. Li , Y. Ma , F. Huang , Compressive sensing via nonlocallow-rank regularization, IEEE Trans. Image Process. 23 (8) (2014) 3618–3632 .
[19] E.M. Eksioglu , Decoupled algorithm for MRI reconstruction using nonlo-cal block matching model: BM3D-MRI, J. Math. Imaging Vis. 56 (3) (2016)
430–440 . [20] S. Wang , Z. Su , L. Ying , X. Peng , S. Zhu , F. Liang , D. Feng , D. Liang , Accelerating
magnetic resonance imaging via deep learning, in: International Symposium
on Biomedical Imaging, 2016, pp. 514–517 . [21] D. Lee , J. Yoo , J.C. Ye , Deep residual learning for compressed sensing MRI, in:
International Symposium on Biomedical Imaging, 2017, pp. 15–18 . [22] G. Yang , S. Yu , H. Dong , G. Slabaugh , P.L. Dragotti , X. Ye , F. Liu , S. Arridge , J. Kee-
gan , Y. Guo , et al. , Dagan: deep de-aliasing generative adversarial networks forfast compressed sensing MRI reconstruction, IEEE Trans. Med. Imaging 37 (6)
(2018) 1310–1321 .
[23] T.M. Quan , T. Nguyen-Duc , W.-K. Jeong , Compressed sensing MRI reconstruc-tion using a generative adversarial network with a cyclic loss, IEEE Trans. Med.
Imaging 37 (6) (2018) 1488–1497 . [24] J. Schlemper , J. Caballero , J.V. Hajnal , A. Price , D. Rueckert , A deep cascade
of convolutional neural networks for MR image reconstruction, in: Interna-tional Conference on Information Processing in Medical Imaging, Springer,
2017, pp. 647–658 .
[25] K. Hammernik , T. Klatzer , E. Kobler , M.P. Recht , D.K. Sodickson , T. Pock , F. Knoll ,Learning a variational network for reconstruction of accelerated MRI data,
Magn. Reson. Med. 79 (6) (2018) 3055–3071 . [26] Y. Yang , J. Sun , H. Li , Z. Xu , ADMM-CSNet: a deep learning approach for image
compressive sensing, IEEE Trans. Pattern Anal. Mach. Intell. (2018) . [27] K. Gregor , Y. LeCun , Learning fast approximations of sparse coding, in: Inter-
national Conference on Machine Learning, Omnipress, 2010, pp. 399–406 . [28] B. Zhu , J.Z. Liu , S.F. Cauley , B.R. Rosen , M.S. Rosen , Image reconstruction by
domain-transform manifold learning, Nature 555 (7697) (2018) 487 .
[29] F. Knoll , K. Hammernik , E. Kobler , T. Pock , M.P. Recht , D.K. Sodickson , Assess-ment of the generalization of learned image reconstruction and the potential
for transfer learning, Magn. Reson. Med. 81 (1) (2019) 116–128 . [30] M.A. Griswold , P.M. Jakob , R.M. Heidemann , M. Nittka , V. Jellus , J. Wang ,
B. Kiefer , A. Haase , Generalized autocalibrating partially parallel acquisitions(grappa), Magn. Reson. Med.: Off. J. Int. Soc. Magn. Reson. Med. 47 (6) (2002)
1202–1210 .
[31] R. Wilson , G.H. Granlund , The uncertainty principle in image processing, IEEETrans. Pattern Anal. Mach. Intell. (6) (1984) 758–767 .
[32] C. Knaus , M. Zwicker , Dual-domain filtering, SIAM J. Imaging Sci. 8 (3) (2015)1396–1420 .
[33] C. Knaus , M. Zwicker , Dual-domain image denoising, in: International Confer-ence on Image Processing, IEEE, 2013, pp. 4 40–4 4 4 .
[34] N. Pierazzo , M. Lebrun , M. Rais , J.-M. Morel , G. Facciolo , Non-local dual im-
age denoising, in: International Conference on Image Processing, IEEE, 2014,pp. 813–817 .
[35] N. Pierazzo , M.E. Rais , J.-M. Morel , G. Facciolo , DA3D: fast and data adaptivedual domain denoising, in: International Conference on Image Processing, IEEE,
2015, pp. 432–436 . [36] A.N. Hirani , T. Totsuka , Dual domain interactive image restoration: basic algo-
rithm, in: Proceedings of the International Conference on Image Processing,
1996, 1, IEEE, 1996, pp. 797–800 . [37] A.N. Hirani , T. Totsuka , Combining frequency and spatial domain informa-
tion for fast interactive image noise removal, in: Proceedings of the 23rd An-nual Conference on Computer Graphics and Interactive Techniques, ACM, 1996,
pp. 269–276 . [38] Z. Wang , D. Liu , S. Chang , Q. Ling , Y. Yang , T.S. Huang , D3: deep dual-domain
based fast restoration of JPEG-compressed images, in: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2016, pp. 2764–2772 .
[39] J.A. Fessler , B.P. Sutton , Nonuniform fast fourier transforms usingmin-max interpolation, IEEE Trans. Signal Process. 51 (2) (2003)
560–574 . [40] F. Marvasti , Nonuniform Sampling: Theory and Practice, Springer Science &
Business Media, 2012 . [41] K.H. Jin , D. Lee , J.C. Ye , A general framework for compressed sensing and par-
allel MRI using annihilating filter based low-rank hankel matrix, IEEE Trans.Comput. Imaging 2 (4) (2016) 4 80–4 95 .
[42] P.J. Shin , P.E. Larson , M.A. Ohliger , M. Elad , J.M. Pauly , D.B. Vigneron ,
M. Lustig , Calibrationless parallel imaging reconstruction based on struc-tured low-rank matrix completion, Magn. Reson. Med. 72 (4) (2014)
959–970 . [43] J.P. Haldar , Low-rank modeling of local k-space neighborhoods (LORAKS) for
constrained MRI, IEEE Trans. Med. Imaging 33 (3) (2014) 66 8–6 81 . [44] Y. Han , J.C. Ye , K-space deep learning for accelerated MRI, IEEE T. Med. Imaging
(2019) .
[45] T. Eo , Y. Jun , T. Kim , J. Jang , H.-J. Lee , D. Hwang , Kiki-net: cross-domain convo-lutional neural networks for reconstructing undersampled magnetic resonance
images, Magn. Reson. Med. 80 (5) (2018) 2188–2201 . [46] K. He , X. Zhang , S. Ren , J. Sun , Deep residual learning for image recognition, in:
Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni-tion, 2016, pp. 770–778 .
[47] C. Trabelsi , O. Bilaniuk , D. Serdyuk , S. Subramanian , J.F. Santos , S. Mehri ,
N. Rostamzadeh , Y. Bengio , C.J. Pal , Deep complex networks, Internet Confer-ence on Learning Representation (2018) .
[48] K. He , X. Zhang , S. Ren , J. Sun , Identity mappings in deep residual net-works, in: European Conference on Computer Vision, Springer, 2016, pp.
630–645 . [49] E. Sejdi ́c , I. Djurovi ́c , J. Jiang , Time–frequency feature representation using en-
ergy concentration: an overview of recent advances, Digit. Signal. Process. 19
(1) (2009) 153–183 . [50] S.G. Mallat , A theory for multiresolution signal decomposition: the wavelet
representation, IEEE Trans. Pattern Anal. Mach. Intell. 11 (7) (1989) 674–693 .
[51] M. Honig , D. Messerschmitt , Convergence properties of an adaptive digital lat-tice filter, IEEE Trans. Circuits Syst. 28 (6) (1981) 4 82–4 93 .
[52] Z. Wang , A.C. Bovik , H.R. Sheikh , E.P. Simoncelli , Image quality assessment:
from error visibility to structural similarity, IEEE Trans. Image Process. 13 (4)(20 04) 60 0–612 .
Liyan Sun received the B.S. degree from ZhengzhouUniversity, Zhengzhou, Henan, China in 2014. He is
currently pursuing the Ph.D. degree with the Schoolof Information Science and Engineering from Xiamen
University, Xiamen, Fujian, China. From 2018 to 2019, He
was a Visiting Scholar with the Department of ElectricalEngineering, Columbia University, New York, NY, USA.
His research interests mainly focus on machine learning,medical image reconstruction and analysis.
Yawen Wu received the B.S. degree from Fuzhou Univer-
sity, China in 2017. She is currently pursuing the masterdegree in the Department of Communication Engineering,
Xiamen University, China. Her research interests mainlyfocus on deep learning and medical image analysis.
Binglin Shu received the B.S. degree from Xiamen Univer-
sity, China in 2016. He is currently pursuing the masterdegree in the Department of Communication Engineering,
Xiamen University, China. His research interests mainly
focus on machine learning medical image analysis.
L. Sun, Y. Wu and B. Shu et al. / Neurocomputing 397 (2020) 94–107 107
Xinghao Ding was born in Hefei, China, in 1977. He
received the B.S. and Ph.D. degrees from the Depart-ment of Precision Instruments, Hefei University of Tech-
nology, Hefei, in1998 and 2003, respectively. He was a
Post-Doctoral Researcher with the Department of Electri-cal and Computer Engineering, Duke University, Durham,
NC, USA, from 2009 to 2011. Since 2011, he has been aProfessor with the School of Information Science and En-
gineering, Xiamen University, Xiamen, China. His main re-search interests include machine learning, representation
learning, medical image analysis and computer vision.
Congbo Cai received Ph.D. degrees from Xiamen Uni-versity, China in 2015. He is currently a Professor with
the School of Electronic Science and Engineering, Xia-
men University. His main research interests include med-ical image processing, magnetic resonance imaging and
spectroscopy.
Yue Huang received the B.S. degree from Xiamen Uni-
versity, Xiamen, China, in 2005, and the Ph.D. degreefrom Tsinghua University, Beijing, China, in 2010. She was
a Visiting Scholar with Carnegie Mellon University from
2015 to 2016. She is currently an Associate Professor withthe Department of Communication Engineering, School of
Information Science and Engineering, Xiamen University. Her main research interests include machine learning and
image processing.
John Paisley received the B.S., M.S., and Ph.D. degrees in
electrical engineering from Duke University, Durham, NC,USA. He was a Postdoctoral researcher with the Computer
Science Departments at University of California, Berke-ley and Princeton University. He is currently an Associate
Professor with the Department of Electrical Engineering,Columbia University, New York, NY, USA, and also a mem-
ber of the Data Science Institute, Columbia University. His
current research is machine learning, focusing on modelsand inference techniques for text and image processing
applications.