1

A Review of Convolutional Neural Networks forInverse Problems in Imaging

Michael T. McCann, Member, IEEE, Kyong Hwan Jin, Michael Unser, Fellow, IEEE

Abstract—In this survey paper, we review recent uses ofconvolution neural networks (CNNs) to solve inverse problemsin imaging. It has recently become feasible to train deep CNNson large databases of images, and they have shown outstandingperformance on object classification and segmentation tasks. Mo-tivated by these successes, researchers have begun to apply CNNsto the resolution of inverse problems such as denoising, deconvo-lution, super-resolution, and medical image reconstruction, andthey have started to report improvements over state-of-the-artmethods, including sparsity-based techniques such as compressedsensing. Here, we review the recent experimental work in theseareas, with a focus on the critical design decisions: Where doesthe training data come from? What is the architecture of theCNN? and How is the learning problem formulated and solved?We also bring together a few key theoretical papers that offerperspective on why CNNs are appropriate for inverse problemsand point to some next steps in the field.

I. INTRODUCTION

The basic ideas underlying the use of convolutional neuralnetworks (CNNs, also known as ConvNets) for inverse prob-lems are not new. Here, we give a very condensed historyof CNNs to give context to what follows. For more historicalperspective, see [1], and for an accessible introduction to deepneural networks and a summary of their recent history, see[2]. The CNN architecture was proposed in 1986 in [3] andneural networks were developed for solving inverse imagingproblems as early as 1988 [4]. These approaches, which usednetworks with a few parameters and did not always includelearning, were largely superseded by compressed sensing (or,broadly, convex optimization with regularization) approachesin the 2000s. As computer hardware improved, it became fea-sible to train larger and larger neural networks, until, in 2012,Krizhevsky et al. [5] achieved a significant improvement overthe state of the art on the ImageNet classification challengeby using a GPU to train a CNN with 5 convolutional layersand 60 million parameters on a set of 1.3 million images. Thiswork spurred a resurgence of interest in neural networks, andspecifically CNNs, for not only computer vision tasks, but alsoinverse problems and more.

Michael McCann is with the Center for Biomedical Imaging, SignalProcessing Core and the Biomedical Imaging Group, EPFL, Lausanne,Switzerland (email: michael.mccann@epfl.ch).

K.H. Jin is with the Biomedical Imaging Group, EPFL, Lausanne, Switzer-land.

K.H. Jin acknowledges the support from the “EPFL Fellows” fellowshipprogram co-funded by Marie Curie from the European Unions Horizon 2020Framework Programme for Research and Innovation under grant agreement665667.

Michael Unser is with the Biomedical Imaging Group, EPFL, Lausanne,Switzerland.

The purpose of this review is to summarize the recentworks using CNNs for inverse problems in imaging; i.e., inproblems most naturally formulated as recovering an imagefrom a set of noisy measurements; this criterion excludesdetection, segmentation, classification, quality assessment, etc.We also focus on CNNs, avoiding other architectures suchas recurrent neural networks, fully-connected networks, andstacked denoising autoencoders. We organized our literaturesearch by application, looking for topics of broad interestwhere we could find at least three peer-reviewed papers fromthe last ten years.1 The resulting applications and referencesare summarized in Table I. The aim of this constrained scopeis to allow us to draw meaningful generalizations from thesurveyed works.

TABLE IREVIEWED APPLICATIONS AND ASSOCIATED REFERENCES.

denoising deconvolution super-resolution MRI CT

[6]–[11] [10], [12]–[14] [9], [15]–[20] [21]–[23] [24]–[27]

The manuscript is organized as follows. We begin in Sec-tion II with a brief background on inverse imaging problemsand how they can be formulated as learning problems. Wecontinue in Section III, which summarizes the recent resultsobtained by using CNNs for a variety of image reconstruc-tion applications. We then survey the recent CNN-based re-construction methods in detail in Section IV, with a focuson design decisions involving the training set, the networkarchitecture, the formulation of the learning problem, and theoptimization procedure itself. We briefly cover some of thetheoretical perspectives on the good performance of CNNsfor inverse problems in Section V. We then discuss critiquesof the CNN-based approach in Section VI and conclude inSection VII with our view of the future directions for the field.

II. BACKGROUND

We begin by introducing inverse problems and contrastingthe traditional approach to solving them with a learning-based approach. For a textbook treatment of inverse problems,see [28]. Throughout the section, we use X-ray CT as arunning example, and Figure 1 shows images of the variousmathematical quantities we mention.

1 Much of the work on the theory and practice of CNNs is posted on thepreprint server arXiv.org before eventually appearing in a traditional journal.Because of the lack of peer review on arXiv.org, we have preferred not to citethese papers, except in cases where we are trying to illustrate a very recenttrend or future direction for the field. When citing a paper from arXiv, wefollow the inline citation with an asterisk, e.g. [30]*.

arX

iv:1

710.

0401

1v1

[ee

ss.I

V]

11

Oct

201

7

2

H

HT H−1 Rreg

CNN θ CNN θ CNN θ

CNN θ

? [25] [27] [25]

?

x

y

HT {y} H−1{y} x

Fig. 1. Block diagram of image reconstruction methods, using images from X-ray CT as examples. An image, x creates measurements, y, that can be usedto estimate x in a variety of ways. The traditional approach is to apply a direct inversion, H−1, which is artifact-prone in the sparse-measurement case (notethe stripes in the reconstruction). The current state of the art is a regularized reconstruction, Rreg, written in general in (2). Several recent works apply CNNsto the result of the direct inversion or an iterative reconstruction, but it might also be reasonable to use as input the measurements themselves or the backprojected measurements.

A. Learning for Inverse Problems in Imaging

Mathematically speaking, an imaging system is an operatorH : X → Y that acts on an image x ∈ X , to create a vectorof measurements y ∈ Y , with H{x} = y. The underlyingfunction/vector spaces are• the space, X , of acceptable images which can be 2D,

3D, or even 3D+time, with its values representing aphysical quantity of interest, such as X-ray attenuationor concentration of fluorophores; and

• the space, Y , of measurement vectors which depends onthe imaging operator and could include images (discretearrays of pixels), Fourier samples, line integrals, etc.

We typically consider x to be a continuous object (function ofspace), while y is usually discrete: Y = RM . For example, inX-ray CT, x is an image representing X-ray attenuations, Hrepresents the physics of the X-ray source and detector, and yis the measured sinogram (see Figure 1).

In an inverse imaging problem, we aim to develop a recon-struction algorithm (which is also an operator), R : Y → X inorder to recover the original image, x, from the measurements,y. The dominant approach for reconstruction, which we callthe objective function approach, is to model H and recoveran estimate of x from y by

Robj{y} = arg minx∈X

f(H{x}, y), (1)

where H : X → Y is the system model, which is usuallylinear, and f : Y × Y → R+ is an appropriate measure oferror. Continuing the CT example, H would be a discretizationof the X-ray transform (such as Matlab’s radon) and fcould be the Euclidean distance, ‖H{x} − y‖2. For manyapplications, decades of engineering have gone into developinga fast and reasonably accurate inverse operator, H−1, so Eq.(1) is easily solved with Robj{y} = H−1{y}; for CT, H−1

is the filtered back projection (FBP) algorithm. An important,

related operator is the back projection, HT : Y → X , whichcan be interpreted as the simplest way to put measurementsback into the image domain (see Figure 1).

These direct inverses begin to show significant artifactswhen the number or quality of the measurements decreases,either because the underlying discretization breaks down, orbecause the inversion of (1) becomes ill-posed (lacking asolution, lacking a unique solution, or being unstable withrespect to the measurements). Unfortunately, in many real-world problems, measurements are costly (in terms of time,or, e.g., X-ray damage to the patient), which motivates us tocollect as few as possible. In order to reconstruct from sparseor noisy measurements, it is often better to use a regularizedformulation,

Rreg{y} = arg minx∈X

f (H{x}, y) + g(x), (2)

where g : X → R+ is a regularization functional that promotessolutions that match our prior knowledge of x, and, simulta-neously, makes the problem well-posed. For CT, g could bethe total variation (TV) regularization, which penalizes largegradients in x.

From this perspective, the challenge of solving an inverseproblem is designing and implementing (2) for a specificapplication. Much effort has gone into designing general-purpose regularizers and minimization algorithms. For ex-ample, compressed sensing [29] provides sparsity-promotingregularizers. Nonetheless, in the worst case, a new applicationnecessitates developing accurate and efficient H , g, and f ,along with a minimization algorithm.

An alternative to the objective function approach is thelearning approach, where a training set of ground truthimages and their corresponding measurements, {(xn, yn)}Nn=1,is known. A parametric reconstruction algorithm, Rlearn, is

3

then learned by solving

Rlearn = arg minRθ,θ∈Θ

N∑n=1

f(xn, Rθ{yn}) + g(θ), (3)

where Θ is the set of all possible parameters, f : X×X → R+

is a measure of error, and g : Θ → R+ is a regularizeron the parameters with the aim of avoiding overfitting. Oncethe learning step is complete, Rlearn can then be used toreconstruct a new image from its measurements.

To summarize, in the objective function approach, the recon-struction function is itself a regularized minimization problem,while in the learning approach, the solution of a regularizedminimization problem is a parametric function that can beused to solve the inverse problem. The learning formulationis attractive because it overcomes many of the limitations ofthe objective function approach: there is no need to handcraftthe forward model, cost function, regularizer, and optimizerfrom (2). On the other hand, the learning approach requires atraining set, and the minimization (3) is typically more difficultthan (2) and requires a problem-dependant choice of f , g, andthe class of functions described by R and Θ.

Finally, we note that the learning and objective functionapproaches describe a spectrum rather than a dichotomy.In fact, the learning formulation is strictly more general,including the objective function formulation as a special case.As we will discuss further in Section IV-B, which (if any)aspects of the objective formulation approach to retain is acritical choice in the design of learning-based approaches toinverse problems in imaging.

B. Convolutional Neural Networks

Our focus here is the formulation of (3) using CNNs. Usinga CNN means, roughly, fixing the set of functions, Rθ, tobe a sequence of filtering operations alternating with simplenonlinear operations. This class of functions is parametrizedby the values of the filters used (also known as filter weights),and these filter weights are the parameters over which theminimization occurs. For illustration, Figure 2 shows a typicalCNN architecture.

We will describe some of the theoretical motivations forusing CNNs as the learning architecture for inverse problemsin Section V, but we mention some practical advantages here.First, the forward operation of a CNN consists of (usuallysmall) convolutions and simple, pointwise nonlinear functions.This means that once training is complete, the execution ofRlearn is very fast and amenable to hardware acceleration onGPUs. Second, the gradient of (3) is computable via the chainrule, and these gradients again involve small convolutions,meaning that the parameters can be learned efficiently viagradient descent.

When the first CNN-based method entered the ImageNetLarge Scale Visual Recognition Challenge in 2012 [5], itserror rate on the object localization and classification task was15.3%, as compared to an error rate 26.2% for the next closestmethod and 25.8% for the winner from 2011. In subsequentcompetitions (2013-2016), the majority of the entries (and allof the winners) were CNN-based and continued to improve

substantially, with the 2016 winner achieving an error rateof just 2.99%. Can we expect such large gains in inverseproblems? That is, can we expect denoising results to improveby an order of magnitude (20 dB) in the next few years? Inthe next section, we answer this question by surveying theresults reported by recent CNN-based approaches to imagereconstruction.

III. CURRENT STATE OF PERFORMANCE

Of the inverse problems we review here, denoising providesthe best look at recent trends in results because there arestandard experiments that appear in most papers. Work onCNN-based denoising from 2009 [6] showed an average PSNRof 28.5 on the Berkeley Segmentation Dataset, a less than1 dB improvement over contemporary wavelet and Markovrandom field-based approaches. For comparison, one very re-cent denoising work [11] reported a 0.7 dB improvement on asimilar experiment, which remains a less than 1 dB better thancontemporary non-CNN methods (including BM3D, whichhad remained the state-of-the-art for years). As another pointof reference, in 2012, one CNN approach [7] reported anaverage PSNR of 30.2 dB on a set of standard test images(Lena, peppers, etc.), less than 0.1 dB better than comparisons,and another [8], reported an average of 30.5 dB on the sameexperiment. The recent [11] achieves an average of 30.4 dBunder the same conditions. One important perspective on thesedenoising results is that the CNN is learning the distribution ofnatural images (or equivalently, is learning a regularization).Such a CNN could be reused inside an iterative optimizationas a proximal operator to enforce this learned regularizationfor any inverse problem.

The trends are similar in deblurring and super-resolution,though experiments are more varied and therefore harder tocompare. For deblurring, [12] showed around a 1 dB PSNRimprovement over comparison methods, and [13] showed afurther improvement of around 1 dB. For super-resolution,work from 2014 [15] reported a less than 0.5 dB improvementin PSNR over comparisons. In the next two years, [16] and[19] both reported a 0.5 dB PSNR increase over this baseline.Even more recent work, [30]*, improves the 2014 work byaround 1.5 dB in PSNR. For video super-resolution, [18]improves on non-CNN-based methods by about 0.5 dB PSNRand [20] improves upon that result by another 0.5 dB.

For inverse problems in medical imaging, direct comparisonbetween works is impossible due to the wide variety of exper-imental setups. A 2013 CNN-based work [24] shows improve-ment in limited-view CT reconstruction over direct methodsand unregularized iterative methods, but does not compare toregularized iterative methods. In 2015, [25] showed in full-view CT an improvement of several dB in SNR over directreconstruction and around 1 dB improvement over regularizediterative reconstruction. Recently, [26] showed about 0.5 dBimprovement in PSNR over TV-regularized reconstruction,while [27] showed a larger (1-4 dB) improvement in SNRover a different TV-regularized method (Figure 3). In MRI,[22] demonstrates performance equal to the state-of-the-art,with advantages in running time.

4

Fig. 2. Illustration of a typical CNN architecture for 2562 px RGB images, including the objective function used for training. T (·) is the ReLU function(point-wise nonlinear function). ◦ denotes a 2-D convolution. The convolutions in each layer is described by a 4-D tensor representing a stack of 3D filters.

Do these improvements matter? CNN-based methods havenot, so far, had the profound impact on inverse problemsthat they have for object classification. Indeed, the differencebetween 30 and 30.5 dB is impossible to see by eye. Onthe other hand, these improvements occur in heavily studiedfields: we have been denoising the Lena image since the1970s. Further, CNNs offer some unique advantages overmany traditional methods. The design of the CNN architecturecan be more or less decoupled from the application at handand can be reused from problem to problem. They can alsobe expanded in straightforward ways as computer memorygrows and there is some evidence that larger networks lead tobetter performance. Finally, once trained, running the model isfast (dozens of convolutions per image, usually less than onesecond). This means that CNN-based methods can be attractivein terms of running time even if they do not improve uponstate-of-the-art performance.

IV. DESIGNING CNNS FOR INVERSE PROBLEMS

In this section, we survey the design decisions neededto develop CNN-based approaches for inverse problems inimaging. We organize the section around the learning equationas summarized in Figure 4, first describing how the trainingset is created, then how the network architecture is designed,and then how the learning problem is formulated and solved.

A. Training Set

Learning requires a suitable training set, i.e. the (input,output) pairs from which the CNN will learn. In a typicallearning problem, training outputs are provided by some oraclelabeling a set of inputs. For example, in object classification, aset of human graders might view a large number of images andprovide annotations for each. In the inverse problem setting,this is considerably more difficult because no such oracleexists. For example, in X-ray CT, to generate a training set

we would need to image a large number of physical phantomsfor which we have exact 3D models, which is not feasiblein practice. The choice of the training set also constrainsthe network architecture because the input and output of thenetwork must match the dimensions of yn and xn, respectively.

1) Generating Training Data: In some cases, generatingtraining data is straightforward because the forward modelwe aim to invert is known exactly and easily computable.In denoising, training data is generated by corrupting imageswith noise; the noisy image then serves as training inputand the clean image as the training output, as in, e.g., [6],[7]. Or, the noise itself can serve as the oracle output, in ascheme called residual learning [23], [11]. super-resolutionfollows the same pattern, where training pairs are easilygenerated by downsampling, as in, e.g., [19]. The same istrue for deblurring, where training pairs can be generated byblurring [12]–[14].

In medical imaging, the focus is on reconstructing fromreal measurements and the corresponding ground truth is notusually known. The emerging paradigm is to learn to recon-struct from sparse measurements, using reconstructions fromfully-sampled measurements to train. For example, in MRIreconstruction, [22] trains using under-sampled k-space dataas inputs and reconstructions from fully-sampled k-space dataas outputs. Likewise, [27] uses a low-view CT reconstructionas input and a high-view CT reconstruction as output. Or, theCNN can learn from low-dose (noisy) measurements [25].

2) Preprocessing: Another aspect of training data prepara-tion is whether the training inputs are the measurements them-selves, or whether some preprocessing occurs. In denoising, itis natural to use the raw measurements, which are of the samedimensions as the reconstruction. But, in the other applica-tions, the trend is to use a direct inverse operator to preprocessthe network input. Following the notation in Section II-A, thiscan be viewed as a combination of the objective function and

5

Fig. 3. An example of X-ray CT reconstructions. The ground truth (left column) comes from an FBP reconstruction using 1000 views. The next three columnsshow reconstructions from just 50 views using FBP, a regularized reconstruction, and from a CNN-based approach (images reproduced with permission from[27]). The CNN-based reconstruction preserves more of the texture present in the ground truth and results in a significant increase in SNR.

Fig. 4. The learning equation, repeated from the introduction, which we use to organize the parts of Section IV.

learning approach, where instead of Rlearn being a CNN, it isthe composition of a CNN with a direct inverse: Rθ◦H−1. Forexample, in super-resolution, [16], [18], [19] first upsampleand interpolate the low-resolution input images; in CT, [27]and [25] preprocess with the FBP, [25] also preprocesses withan iterative reconstruction; and, in MRI, [21] preprocesseswith the inverse Fourier transform.

Without preprocessing, the CNN must learn the underlyingphysics of the inverse problem. It is not even clear that this ispossible with CNNs (e.g., what is the meaning of filtering anX-ray CT sinogram?). Preprocessing is also a way to leveragethe significant engineering effort that has gone into designingthese direct inverses over the past decades. Superficially, thistype of preprocessing appears to be inversion followed bydenoising, which is a standard, if ad hoc, approach to inverseproblems. What is unique here is that the artifacts caused bydirect inversion, especially in the sparse measurement case, areusually highly structured and therefore not good candidates forgeneric denoising approaches. Instead, the CNN is allowed tolearn the specific character of these artifacts.

A practical aspect of preprocessing is controlling the dy-namic range of the input. While not typically a problem whenworking with natural images or standardized datasets, theremay be huge fluctuations in the intensity or contrast of themeasurements in certain inverse problems. To avoid a smallset of images dominating the error during training, it is bestto scale the dynamic range of the training set [23], [27].Similarly, it may be advantageous to discard training patcheswithout sufficient contrast.

3) Training Size: CNNs typically have at least thousands ofparameters to train; thus, the number of (input, output) pairs inthe training set is of important practical concern. The numberof training pairs varied among the papers we surveyed. Thebiomedical imaging papers tended to have the fewest samples(e.g., 500 brain images [21] or 2000 CT images [24]), whilepapers on natural images had the most (e.g., pretraining on395,909 natural images [20]).

A further complication is that, depending on the networkarchitecture, images may be split into patches for training.Thus, depending on the dimensions of the training images

6

and the chosen patch size, numerous patches can be createdfrom a small training set. The patch size also has importantramifications for the performance of the network and is linkedto its architecture, with larger filters and deeper networksrequiring larger training patches [17].

With a large CNN and a small training set, overfitting mustbe avoided by regularization during learning and/or the use ofa validation set (e.g., [24]) (discussed more in Sections IV-Cand IV-D). These strategies are necessary to produce a CNNthat generalizes at all, but they do not overcome the fact thatthe performance of the CNN will be limited by the size andvariety of the training set. One strategy to increase the trainingset size is data augmentation, where new (input, output) pairsare generated by transforming existing ones. For example,[20] augmented training pairs by scaling them in space andtime, turning 20,000 pairs into 70,000 pairs. The augmentationmust be application-specific because the trained network willbe approximately invariant to the transforms used. Anotherstrategy to effectively increase the training set size is to usea pretrained network. For example, [18] first trains a CNNfor image super-resolution with a large image dataset, thenretrains with videos.

B. Network Architecture

By network architecture, we mean the choice of the familyof CNNs, Rθ parameterized by θ. In our notation, Rθ rep-resents a CNN with a specific architecture while θ are theweights to be learned during the training. There is great varietyamong CNN-based methods regarding their architecture: howmany convolutional layers, what filter sizes, which nonlinear-ities, etc. For example, [19] uses 8,032 parameters, while [20]uses on the order of one hundred thousand. In this section,we survey recent approaches to CNN architecture design forinverse problems.

The simplest approach to architecture design is simply stackof series of convolutional layers and non-linear functions [10],[26], see Figure 2. This provides a baseline to check thefeasibility of the network for the given application. It isstraightforward to adjust the size of such a network, eitherby changing the number of layers, the number of channelsper layer, or the size of the filters in each layer. For example,keeping the filters small (3× 3× 3 px) allows the network tobe deeper for a given number of parameters [23]; constrainingthe filters to be separable [12] further reduces the numberof parameters. Doing this can give the experimenter a senseof the training time required on their hardware as well asthe effects of the network size on performance. From thissimple starting point, the architecture can be tweaked forgreater performance; for example, by adding downsamplingand upsampling operations [27], or by simply adding morelayers [20].

Instead of using ad hoc architecture design, one can adapta successful CNN architecture from another application. Forexample, [27] adapts a network designed for biomedical imagesegmentation to CT reconstruction by changing the number ofoutput layers from two (background and foreground images)to one (reconstructed image). These architectures can also be

connected end-to-end, creating modular or hierarchical de-signs. For example, a four-times super-resolution architecturecan be created by connecting two two-times super-resolutionnetworks [16]. This is distinct from training a two-times super-resolution network and applying it twice because the twomodules of the CNN are trained as a unit.

A second approach is to begin with an iterative optimizationalgorithm and unroll it, turning each iteration into a layerof a network. In such a scheme, filters that are normallyfixed in the iterative minimization are instead learned. Theapproach was pioneered in [31], for sparse coding; their resultsshowed that the learned algorithms could achieve a givenerror in fewer iterations than the standard ones. Because manyiterative optimization algorithms alternate filtering steps (linearupdates) with pointwise nonlinear steps (proximal/shrinkageoperations), the resulting network is often a CNN. This wasthe approach in [22], where the authors unrolled the ADMMalgorithm to design a CNN for MRI reconstruction, withstate-of-the-art results and improvements in running time. Fornetworks designed in this way, the original algorithm is aspecific case and, therefore, the performance of the networkcannot be worse than the original algorithm if training issuccessful. The concept of unrolling can also be applied ata coarser scale, as in [13], where the modules of the networkmimic the steps of a typical blind deconvolution pipeline:extract features, estimate kernel, estimate image, repeat.

Another promising design approach, similar to unrolling,is to learn only some part of an existing iterative method.For example, given the modular nature of popular iterativeoptimization schemes such as the ADMM, a CNN can beemployed as a proximal (denoising) operator while the rest ofthe algorithm remains unchanged [32]*.This design combinesmany of the good aspects of both the objective function andlearning-based approaches, and allows a single CNN to beused for several different inverse problems without retraining.

C. Cost Function and Regularization

In this section, we survey the approaches taken to actuallytrain the CNN, including the choice of a cost function, f ,and regularizer, g. For a textbook coverage of the subject oflearning, see [33].

Understanding the learning minimization problem as a sta-tistical inference can provide useful insight into the selectionof the cost and regularization functions. From this perspec-tive, we can formulate the goal of learning as maximizingthe conditional likelihood of each training output given thecorresponding training input and CNN parameters,

given {(xn, yn)}Nn=1,

Rlearn = arg maxRθ,θ∈Θ

N∏n=1

P (yn | xn, θ),

where P is a conditional likelihood. When this likelihood fol-lows a Gaussian distribution, this optimization is equivalent tothe one from the introduction, (3), with f being the Euclideandistance and no regularization. Put another way, learning withthe standard, Euclidean cost and no regularization implicitly

7

assumes a Gaussian noise model; this is a well-known fact ininverse problems in general. This formulation is used in mostof the works we surveyed, [6], [7], [11], [12], [18], [19], [23],[25], [26], despite the fact that several raise questions aboutwhether it is the best choice [25], [34].

An alternative is the maximum a posteriori formulation,which maximizes the joint probability of the training data andthe CNN parameters, which can be decomposed into severalterms using Bayes rule,

given {(xn, yn)}Nn=1,

Rlearn = arg maxRθ,θ∈Θ

N∏n=1

P (yn | xn, θ)P (θ).(4)

This formulation explicitly allows prior information about thedesired CNN parameters, θ, to be used. Under a Gaussianmodel for the weights of the CNN as well as the noise,this formulation results in a Euclidean cost function and aEuclidean regularization on the weights of the CNN, g(θ) =σ−2‖θ‖22. Other examples of regularizations for CNNs are thetotal generalized variation norm or sparsity on the coefficients.Regularized approaches are taken in [10], [15], [21], [22].

D. Optimization

Once an objective function for learning has been fixed, itremains to actually minimize it. This is a crucial and deeptopic, but, from the practical perspective, it can be treatedas a black box due to the availability of several high-qualitysoftware libraries that can perform efficient training of user-defined CNN architectures. For a comparison of these libraries,refer to [35]*; here, we provide a basic overview.

The popular approaches to CNN learning are variations ongradient descent. The most common is stochastic gradientdescent (SGD), used in [16], [25], where, at each iteration, thegradient of the cost function is computed using random subsetsof the available training. This reduces the overall computationcompared to computing the true gradient, while still providinga good approximation. The process can be further tuned byadding momentum, i.e., combining gradients from previousiterations in clever ways, or by using higher order gradientinformation as in BFGS [22].

Initial weights can be set to zero, or chosen from somerandom distribution (Gaussian or uniform). Because learningis nonconvex, the initialization does potentially change whichminimum the network converges to, but, not much differenceis observed in practice. However, good initializations canimprove the speed of convergence. This explains the popularityof taking pretrained networks, or, in the case of an unrolledarchitecture, initializing the network weights based on cor-responding known filters. Recently, a procedure called batchnormalization, where the inputs to each layer of the networkare normalized, was proposed as a way to increase learningspeed and reduce sensitivity to initialization [36].

As mentioned is Section IV-A, overfitting is a serious riskwhen training networks with potentially millions of parame-ters. In addition to augmenting the training set, steps can betaking during training to reduce overfitting. The simplest is tosplit the training data into a set used for optimization and a

set used for validation. During training, the performance ofthe network on the validation set is monitored and training isterminated when the performance on the validation set beginsto drop. Another method is dropout [37], where individualunits of the network are randomly deleted during training.The motivation for dropout is the idea that the network shouldbe regularized by forming a weighted average of all possibleparameter settings, with weights determined by their perfor-mance. While this regularization is not feasible, removingunits during training provides a reasonable approximation thatperforms well in practice.

V. THEORY

The excellent performance of CNNs for various applicationsis undisputed, but the question of why remains mostly unan-swered. Here, we bring together a few different theoreticalperspectives that begin to explain why CNNs are a good fitfor solving inverse problems in imaging.

1) Universal approximation: We know that neural net-works are universal approximators. More specifically, a fully-connected neural network with one hidden layer can approxi-mate any continuous function arbitrarily well provided that itshidden layer is large enough [38]. The result does not directlyapply to CNNs because they are not fully connected, but, if weconsider the network patch by patch, we see that each inputpatch is mapped to the corresponding output patch by a fullyconnected network. Thus, CNNs are universal approximatorsfor shift-invariant functions. From this perspective, statementssuch as “CNNs work well because they generalize X algo-rithm” are vacuously true because CNNs generalize all shift-invariant algorithms. On the other hand, the notion of universalapproximation tells us what the network can learn, not what itdoes learn, and comparison to established algorithms can helpguide our understanding of CNNs in practice.

2) Unrolling: The most concrete perspective on CNNs asgeneralizations of established algorithms comes from the ideaof unrolling, which we discussed in Section IV-B. The ideaoriginated in [31], where the authors unrolled the ISTA algo-rithm for sparse coding into a neural network. This network isnot a typical CNN because it includes recurrent connections,but it does share the alternating linear/nonlinear motif. A moregeneral perspective is that nearly all state-of-the-art iterativereconstruction algorithms alternate between linear steps andpointwise nonlinear steps, so it follows that CNNs should beable to perform similarly well given appropriate training. Onerefinement of this idea comes from [27], which establishesconditions on the forward model, H , that ensure that thelinear step of the iterative method is a convolution. All ofthe inverse problems surveyed here meet these conditions,but the theory predicts that certain inverse problems, e.g.structured illumination microscopy, should not be amenableto reconstruction via CNNs. Another refinement concerns thepopular rectified linear unit (ReLU) employed as the non-linearity by most CNNs: results from spline theory can beadapted to show that combinations of ReLUs can approximateany continuous function. This suggests that the combinationsof ReLUs usually employed in CNNs are able to closely

8

approximate the proximal operators used in traditional iterativemethods.

3) Invariance: Another perspective comes from work onscattering transforms, which are cascades of linear opera-tions (convolutions with wavelets) and nonlinearities (abso-lute value) [39] with no combinations formed between thedifferent channels. This simplified model shows invariance totranslation and, more importantly, to small deformations ofthe input (diffeomorphisms). CNNs generalize the scatteringtransform, giving the potential for additional invariances, e.g.,to rigid transformations, frequency shifts, etc. Such invariancesare attractive for image classification, but more work is neededto connect these results to inverse problems.

VI. CRITIQUES

While the papers we have surveyed present many reasonsto be optimistic about CNNs for inverse problems, we alsowant to mention a few general critiques of the approach. Wehope these can be useful points to think about when writingor reviewing manuscripts in the area, as well as jumping-offpoints for future research.

1) Algorithm Descriptions and Reproducibility: Whenplanning this survey, we aimed to measure quantitative trendsin the literature, e.g., to plot the number of training samplesversus the number of parameters for each network. We quicklydiscovered this is nearly impossible. Very few manuscriptsclearly noted the number of parameters they were training,and only some provided a clear-enough description of thenetwork for us to calculate the value. Many more includeda figure of network architecture along the lines of Figure 2,but without a clear statement of the dimensions of each layer.Similar problems exist in the description of the training andevaluation procedures, where it is not always clear whether theevaluation data comes from simulation or from a real dataset.As the field matures, we hope papers converge on a standardway to describe network architecture, training, and evaluation.

The lack of clarity presents a barrier to the reproducibilityof the work. Another barrier is the fact that training oftenrequires specialized or expensive hardware. While GPUs havebecome more ubiquitous, the largest (and best-performing)CNNs remain difficult for small research groups to train. Forexample, the CNN that won the ImageNet Large-Scale VisualRecognition Challenge in 2012 took “between five and sixdays to train on two GTX 580 3GB GPU” [5].

2) Robustness of Learning: The success of any CNN-based algorithm hinges on finding a reasonable solution tothe learning problem, (3). As stated before, this is a non-convex problem, where the best solution we can hope for isto find one of many local minima of the cost. This raisesquestions about the robustness of the learning to changesin the initialization of parameters and the specifics of theoptimization method employed. This is in contrast to thetypical convex formulations of inverse problems, where thespecifics of the initialization and optimization scheme provablydo not affect the quality of the result.

The uncertainty about learning complicates the comparisonof any two CNN-based methods. Does A outperform B

because of its superior architecture, or simply because theoptimization of A fell into a superior local minimum? As anexample of the confusion this can cause, [34] shows, in thecontext of denoising, super-resolution, and JPEG deblocking,that a network trained with the l1 cost function can outperforma network trained with the l2 cost function even with regardto the l2 cost. In the authors’ analysis of this highly disturbingresult, they attribute it to the l2 learning being stuck in alocal optimum. Regardless, the vast majority of work relies onthe l2 cost, which is computationally convenient and providesexcellent results.

There is some indication that large networks trained withlots of data can overcome this problem. In [40], the authorsshow that larger networks have more local minima, but thatmost local minima are equivalent in terms of testing per-formance. They also identify that the global minima likelycorrespond to parameter settings that overfit the training set.More work on the stability of the learning process will bean important step towards wider acceptance of CNNs in theinverse problem community.

More generally, how sensitive are the results of a givenexperiment to small changes in the training set, networkarchitecture, or optimization procedure? Is it possible for theexperimenter to overfit the testing set by iteratively tweakingthe network architecture (or the experimental parameters)until state-of-the-art results are achieved? To combat this,CNN-based approaches should provide carefully-constructedexperiments with results reported on a large number of testingimages. Even better are competition datasets where the testingdata is hidden until algorithm development is complete.

3) Can We Trust the Results?: Once trained, CNNs remainnon-linear and highly complex. Can we trust reconstructionsgenerated by such systems? One way to look at this is toevaluate the sensitivity of the network to noise: ideally, smallchanges to the input should cause only small changes to theoutput; data augmentation during training can help achievethis. Similarly, demonstrating generalization between datasets(where the network learns on one dataset, but is evaluated onanother) can help improve confidence in the results by showingthat the performance of the network is not dependent on somesystematic bias of the dataset.

A related question is how to measure the quality of the re-sults. Even if a robust SNR improvement can be demonstrated,practitioners will inevitably want to know, e.g., whether theresulting images can be reliably used for diagnosis. To thisend, as much as possible, methods should be accessed withrespect to the ultimate application of the reconstruction (di-agnosis, quantification of biological phenomenon, etc.) ratherthan an intermediate measure such as SNR or SSIM. Whilethis critique can be made of any approach to inverse problems,it is especially relevant for CNNs because they are oftentreated as black boxes, and because the reconstructions theygenerate are plausible-looking by design, hiding areas wherethe algorithm is less sure of the result.

VII. NEXT STEPS

We have so far given a small look into the wide varietyof ways that researchers have applied CNNs to solve inverse

9

problems in imaging. Because CNNs are so powerful andflexible, we believe there remains plenty of room to createeven better systems. In this final section, we suggest a fewdirections that this future research might take.

1) Biomedical Imaging: CNNs have so far been appliedmost to inverse problems where the measurements take theform of an image and where the measurement model is simple,and less so for CT and MRI, which have relatively morecomplicated models. A search on arXiv.org reveals dozensmore CT and MRI papers submitted within the last fewmonths, suggesting many incoming contributions in theseareas. We expect diffusion into other modalities such as PET,SPECT, optical tomography, TEM, SIM, ultrasound, super-resolution microscopy, etc. to follow.

Central to this work will be questions of how best tocombine CNNs with knowledge of the underlying physics aswell as direct and iterative inversion techniques. Most of thesurveyed works involve using a CNN to correct the artifactscreated by a direct or iterative methods, where it remainsan open question what is the best such prereconstructionmethod. One creative approach is to build the inverse operatorinto the network architecture as in [22], where the networkcan compute inverse Fourier transform. Another would be touse the back projected measurements, HT y, which at leasttake the form of an image and could reduce the burden onthe CNN to learn the physics of the forward model. CNNscould be deployed in a variety of other ways here, too, e.g.using a CNN to approximate a high quality, but extremelyslow reconstruction method. With enough computing power,a training set could be generated by running the slow methodon real data, and, once trained, the resulting network couldprovide very fast and accurate reconstructions.

2) Cross-Task Learning: In cross-task learning (also calledtransfer learning, though this can have other meanings aswell), an algorithm is trained with one dataset and deployedon a different, but related, task. This is attractive in the inverseproblem setting because it avoids the costly retraining of thenetwork when imaging parameters change (different noiselevels, image dimensions, etc.), which may occur often. Or,we could imagine a network that transfers between completelydifferent imaging modalities, especially when training datafor the target modality is scarce; e.g., a network could trainon denoising natural images and then be used to reconstructMRI images. Recent work has made progress in the directionby learning a CNN-based proximal operator which can beused inside an iterative optimization method for any inverseproblem [32]*.

3) Multidimensional Signals: Modern inverse problems inimaging increasing involve reconstruction of 3D or 3D+timeimages. However, most CNN-based approaches to these prob-lems involve 2D inputs and outputs. This is likely becausemuch of the work on deep neural networks in general has beenin 2D, and because of practical considerations. Specifically,learning strongly relies GPU computation, but current GPUshave maximally 24 GB of physical memory. This limitationmakes training a large network with 3D inputs and outputsinfeasible.

One way to overcome this issue is model parallelism, in

which a large model is partitioned onto separable comput-ers. Another is data parallelism, where it is the data thatis split. When used together, large computational gains areachieved [41]. Such approaches will be key in tackling multi-dimensional imaging problems.

4) Generative Adversarial Networks and Perceptual Loss:CNN-based approaches to inverse problems also stand to ben-efit from new developments neural network research. One suchdevelopment is the generative adversarial network (GAN) [42],which may offer a way to break current limits in supervisedlearning. Basically, two networks are trained in competition,the generator tries to learn a mapping between training sam-ples, while the discriminator attempts to distinguish betweenthe output of the generator and real data. Such a setup can,e.g., produce a generator capable of creating plausible naturalimages from noise. The GAN essentially revises the learningformulation (3) by replacing the cost function f with anotherneural network. In contrast to a designed cost function, whichwill be suboptimal if the assumed noise model is incorrect, thediscriminator network will act as a learned cost function thatcorrectly models the probability density function of the realdata from. GANs have already begun to be used for inverseproblems, e.g., for super-resolution in [30]* and deblurring in[14].

A related approach is perceptual loss, where a networkis trained to compute a loss function that matches humanperception. The method has already been used for styletransfer and super-resolution [43]. Compared to the standardEuclidean loss, networks trained with perceptual loss givebetter looking results, but do not typically improve the SNR.It remains to be seen whether these ideas can gain acceptancefor applications such as medical imaging, where the resultsmust be quantitatively accurate.

REFERENCES

[1] J. Schmidhuber, “Deep learning in neural networks: An overview,”Neural Netw., vol. 61, pp. 85–117, Jan. 2015.

[2] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521,no. 7553, pp. 436–444, May 2015.

[3] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internalrepresentations by error propagation,” in Parallel Distributed Process-ing: Explorations in the Microstructure of Cognition, Vol. 1. Cambridge,MA, USA: MIT Press, 1986, pp. 318–362.

[4] Y. T. Zhou, R. Chellappa, A. Vaid, and B. K. Jenkins, “Image restorationusing a neural network,” IEEE Trans. Acoust., Speech, Signal Process.,vol. 36, no. 7, pp. 1141–1151, Jul. 1988.

[5] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classificationwith Deep Convolutional Neural Networks,” in Advances in NeuralInformation Processing Systems 25, 2012, pp. 1097–1105.

[6] V. Jain and S. Seung, “Natural Image Denoising with ConvolutionalNetworks,” in Advances in Neural Information Processing Systems 21,2009, pp. 769–776.

[7] H. C. Burger, C. J. Schuler, and S. Harmeling, “Image denoising: Canplain neural networks compete with BM3D?” in 2012 IEEE Conferenceon Computer Vision and Pattern Recognition, Jun. 2012, pp. 2392–2399.

[8] J. Xie, L. Xu, and E. Chen, “Image Denoising and Inpainting withDeep Neural Networks,” in Advances in Neural Information ProcessingSystems 25, 2012, pp. 341–349.

[9] R. Wang and D. Tao, “Non-Local Auto-Encoder With CollaborativeStabilization for Image Restoration,” IEEE Trans. Image Process.,vol. 25, no. 5, pp. 2117–2129, May 2016.

[10] Y. Chen and T. Pock, “Trainable Nonlinear Reaction Diffusion: AFlexible Framework for Fast and Effective Image Restoration,” IEEETrans. Pattern Anal. Mach. Intell., vol. PP, no. 99, pp. 1–1, 2016.

10

[11] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a Gaus-sian Denoiser: Residual Learning of Deep CNN for Image Denoising,”IEEE Trans. Image Process., vol. PP, no. 99, pp. 1–1, 2017.

[12] L. Xu, J. S. Ren, C. Liu, and J. Jia, “Deep Convolutional NeuralNetwork for Image Deconvolution,” in Advances in Neural InformationProcessing Systems 27, 2014, pp. 1790–1798.

[13] C. J. Schuler, M. Hirsch, S. Harmeling, and B. Schlkopf, “Learning toDeblur,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 7, pp.1439–1451, Jul. 2016.

[14] K. Schawinski, C. Zhang, H. Zhang, L. Fowler, and G. K. Santhanam,“Generative adversarial networks recover features in astrophysical im-ages of galaxies beyond the deconvolution limit,” Monthly Notices of theRoyal Astronomical Society: Letters, vol. 467, no. 1, pp. L110–L114,May 2017.

[15] Z. Cui, H. Chang, S. Shan, B. Zhong, and X. Chen, “Deep NetworkCascade for Image Super-resolution,” in Computer Vision ECCV 2014.Springer, Cham, Sep. 2014, pp. 49–64.

[16] Z. Wang, D. Liu, J. Yang, W. Han, and T. Huang, “Deep Networks forImage Super-Resolution With Sparse Prior,” in Proceedings of the IEEEInternational Conference on Computer Vision, 2015, pp. 370–378.

[17] J. Kim, J. Kwon Lee, and K. Mu Lee, “Accurate Image Super-ResolutionUsing Very Deep Convolutional Networks,” in 2016 IEEE Conferenceon Computer Vision and Pattern Recognition, 2016, pp. 1646–1654.

[18] A. Kappeler, S. Yoo, Q. Dai, and A. K. Katsaggelos, “Video Super-Resolution With Convolutional Neural Networks,” IEEE Trans. Comput.Imaging, vol. 2, no. 2, pp. 109–122, Jun. 2016.

[19] C. Dong, C. C. Loy, K. He, and X. Tang, “Image Super-Resolution UsingDeep Convolutional Networks,” IEEE Trans. Pattern Anal. Mach. Intell.,vol. 38, no. 2, pp. 295–307, Feb. 2016.

[20] D. Li and Z. Wang, “Video Super-Resolution via Motion Compensationand Deep Residual Learning,” IEEE Trans. Comput. Imaging, vol. PP,no. 99, pp. 1–1, 2017.

[21] S. Wang, Z. Su, L. Ying, X. Peng, S. Zhu, F. Liang, D. Feng, andD. Liang, “Accelerating magnetic resonance imaging via deep learning,”in 2016 IEEE 13th International Symposium on Biomedical Imaging(ISBI), Apr. 2016, pp. 514–517.

[22] Y. Yang, J. Sun, H. Li, and Z. Xu, “Deep ADMM-Net for CompressiveSensing MRI,” in Advances in Neural Information Processing Systems29. Curran Associates, Inc., 2016, pp. 10–18.

[23] O. Oktay, W. Bai, M. Lee, R. Guerrero, K. Kamnitsas, J. Caballero,A. d. Marvao, S. Cook, D. ORegan, and D. Rueckert, “Multi-input Car-diac Image Super-Resolution Using Convolutional Neural Networks,”in Medical Image Computing and Computer-Assisted Intervention –MICCAI 2016. Springer, Cham, Oct. 2016, pp. 246–254.

[24] D. M. Pelt and K. J. Batenburg, “Fast Tomographic Reconstruction FromLimited Data Using Artificial Neural Networks,” IEEE Trans. ImageProcess., vol. 22, no. 12, pp. 5238–5251, Dec. 2013.

[25] D. Boublil, M. Elad, J. Shtok, and M. Zibulevsky, “Spatially-AdaptiveReconstruction in Computed Tomography Using Neural Networks,”IEEE Trans. Med. Imag., vol. 34, no. 7, pp. 1474–1485, Jul. 2015.

[26] H. Chen, Y. Zhang, W. Zhang, P. Liao, K. Li, J. Zhou, and G. Wang,“Low-dose CT via convolutional neural network,” Biomed. Opt. Express,vol. 8, no. 2, pp. 679–694, Feb. 2017.

[27] K. H. Jin, M. T. McCann, E. Froustey, and M. Unser, “Deep convo-lutional neural network for inverse problems in imaging,” 2017, to bepublished.

[28] A. Kirsch, An introduction to the mathematical theory of inverseproblems. Springer Science & Business Media, 2011, vol. 120.

[29] E. J. Candes, J. Romberg, and T. Tao, “Robust uncertainty principles:exact signal reconstruction from highly incomplete frequency informa-tion,” IEEE Trans. Inf. Theory, vol. 52, no. 2, pp. 489–509, Feb. 2006.

[30] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta,A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, “Photo-Realistic Sin-gle Image Super-Resolution Using a Generative Adversarial Network,”arXiv:1609.04802 [cs, stat], Sep. 2016.

[31] K. Gregor and Y. LeCun, “Learning fast approximations of sparse cod-ing,” in Proceedings of the 27th International Conference on MachineLearning (ICML-10), 2010, pp. 399–406.

[32] K. Zhang, W. Zuo, S. Gu, and L. Zhang, “Learning Deep CNN DenoiserPrior for Image Restoration,” arXiv:1704.03264 [cs], Apr. 2017.

[33] T. M. Mitchell, Machine Learning. New York: McGraw-Hill Education,Mar. 1997.

[34] H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss Functions for ImageRestoration With Neural Networks,” IEEE Trans. Comput. Imaging,vol. 3, no. 1, pp. 47–57, Mar. 2017.

[35] S. Bahrampour, N. Ramakrishnan, L. Schott, and M. Shah, “ComparativeStudy of Deep Learning Software Frameworks,” arXiv:1511.06435 [cs],Nov. 2015.

[36] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deepnetwork training by reducing internal covariate shift,” in InternationalConference on Machine Learning, 2015, pp. 448–456.

[37] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhut-dinov, “Dropout: A Simple Way to Prevent Neural Networks fromOverfitting,” J. Mach. Learn. Res., vol. 15, no. 1, pp. 1929–1958, Jan.2014.

[38] K. Hornik, “Approximation Capabilities of Multilayer FeedforwardNetworks,” Neural Netw., vol. 4, no. 2, pp. 251–257, Mar. 1991.

[39] S. Mallat, “Understanding deep convolutional networks,” Phil. Trans. R.Soc. A, vol. 374, no. 2065, p. 20150203, Apr. 2016.

[40] A. Choromanska, M. Henaff, M. Mathieu, G. Ben Arous, and Y. LeCun,“The loss surfaces of multilayer networks,” in AISTATS, 2015, pp. 192–204.

[41] J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior,P. Tucker, K. Yang, Q. V. Le et al., “Large scale distributed deepnetworks,” in Advances in neural information processing systems, 2012,pp. 1223–1231.

[42] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Nets,” inAdvances in Neural Information Processing Systems 27, Z. Ghahramani,M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, Eds.Curran Associates, Inc., 2014, pp. 2672–2680.

[43] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual Losses for Real-TimeStyle Transfer and Super-Resolution,” in Computer Vision ECCV 2016,ser. Lecture Notes in Computer Science. Cham, Switzerland: Springer,Oct. 2016, pp. 694–711.

PLACEPHOTOHERE

Michael McCann (S’10-M’15) received the B.S.E.in biomedical engineering in 2010 from the Univer-sity of Michigan and the Ph.D. degree in biomedicalengineering from Carnegie Mellon University in2015. He is currently a scientist with the Labora-toire d’imagerie biomedicale and Centre d’imageriebiomedicale, Ecole polytechnique federale de Lau-sanne (EPFL), where he works on X-ray CT recon-struction. His research interest centers on developingsignal processing tools to answer biomedical ques-tions.

PLACEPHOTOHERE

Kyong Hwan Jin received the B.S. and the inte-grated M.S. & Ph.D. degrees from the Departmentof Bio and Brain Engineering, KAIST - KoreaAdvanced Institute of Science and Technology, Dae-jeon, South Korea, in 2008 and 2015, respectively.He was a post doctoral scholar in KAIST from 2015to 2016. He is currently a post doctoral scholar inthe Biomedical Imaging Group, Ecole polytechniquefederale de Lausanne (EPFL), Switzerland. His re-search interests include low rank matrix completion,sparsity promoted signal recovery, sampling theory,

biomedical imaging, and image processing in various applications.

11

PLACEPHOTOHERE

Michael Unser (M’89-SM’94-F’99) is professorand director of EPFL’s Biomedical Imaging Group,Lausanne, Switzerland. His primary area of investi-gation is biomedical image processing. He is inter-nationally recognized for his research contributionsto sampling theory, wavelets, the use of splines forimage processing, stochastic processes, and compu-tational bioimaging. He has published over 250 jour-nal papers on those topics. He is the author with P.Tafti of the book An introduction to sparse stochasticprocesses, Cambridge University Press 2014. From

1985 to 1997, he was with the Biomedical Engineering and InstrumentationProgram, National Institutes of Health, Bethesda USA, conducting researchon bioimaging. Dr. Unser has held the position of associate Editor-in-Chief(2003-2005) for the IEEE Transactions on Medical Imaging. He is currentlymember of the editorial boards of SIAM J. Imaging Sciences, IEEE J. SelectedTopics in Signal Processing, and Foundations and Trends in Signal Processing.He is the founding chair of the technical committee on Bio Imaging and SignalProcessing (BISP) of the IEEE Signal Processing Society. Prof. Unser is afellow of the IEEE (1999), an EURASIP fellow (2009), and a member ofthe Swiss Academy of Engineering Sciences. He is the recipient of severalinternational prizes including three IEEE-SPS Best Paper Awards and twoTechnical Achievement Awards from the IEEE (2008 SPS and EMBS 2010).