Gaussian Processes and Bayesian InferenceApplied to Deep Image Priors

Diana Na Kyoung Lee

March 5, 2020

1 Introduction

Deep image priors are powerful convolutional neural nets that can perform image enhancingtasks with no training data, aside from the input image itself. In the terrain of deep learning,where models often rely on massive banks of training images in order to learn, such self-sufficiency can be considered revolutionary.

In this project, I will first learn about Gaussian processes and Bayesian inference, twomathematical concepts that can help explain the effectiveness of deep image priors. Then, Iwill examine their specific applications to deep image priors by reviewing two papers, “DeepImage Prior” [5] and “A Bayesian Perspective on the Deep Image Prior” [1].

Deep image priors are potent tools for image restoration tasks, which can include denois-ing, super resolution, and in-painting, as visualized in Figure 1. This technology is highlyapplicable and can help improve photos taken in low light, clarify satellite images, restoreancient artwork, and more.

Figure 1: Examples of applications of deep image priors[5]

Convolutional neural networks were first used for a significant image recognition task inthe 1990s. In “Gradient-Based Learning Applied to Document Recognition,” the authorstrained a neural network to recognize handwritten digits from the MNIST database [4]. Themodel architecture included two pairs of convolution and subsampling layers, followed bydense layers. In the original paper, the model was trained on 60,000 images. The next jumpin performance came from AlexNet, which won the ILSVRC-2012 competition by classifyingimages into 1000 classes with a top-5 error rate of 15.3% [3]. AlexNet was trained on 1.2million images. Such historical context makes the fact that deep image priors use no trainingdata even more impressive.

2 Background

The maximum a posteriori (MAP) estimator is a means of approximating parameters for amodel that might best describe a set of data. It is Bayesian in nature because it calculatesthe conditional probability of observing the dataset given such a model.

Bayes’ Rule is:

P (A|B) =P (B|A) ∗ P (A)

P (B),

where P (A|B) is the posterior probability of A given B, and P (A) in the numerator is theprior probability of A. Using the likelihood of observing B given A, we update our estimationof the probability of A occurring given B (thus, the terms prior and posterior).

Let X be a dataset, and θ the parameters of a model describing this dataset. We canestablish using the Bayes’ Rule that:

P (θ|X) ∝ P (X|θ) ∗ P (θ),

where P (θ) represents the joint probability of that specific set of parameters appearing andP (X|θ) the probability of collecting the set of data X, given the parameters θ. [2]

We can apply this insight to image restoration. Let X be the correct image, and Xc thecorrupted image, both in image space I. X∗, the restoration, can be expressed as

X∗ = argmaxX

P (X|Xc) = argmaxX

P (xc|X) ∗ P (X).

In other words, we should choose as the restoration an image that would maximize theprobability that it is the original image given the corrupted image. Using Bayes’ Rule, wecan rewrite the first argmax expression as the product of a likelihood and a prior. The aboveexpression will be equal to

argminX

E(X;Xc) + E(X). (1)

While the second term is usually called ”the penalty,” in this case we call it the imageprior term, since it is the negative log of the prior.

Let g be a function that maps parameters θ to image X. Equation (1) then becomes:

2

E(g(θ);Xc) + E(g(θ).

According to Lempitsky et al., we can remove the image prior term by selecting a ”good”mapping g, redefining g as fθ(z), where f is a ConvNet with parameters θ and a fixed input,z [5]. This strategy leaves us with just the term

argminθ

E(fθ(z);Xc).

In essence, we would be searching for a minimizer over just the space of parameters, and notimage space, using only the corrupted image. We can model the function f as a Gaussianprocess.

3 Proposed Methodology

First, I will begin by strengthening my understanding of multivariate Gaussian distributionsby reading and reviewing the book “Gaussian Processes for Machine Learning.” [6] Next,we will explore how it is connected to deep image priors, by following the papers “DeepImage Prior,” [5] which introduces the technology from a computer vision perspective, and“A Bayesian Perspective on the Deep Image Prior,” [1] which explains the mathematicalreasoning behind why this tool is so effective.

References

[1] Z. Cheng, M. Gadelha, S. Maji, and D. Sheldon, A bayesian perspective on thedeep image prior, in The IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2019.

[2] P. Katte, Demystifying-deep image prior, Feb 2019.

[3] A. Krizhevsky, I. Sutskever, and G. E. Hinton, Imagenet classification with deepconvolutional neural networks, in Advances in Neural Information Processing Systems 25,F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, eds., Curran Associates,Inc., 2012, pp. 1097–1105.

[4] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-based learning ap-plied to document recognition, Proceedings of the IEEE, 86 (1998), pp. 2278–2324.

[5] V. Lempitsky, A. Vedaldi, and D. Ulyanov, Deep image prior, in 2018 IEEE/CVFConference on Computer Vision and Pattern Recognition, June 2018, pp. 9446–9454.

[6] C. E. Rasmussen and C. K. I. Williams, Gaussian processes for machine learning,MIT Press, 2008.

3