Discernible Compressed Images via Deep Perception Consistency

Zhaohui Yang1 , Yunhe Wang2 , Chao Xu1 , Chang Xu3

1 Key Lab of Machine Perception (MOE), Peking University 2 Huawei Noah’s Ark Lab3 School of Computer Science, Faculty of Engineering, The University of Sydney

zhaohuiyang@pku.edu.cn, yunhe.wang@huawei.comxuchao@cis.pku.edu.cn, c.xu@sydney.edu.au

AbstractImage compression, as one of the fundamental low-level image processing tasks, is very essential forcomputer vision. Conventional image compressionmethods tend to obtain compressed images by min-imizing their appearance discrepancy with the cor-responding original images, but pay little attentionto their efficacy in downstream perception tasks,e.g., image recognition and object detection. Incontrast, this paper aims to produce compressedimages by pursuing both appearance and percep-tion consistency. Based on the encoder-decoderframework, we propose using a pre-trained CNNto extract features of original and compressed im-ages. In addition, the maximum mean discrepancy(MMD) is employed to minimize the difference be-tween feature distributions. The resulting compres-sion network can generate images with high imagequality and preserve the consistent perception in thefeature domain, so that these images can be wellrecognized by pre-trained machine learning mod-els. Experiments on benchmarks demonstrate thesuperiority of the proposed algorithm over compar-ison methods.

1 IntroductionRecently, more and more computer vision (CV) tasks suchas image recognition [Simonyan and Zisserman, 2015; He etal., 2015], visual segmentation [Long et al., 2015a], objectdetection [Girshick et al., 2014; Ren et al., 2015], and faceverification [Sun et al., 2014], are well addressed by deepneural networks, which benefits from the large amount of ac-cessible training data and computational power of GPUs. Be-sides these high-level CV tasks, a lot of low-level CV taskshave been enhanced by neural networks, e.g., image denois-ing and inpainting [Burger et al., 2012; Zhang et al., 2017;Xie et al., 2012], single image super-resolution [Dong et al.,2016], and image and video compression [Theis et al., 2017;Toderici et al., 2017; Rippel and Bourdev, 2017].

This paper studies the image compression problem, a fun-damental approach for saving storage and transmission con-sumptions, which represents images with low-bit data and re-constructs them with high image quality. Traditional meth-

ods are mainly based on the time-frequency domain trans-form (e.g., JPEG [Wallace, 1992] and JPEG 2000 [Skodras etal., 2001]), which makes compressed images distorted withblocking artifacts or noises. Since convolutional networkshave shown extraordinary performance on image denoisingand inpainting [Burger et al., 2012; Xie et al., 2012], [Zhanget al., 2017; Dong et al., 2015] proposed using CNN to re-move blocks on JPEG compressed images in order to en-hance the compression performance. Moreover, [Toderici etal., 2015] utilized an encoder-decoder network to implementthe compressing task with fixed input size. [Toderici et al.,2017] further extended the encoder-decoder network to a gen-eral model which supports images with arbitrary sizes. [Sunet al., 2018] utlized the recursive dilated network to establishthe image compression system. [Li et al., 2018] proposd tocompress images by exploiting the importance map to achivehigher compression rates.

Although these methods obtained promising performanceto reduce the storage of digital images, there is an importantissue should be considered. In practice, a number of digitalimages will be taken and stored in electronic devices (e.g.,mobile phones, security camera), and a large proportion ofthem will be recognized or post processed using pre-trainedmachine models for person identification, object recognitionand detection, etc. Therefore, we do not expect that com-pressed images cannot be accurately recognized after com-pressing. Fig. 1 shows a toy experiment, we directly removea proportion of frequency coefficients of the original imagein the DCT domain and recognize compressed image using apre-trained ResNet [He et al., 2015]. Although there is onlya very small appearance difference between the original im-age and the compressed image, some underlying structure andtextual changes will affect the calculation of the subsequentneural network. The network recognizes some compressedPepper images as Hamper, though there is no hurdle for useto recognize them by eyes. On the other side, although wecould retrain existing models (e.g., classification or detection)for fitting these compressed images, the time consumption isnot tolerable. Therefore, an image compression method forgenerating compressed images with perception consistencyin the feature domain is urgently required.

To address the aforementioned problems, this paper devel-ops a novel image compression framework which simulta-neously executes image compression and image recognition

arX

iv:2

002.

0681

0v1

[cs

.CV

] 1

7 Fe

b 20

20

Pepper, 98.29% Pepper, 52.23% Hamper, 35.61% Hamper, 17.89%

Figure 1: Recognition results of compressed images using a pre-trained ResNet on an image labeled as Pepper. From left to right are theoriginal image and images after discarding the smallest 95%, 97%, and 98% frequency coefficients, respectively. Recognition results (labeland score) are shown at the top of images.

tasks. In specific, an encoder-decoder network is used forgenerating compressed data and reconstructed images, anda pre-trained CNN is adopted to perceive the difference ofimages after and before compression. By jointly optimizingthese two objectives, the proposed method can produce com-pressed images with low storage which can also be accuratelyperceived as usual by pre-trained CNN models. To the bestof our knowledge, this is the first time to simultaneously in-vestigate the physical compression and visual perception ofimages using deep learning methods. Experiments conductedon benchmark datasets demonstrate the superiority of the pro-posed algorithm over the state-of-the-art methods for com-pressing digital images.

2 Perception-consistent Image CompressionThe encoder-decoder network receives and outputs imagesand represents them with activations of a number of hid-den neurons, which is naturally suitable for implementing theimage compression task [Theis et al., 2017; Toderici et al.,2015; Toderici et al., 2017].

2.1 Encoder-decoder for Image CompressionGenerally, the loss function of an encoder-decoder based im-age compression network can be written as

minθ1,θ2

1

n

n∑i=1

||D(θ2, E(θ1, xi)

)− xi||2, (1)

where E(·) is the encoder network with parameter θ1 for com-pressing the given image xi, n is the number of images, andD(·) is the decoder network with parameter θ2 for recoveringthe compressed data to the original image. To simplify theexpression, the compressed data c and decoded image y aredenoted as

ci = E(θ1, xi), yi = D(θ2, c

i), (2)

respectively. Since the encoder network E consists of a seriesof transforms such as convolution, pooling, binary (or quan-tization) and pruning, the decoded image yi resulted fromconventional JPEG algorithm [Wallace, 1992] often has somedistortions such as blocks and artifacts. Therefore, a feasibleway for enhancing the image quality of yi is to use anotheroperation or model for refining the decoded image yi:

miny

1

n

n∑i=1

(||yi − yi||2 + λR(yi)

), (3)

where yi is the recovered image and R(·) can be some con-ventional regularizations for natural images such as total vari-ation (TV) norm, `1 norm, etc. These techniques have beenwidely used in conventional image processing methods [Gu etal., 2014; Dong et al., 2011]. In contrast, neural networks ofstrong capacity can generate clearer images than those of tra-ditional methods. Additional layers can be easily added afterD(·) for refining y, so that the above function can be absorbedby Fcn. 1 to form an end-to-end model learning framework.

An ideal image compression algorithm should not onlyconcentrate on the low storage of generated images, but alsohave to retain the downstream performance of tasks such asimage recognition and object detection, etc. Therefore, a neu-ral network with parameter θ3 is introduced for supervisingthe generated image:

minθ1,θ2,θ3

1

n

n∑i=1

(||yi − xi||2 + λL(yi, θ3)

),

s.t. yi = D(θ2, E(θ1, xi)

),

(4)

where L(·) can be various based on different applications,e.g., cross-entropy loss, regression loss.

There are a number of off-the-shelf visual models (e.g.,ResNet [He et al., 2015], VGGNet [Simonyan and Zisser-man, 2015], RCNN [Girshick et al., 2014]) well trained onlarge-scale datasets (e.g., ISLVRC [Russakovsky et al., 2015]and COCO [Lin et al., 2014]), and completely retraining themis a serious waste of resources and is time consuming. In ad-dition, most existing deep learning models only support inputimages with fixed sizes (e.g., 224 × 244), while a completeimage compression system should be used for processing im-ages with different sizes.

Hence, we propose to use a pre-trained neural networkF(·) as a “perceptron” to process of original and compressedimages simultaneously. Therefore, we formulate a novel im-age compression method giving consideration to both appear-ance and perception consistency:

minθ1,θ2

1

n

n∑i=1

(||yi − xi||2 + λ||F(yi)−F(xi)||2

),

s.t. yi = D(θ2, E(θ1, xi)

),

(5)

where λ is the trade-off parameter, and parameters in F(·)are fixed. The diagram of the proposed network is shown inFig. 2. Since the size of decoded image can be various ac-cording to different sizes of input image, F(·) cannot be the

Compressed Data

Encoder-Decoder NetworkPre-trained CNN

Feature Space

Original ImagesCompressed Images

Figure 2: The diagram of the proposed method for compressing digital images, which consists of an encoder-decoder network and a pre-trained convolutional neural network. Wherein, the encoder-decoder network represents images with limited storage and the subsequent CNNis used for extracting features of all images simultaneously. Features of compressed images will then align to those of original images byminimizing their distances and distributions.

same as the original pre-trained network. In practice, F(·)is a pre-trained CNN after discarding the last several layers,which can generate features with different dimensionalitiesgiven different images. Although Fcn. 5 does not explicitlyoptimize the recognition performance, considerable convolu-tion filters in F(·) trained over a number of images can bebeneficial for perceiving. Therefore, minimizing Fcn. 5 willencourage the perception consistency between compressedimage y and its corresponding original image.

2.2 Feature Distribution Optimization

A novel image compression model was proposed in Fcn. 5,which introduces a new module F(·) for extracting visualfeatures of original and compressed images. Since there areconsiderable neurons in a well-designed neural network,F(·)will convert input images into high-dimensional (e.g., 512)features, and it is very hard to directly minimize differencesbetween features of these images. Therefore, we propose touse another measurement to supervise the compression task,i.e., maximum mean discrepancy (MMD [Sejdinovic et al.,2013; Long et al., 2015b]), which is used for describing dif-ferences of two distributions by mapping sample data in ker-nel spaces.

Suppose we are given an image dataset with n images, andtwo sets of image features X = {F(xi)}ni=1 sampled fromdistribution p and Y = {F(yi)}ni=1 sampled from distribu-tion q, where xi and yi are the i-th original image and the i-thcompressed image, respectively. The squared formulation ofMMD distance between p and q is defined as:

LMMD(X ,Y) , 1

n||

n∑i=1

ψ(F(xi))−n∑

i=1

ψ(F(yi))]||2, (6)

where ψ(·) is an explicit mapping function. It is clear thatfeature distributions of original and compressed images areexactly the same iff LMMD = 0, i.e., p = q [Sejdinovic etal., 2013]. The above function can be further expanded with

the kernel trick:

LMMD(X ,Y) = 1

n2

[n∑

i=1

n∑i′=1

k(F(xi),F(xi′))+

n∑i=1

n∑i′=1

k(F(yi),F(yi′))−

n∑i=1

n∑j=1

k(F(xi),F(yj))

],

where k(·, ·) is a kernel function for projecting given datainto a higher or infinite dimensional space, which can be setas linear kernel, Gaussian kernel, etc. Since each kernel hasits own functionality for measuring distributions of data, it isvery hard to determine which one is the best in practice with-out time consuming cross-validation. Therefore, we borrowthe strategy in [Long et al., 2015b] to use a set of kernels forprojecting features:{

k =

m∑u=1

βuku :

m∑u=1

βu = 1, βu ≥ 0,∀u

},

where m is the number of kernels, βu is the coefficient of theu-th kernel which can be optimized iteratively. Therefore, wereformulate Fcn. 5 as

LComp(θ1, θ2) =1

n2

n∑i=1

||yi − xi||2+

λ

n2

n∑i=1

||F(yi)−F(xi)||2 + γLMMD(X ,Y),

s.t. yi = D(θ2, E(θ1, xi)

),

X = {F(xi)}ni=1,Y = {F(yi)}ni=1,

(7)

where γ is the weight parameter for the MMD loss. By simul-taneously optimizing the compression loss and the perceptionloss, we can obtain a model which generates compressed im-ages of the consistent perception with original images for aseries of downstream tasks such as image recognition andsegmentation, etc. Alg. 1 summarizes the mini-batch strat-egy of the proposed method for learning image compressionnetwork. In addition, the pre-trained network F(·) will bedropped after the training process.

Algorithm 1 Discernible Images Compression Method.

Require: An image datasets {x1, ..., xn} with n images, apre-trained CNN F(·), λ, γ, and batch size b.

1: Initialize the compression network with an encoder E(·)and a decoder D(·) and their parameters θ1 and θ2, re-spectively.

2: repeat3: Randomly select a batch of b image {x1, ..., xb};4: for i = 1 to b do5: Compress the given image ci ← E(xi);6: Decode the compressed data yi ← D(ci);7: Generate image features F(xi) and F(yi);8: end for9: Calculate LComp(θ1, θ2) according to Fcn. 7;

10: Update θ1 and θ2 according to LComp(θ1, θ2);11: until convergenceEnsure: The optimal compression model with E and D.

Discussion. The proposed method includes the imagecompression task and the visual perception task. Wherein,a pre-trained neural network is utilized for extracting featuresof original and compressed images, which is similar to twocategories of works, i.e., transfer learning [Long et al., 2015b]and teacher-student learning paradigm [Hinton et al., 2015;Romero et al., 2014], which also utilize a pre-trained modelfor inheriting useful information for helping the training pro-cess. The main difference is that we do not train any newparameters for the visual recognition task, and parameters inthe pre-trained network are fixed, which is used as a power-ful regularization for supervising the learning of the encoder-decoder network therefore improves the compression perfor-mance.

3 ExperimentsBaseline Model: There are a number of CNN based imagecompression models [Theis and Bethge, 2015; Theis et al.,2017; Luo et al., 2016; Sun et al., 2018; Toderici et al., 2017;Baig et al., 2017; Mentzer et al., 2018], each of which hasits own pros and cons. We selected [Toderici et al., 2017]as the baseline model, which utilizes a recurrent neural net-work (RNN) for compressing images for the following tworeasons: 1) the RNN based encoder-decoder network can pro-vide compressed images with different compression rates ineach iteration; 2) this model allows the size of the input im-age to be arbitrary, which is more flexible than comparisonmethods with fixed input size.

Training Setup: To have a fair comparison, we followthe setting in [Toderici et al., 2017] to conduct the imagecompression experiment. Each image in the training datasetwas decomposed into 32 × 32 non-overlapping patches andwe train 200 epochs in total. The architecture of RNNused in following experiments is the same as that describedin [Toderici et al., 2017], and networks were trained usingthe PyTorch toolbox. As for the subsequent CNN F(·) (seeFcn. 7) for extracting visual features of original and com-pressed images, we used the ResNet-18 network [He et al.,2015] which shows excellent performance on visual recogni-

tion tasks (e.g., a 89.1% top-5 accuracy and a 69.8% top-1accuracy on the ILSVRC 2012 dataset with 1000 different la-bels). All parameters in this network were pre-trained on theImageNet dataset and will be fixed in following experiments.Note that, F(·) used here consists of the first 14 convolu-tional layers in ResNet-18, since the size of the input imageis much smaller than that in the original ResNet-18. In spe-cific, for a given input image size of 32 × 32, F(·) outputs a512-dimensional feature.

Evaluation Metrics: Peak signal to noise ratio (PSNR)and structural similarity (SSIM) are two widely used criteriafor evaluating image quality by comparing original imageswith compressed images. However, the PSNR only measuresthe mean square error between the compressed image and itsoriginal one, and SSIM ignores image differences in differentscales. According to [Toderici et al., 2017], besides PSNRand SSIM, we also employ the multi-scale structural similar-ity (MS-SSIM) [Wang et al., 2003] for evaluating the perfor-mance of the proposed image compression algorithm. TheMS-SSIM is applied on each RGB channel and we averagedthem as the evaluation result. In addition, MS-SSIM valuesare between 0 and 1. For a given compression rate, a higherMS-SSIM value implies better compression performance.

Table 1: Recognition results on the ILSVRC 2012 dataset with dif-ferent hyper-parameters λ.

Method ResNet-18 MS-SSIM PSNRtop-1 top-5Original 69.8% 89.1% 1.000 -

FRIC 63.5% 85.0% 0.921 24.442λ = 10−4 63.3% 84.8% 0.917 24.433λ = 10−5 63.8% 85.2% 0.922 24.453λ = 10−6 64.0% 85.4% 0.925 24.465λ = 10−7 63.9% 85.3% 0.924 24.460

Impact of parameters: The proposed image compressionmethod as detailed in Alg. 1 has several important parame-ters: the weight parameter λ for reducing difference betweenfeatures of original and compressed images similar, γ is fur-ther used for making features distributions of these imagessimilar which was equal to 1, u was set as 8, and k was setas a serious of Gaussian kernels, as analyzed in [Long et al.,2015b]. b is the batch size, which was set as 192 as suggestedin [Toderici et al., 2017]. It is clear that λ is the most im-portant parameter for balancing appearance difference eval-uated by human eyes and perception difference assessed bymachines. Therefore, we first tested the impact of this pa-rameter on the ILSVRC-2012 dataset using the ResNet-18network as shown Table. 1. Wherein, each model is trainedon the COCO dataset [Lin et al., 2014] is as the same as thatof the baseline FRIC (Full Resolution Image Compression)method [Toderici et al., 2017], and we then employ them onthe validation dataset of ILSVRC 2012, respectively. Thisdataset consists of 50,000 images with different scales andground-truth labels. All images were first compressed by theproposed DIC and several state-of-the-art methods, respec-tively, and then recognized by ResNet-18. In addition, thebpp value of original RGB images is 24, and we can achievea 48× compression rate when bpp = 0.5, e.g., the file size of a1MB image after compression is about 20KB, which is totally

Table 2: Recognition results on the ILSVRC 2012 dataset, bpp = 0.5.

Method ResNet-18 ResNet-50 MS-SSIM PSNRtop-1 acc. top-5 acc. top-1 acc. top-5 acc.

Original 69.8% 89.1% 76.2% 92.9% 1.000 -JPEG 62.5% 84.2% 68.5% 88.6% 0.919 24.436

FRIC [Toderici et al., 2017] 63.5% 85.0% 69.2% 89.0% 0.921 24.442DIC w/o MMD 64.0% 85.4% 69.6% 89.1% 0.925 24.465DIC w/ MMD 64.3% 85.5% 69.8% 89.2% 0.929 24.471

FRCI: Street Sign Ours: Birdhouse

Figure 3: Ground-truth: Birdhouse.

FRIC: Barbershop Ours: Trombone

Figure 4: Ground-truth: Trombone.

FRCI: Piano Ours: Shoji

Figure 5: Ground-truth: Shoji.

FRIC: Pointer Ours: Cat

Figure 6: Ground-truth: Cat.

enough for recent mobile devices. Therefore, in Table 1, bppvalues of the conventional method and the proposed methodare both equal to 0.5 for having a better trade-off betweenimage quality and compression rate.

It can be found from Table 1 that a larger λ significantly re-duces the compression performance, i.e.the MS-SSIM values,and a suitable λ improves MS-SSIM, since the optimizationon visual features can be seen as a powerful regularization forcompression approaches considering thatF(y) = F(x) if thedecoder network can generate original images, i.e., y = x.When λ = 10−6, the proposed method can obtain more accu-rate results than the baseline FRIC. Thus, we kept λ = 10−6

in the proposed image compression framework, which pro-vides compressed images with better image quality and visualrepresentability.

Comparision Experiments: After investigating the trade-off between the performance of image compression and vi-sual recognition, we then compare the proposed method withstate-of-the-art compression methods on the ILSVRC-2012dataset to verify its effectiveness. All images were first com-pressed by the proposed DIC (Discernible Images Compres-sion) and several state-of-the-art methods, respectively, andthen recognized by ResNet-18. Since the pre-trained net-work F(·) in Fcn. 7 used for extracting features of images

before and after compression is part of the ResNet-18, wealso employed the ResNet-50 network to further recognizethese images to verify the generalization ability of the pro-posed method. This network achieves a 89.2% top-5 accuracyand a 69.8% top-1 accuracy on the ILSVRC 2012 dataset.

Compression results are detailed in Table 2, where bothResNet-18 and ResNet-50 were pre-trained on the ILSVRC2012 dataset, and the experiment here aims to investigatehow the image compression algorithm affects the subsequentmachine learning tasks. Note that, lower bpp values are fre-quently discussed in many works for obtaining higher com-pression rates [Li et al., 2018; Rippel and Bourdev, 2017], butcompressed images with bpp values lower than 0.5 have ob-vious distortions on compressed image, where results of stan-dard JPEG algorithm are also provided for an explicit com-parison.

It can be found in Table 2, compressed images generatedby all methods downgrade the performance of the subse-quent recognition task. It is worth mentioning that, FRICachieved relatively higher results, since the recurrent networkcan recober the compressed data iteratively. In contrast, theproposed method can provide compressed images with thehighest recognition accuracy on both ResNet-18 and ResNet-50 networks. In addition, some recognition results of com-

Original FRIC DIC

Figure 7: Object detection results of images after applying different methods.

Table 3: Object detection results on the VOC 2007 dataset using SSD, bpp = 0.5, SSD model is trained on VOC07+12.

Method Original FRIC DIC JPEGmAP 77.50 71.9 72.3 71.7

MS-SSIM 1.000 0.937 0.940 0.935PSNR - 24.671 24.694 24.655

pressed images are illustrated in Fig. 3-4. Since the pro-posed image compression method can preserve the perceptionconsistency, images compressed by the proposed method canbe recognized, while predictions on FRIC are biased, e.g., aTrombone was recognized as a Barbetshop as shown in Fig. 4.

Moreover, we further removed the MMD loss, i.e., the lastterm in Fcn. 7, and re-trained a new model for compressingimages to test the impact of the introduced feature distribu-tion regularization. This model was denoted as DIC withoutMMD (i.e., γ = 0 in Fcn. 7), and recognition results of com-pressed images using both ResNet-18 and ResNet-50 werereported in Table 2. The proposed DIC after removing theMMD regularization has an obvious accuracy decline, e.g.,its top-1 acc. of ResNet-18 is about 0.3% lower than that ofthe whole DIC method. SinceF(·) converts input images into512-dimensional features, and it is very hard to directly min-imize differences between these high-dimensional features oforiginal and compressed images. The MMD loss thus canprovide a more powerful regularization for obtaining betterresults.

Object Detection after Compressing: Besides the im-age classification experiment, we further verify the effective-ness of the proposed discernible compressed image genera-tion method on a more complex computer vision application,i.e., object detection. In practice, the deep neural network re-ceives an input image and then outputs locations and labels ofeach object in the image. Therefore, a slight distortion on theinput image could severely damage the prediction results. Weselected the SSD (Single Shot MultiBox Detector [Liu et al.,2016]) trained on the VOC 0712 (Visual Object Classes [Ev-eringham et al., 2010]) as the baseline model to conduct the

following experiments.Table 3 reports the detailed mAP (mean Average Preci-

sion) values of original and compressed images using dif-ferent methods, and averaged MS-SSIM results on the VOC2007 validation set. It is clear that the proposed DIC main-tains a higher detection performance. In addition, Fig. 7 il-lustrates some object detection results of original images andcompressed images by exploiting the conventional FRIC andthe proposed method. It is obvious that the pre-trained SSDmodel can still detect and recognize objects in compressedimages using the proposed DIC algorithm, but cannot accu-rately recognize those images compressed by exploiting con-ventional methods.

4 ConclusionsThis paper investigates perception consistency for imagecompression by embedding a pre-trained neural network intothe existing encoder-decoder network. Beyond directly min-imizing the distortion between original images and com-pressed ones generated by the decoder network, we take theperception loss on features into consideration. Compared tostate-of-the-art methods, we can generate compressed imagesof higher image quality by retaining their perception resultssimultaneously. Experiments on benchmark datasets showthat the proposed DIC method can not only produce clearerimages of lower storage, but also has limited influence on thedownstream visual recognition tasks. The proposed imagecompression scheme creates a bridge to connect human andmachine perceptions. It can be easily deployed for other ap-plications such as denoising, super-resolution, and tracking.

References[Baig et al., 2017] Mohammad Haris Baig, Vladlen Koltun, and

Lorenzo Torresani. Learning to inpaint for image compression.In NIPS, pages 1246–1255, 2017.

[Burger et al., 2012] Harold C Burger, Christian J Schuler, and Ste-fan Harmeling. Image denoising: Can plain neural networkscompete with bm3d? In CVPR, 2012.

[Dong et al., 2011] Weisheng Dong, Lei Zhang, and GuangmingShi. Centralized sparse representation for image restoration. InICCV, pages 1259–1266, 2011.

[Dong et al., 2015] Chao Dong, Yubin Deng, Chen Change Loy,and Xiaoou Tang. Compression artifacts reduction by a deep con-volutional network. In ICCV, 2015.

[Dong et al., 2016] Chao Dong, Chen Change Loy, Kaiming He,and Xiaoou Tang. Image super-resolution using deep convolu-tional networks. IEEE TPAMI, 38(2):295–307, 2016.

[Everingham et al., 2010] Mark Everingham, Luc Van Gool,Christopher KI Williams, John Winn, and Andrew Zisser-man. The pascal visual object classes (voc) challenge. IJCV,88(2):303–338, 2010.

[Girshick et al., 2014] Ross Girshick, Jeff Donahue, Trevor Darrell,and Jitendra Malik. Rich feature hierarchies for accurate objectdetection and semantic segmentation. In CVPR, 2014.

[Gu et al., 2014] Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xi-angchu Feng. Weighted nuclear norm minimization with appli-cation to image denoising. In CVPR, 2014.

[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, andJian Sun. Deep residual learning for image recognition. arXivpreprint arXiv:1512.03385, 2015.

[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and JeffDean. Distilling the knowledge in a neural network. arXivpreprint arXiv:1503.02531, 2015.

[Li et al., 2018] Mu Li, Wangmeng Zuo, Shuhang Gu, Debin Zhao,and David Zhang. Learning convolutional networks for content-weighted image compression. CVPR, 2018.

[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Belongie,James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, andC Lawrence Zitnick. Microsoft coco: Common objects in con-text. In ECCV, 2014.

[Liu et al., 2016] Wei Liu, Dragomir Anguelov, Dumitru Erhan,Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander CBerg. Ssd: Single shot multibox detector. In ECCV, 2016.

[Long et al., 2015a] Jonathan Long, Evan Shelhamer, and TrevorDarrell. Fully convolutional networks for semantic segmentation.In CVPR, 2015.

[Long et al., 2015b] Mingsheng Long, Yue Cao, Jianmin Wang,and Michael Jordan. Learning transferable features with deepadaptation networks. In ICML, 2015.

[Luo et al., 2016] Ping Luo, Zhenyao Zhu, Ziwei Liu, XiaogangWang, Xiaoou Tang, et al. Face model compression by distill-ing knowledge from neurons. In AAAI, pages 3560–3566, 2016.

[Mentzer et al., 2018] Fabian Mentzer, Eirikur Agustsson, MichaelTschannen, Radu Timofte, and Luc Van Gool. Conditional prob-ability models for deep image compression. In CVPR, 2018.

[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Girshick, andJian Sun. Faster r-cnn: Towards real-time object detection withregion proposal networks. In NIPS, 2015.

[Rippel and Bourdev, 2017] Oren Rippel and Lubomir Bourdev.Real-time adaptive image compression. arXiv preprintarXiv:1705.05823, 2017.

[Romero et al., 2014] Adriana Romero, Nicolas Ballas,Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta,and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXivpreprint arXiv:1412.6550, 2014.

[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su,Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang,Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Ima-genet large scale visual recognition challenge. IJCV, 115(3):211–252, 2015.

[Sejdinovic et al., 2013] Dino Sejdinovic, Bharath Sriperumbudur,Arthur Gretton, and Kenji Fukumizu. Equivalence of distance-based and rkhs-based statistics in hypothesis testing. The Annalsof Statistics, pages 2263–2291, 2013.

[Simonyan and Zisserman, 2015] Karen Simonyan and AndrewZisserman. Very deep convolutional networks for large-scale im-age recognition. ICLR, 2015.

[Skodras et al., 2001] Athanassios Skodras, Charilaos Christopou-los, and Touradj Ebrahimi. The jpeg 2000 still image compres-sion standard. IEEE Signal processing magazine, 18(5):36–58,2001.

[Sun et al., 2014] Yi Sun, Yuheng Chen, Xiaogang Wang, andXiaoou Tang. Deep learning face representation by jointidentification-verification. In NIPS, 2014.

[Sun et al., 2018] Liyan Sun, Zhiwen Fan, Yue Huang, XinghaoDing, and John Paisley. Compressed sensing mri using a recur-sive dilated network. In AAAI, 2018.

[Theis and Bethge, 2015] Lucas Theis and Matthias Bethge. Gen-erative image modeling using spatial lstms. In NIPS, 2015.

[Theis et al., 2017] Lucas Theis, Wenzhe Shi, Andrew Cunning-ham, and Ferenc Huszar. Lossy image compression with com-pressive autoencoders. arXiv preprint arXiv:1703.00395, 2017.

[Toderici et al., 2015] George Toderici, Sean M O’Malley, Sung JinHwang, Damien Vincent, David Minnen, Shumeet Baluja,Michele Covell, and Rahul Sukthankar. Variable rate imagecompression with recurrent neural networks. arXiv preprintarXiv:1511.06085, 2015.

[Toderici et al., 2017] George Toderici, Damien Vincent, NickJohnston, Sung Jin Hwang, David Minnen, Joel Shor, andMichele Covell. Full resolution image compression with recur-rent neural networks. In CVPR, pages 5435–5443, 2017.

[Wallace, 1992] Gregory K Wallace. The jpeg still picture com-pression standard. Consumer Electronics, IEEE Transactions on,38(1):xviii–xxxiv, 1992.

[Wang et al., 2003] Zhou Wang, Eero P Simoncelli, and Alan CBovik. Multiscale structural similarity for image quality assess-ment. In Signals, Systems and Computers, 2004. ConferenceRecord of the Thirty-Seventh Asilomar Conference on, 2003.

[Xie et al., 2012] Junyuan Xie, Linli Xu, and Enhong Chen. Imagedenoising and inpainting with deep neural networks. In NIPS,2012.

[Zhang et al., 2017] Kai Zhang, Wangmeng Zuo, Yunjin Chen,Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Resid-ual learning of deep cnn for image denoising. IEEE TIP, 2017.