Multi-Cell Multi-Task Convolutional Neural Networksfor Diabetic Retinopathy Grading

Kang Zhou1,2, Zaiwang Gu2,3, Wen Liu1, Weixin Luo1, Jun Cheng2, Shenghua Gao1, Jiang Liu2

Abstract— Diabetic Retinopathy (DR) is a non-negligible eyedisease among patients with Diabetes Mellitus, and automaticretinal image analysis algorithm for the DR screening is inhigh demand. Considering the resolution of retinal image isvery high, where small pathological tissues can be detectedonly with large resolution image and large local receptive fieldare required to identify those late stage disease, but directlytraining a neural network with very deep architecture andhigh resolution image is both time computational expensiveand difficult because of gradient vanishing/exploding problem,we propose a Multi-Cell architecture which gradually increasesthe depth of deep neural network and the resolution of inputimage, which both boosts the training time but also improves theclassification accuracy. Further, considering the different stagesof DR actually progress gradually, which means the labels ofdifferent stages are related. To considering the relationships ofimages with different stages, we propose a Multi-Task learningstrategy which predicts the label with both classification and re-gression. Experimental results on the Kaggle dataset show thatour method achieves a Kappa of 0.841 on test set which is the 4-th rank of all state-of-the-arts methods. Further, our Multi-CellMulti-Task Convolutional Neural Networks (M2CNN) solutionis a general framework, which can be readily integrated withmany other deep neural network architectures.

Index Term—Deep Learning, Multi-Cell Architecture, Multi-Task Learning, Medical Image, Diabetic Retinopathy.

I. INTRODUCTIONDiabetic Retinopathy (DR) is an eye disease caused by

diabetes. Usually DR distresses people who has diabetesfor a significant number of years. It will lead DR patientsto blindness if untreated while treatments can be appliedto slow down or stop further vision loss if the conditioncan be detected early. It is of great significance for peoplewith diabetes to have a regular eye screening for DR.Clinicians often use a five-grade system as shown in Fig.1 to describe the severity of DR. Currently the image gradeis obtained manually, which is time-consuming, subjectiveand expensive. Therefore, automatic retinal image analysisalgorithm for DR grading is in high demand.

Most automated systems [3][5][10] use hand-crafted im-age features, such as shape, color, brightness and domainknowledge of diabetic retinopathy, which includes optic disk,blood vessels and macula. Since 2012, Convolution NeuralNetworks (CNN) have shown remarkable performance inimage classification tasks [7]. Recently, there are some worksfor DR diagnosis at image level based on deep learning.

1School of Information Science and Technology, ShanghaiTechUniversity, China. {zhoukang, liuwen, luowx, gaoshh}@shanghaitech.edu.cn

2Ningbo Institute of Materials Technology and Engineering, China.{guzaiwang, chengjun, jimmyliu}@nimte. ac.cn

3Shanghai University, China.

Machael et al. [2] compared the performance between deeplearning method and previous traditional methods, and showsthat deep learning achieves significantly better performancein DR diagnosis. Chanrakumar et al. [8] used Deep CNN toclassify fundus images on the public Kaggle [1] dataset withimage-level labels.

However, because of the characteristics of DR images,deep learning based DR diagnosis has two challenges: i) Theimage resolution of DR images (usually 2048 × 3072 pixelsor larger) is significantly higher than that of general images(469 × 387 pixels on ImageNet benchmark [4]). On the onehand, such high resolution is required because those smallpathological tissues can be found only with high resolutionimages. So directly reducing the resolution of DR imageswith downsampling as the input of CNN would evidentlyreduces the sensitivity of CNN to these early stage disease.But the network training with such large resolution imageis very time consuming. On the other hand, for those latestage disease, the large local receptive field is needed toidentify the disease with larger regions. To increase the localreceptive field, we can either reduce the image resolutionfor the fixed depth CNN, which is not desirable for earlystage disease, or increase the kernel size or depth for thefixed resolution of image, which may lead to gradient van-ishing/exploding problem and more expensive computationalcosts because of more parameters. To tackle this problem,we design a Multi-Cell architecture. We gradually increasethe resolution of images and the depth of CNN which notonly accelerates the training procedure but also improvesthe diseases classification accuracy. ii) For general imageclassification, if one image is misclassified, it’s loss is fixedand it is not related to which category the image is classifiedto. However, for DR diagnosis, on the one hand, we wantthe image to be corrected; on the other hand, the diseasesprogress gradually, so the severities of diseases at differentstages are different, so the loss of misclassifying diseases todifferent incorrect stages are different, either. For example,the price of misclassifying a proliferative DR (grade 4) asno DR (grade 0) is much higher than that of misclassifyinga mild non-proliferative DR as no DR. In other words, evenif the image is not classified, we want it’s predicted labelso to as close to the ground truth as possible. So both thepopular softmax loss (or Cross Entropy, CE) in classificationand Mean Square Error (MSE) loss in regression for generalcomputer vision tasks are not optimal for medical imaging.Thus we propose a Multi-Task Learning strategy: we usenot only CE loss for image classification to guarantee thecorrectness of DR diagnosis but also MSE loss for regression

arX

iv:1

808.

1056

4v2

[cs

.CV

] 1

1 O

ct 2

018

(a) Grade 0 (b) Grade 1 (c) Grade 2 (d) Grade 3 (e) Grade 4

Fig. 1. 5 grades DR image instances. Grade 0, 1, 2, 3, 4 means absence of DR, mild non-proliferative DR (NPDR), moderate non-proliferative DR, severnon-proliferative DR, and proliferative DR(PDR), respectively.

to guarantee the small discrepancy between the ground-truthand predicted label. We term our solution as Multi-CellMulti-Task Convolutional Neural Networks (M2CNN) basedDR grading.

The contributions of our work can be summarized as fol-lows: i) We propose a Multi-Cell CNN architecture which notonly accelerates the training procedure, but also improves theclassification accuracy; ii) We propose a Multi-Task Learningstrategy to simultaneously improves the classification accu-racy and discrepancy between ground-truth and predictedlabel; iii) Experimental results validate the effectiveness ofour method. Further, our solution can be readily integratedwith many other existing CNN based DR image diagnosisand other disease diagnosis.

II. OUR METHODThe overall architecture of our M2CNN is shown in Fig.2.

It consists three modules: 1) Inception-Resnet-v2 module [9].It is used as BaseNet; 2) Multi-Cell architecture module.It chooses different neural network path depending on theresolution of input medical image, and gradually increasesthe depth and local receptive field for higher resolution im-ages. We use small and medial resolution images to pre-trainthe model to accelerate the train speed for large resolutionimages; 3) Multi-Task Learning module. We simultaneouslyconduct image classification and label regression task.

A. PreprocessingSince the fundus retinal images are captured under differ-

ent conditions, these images often vary largely from one toanother in terms of lighting condition, color, the ratio of thefundus area in the image, etc. In this paper, we apply thefollowing preprocessing to reduce the variation. We firstlyremove the extra black pixels from the image and thenperform the image normalization based on the min-poolingfiltering [6].

Ic = αI + βG(ρ) ∗ I + γ (1)Here ∗ denotes the convolution operation, I denotes inputimage and G(ρ) represents the Gaussian filter with a standarddeviation of ρ. Fig.3 shows an example of the original imageand its associated preprocessed one.

B. Multi-Task LearningA commonly used loss function for classification in gen-

eral computer vision is CE loss function. It outputs prob-abilistic predictions by using softmax activation. The lossfunction of CE is

L1 = −1

m

[ m∑i=1

k∑j=1

1{y(i) = j} log(Probj)]

(2)

Fig. 2. The overall network architecture of our M2CNN. The stemproposed in Inception-Resnet-v2 [9] consists of some convolutional layersand max pooling layers. “Cell” means convolutional layers and regard theselayers as a whole. The two types of cell are normal cell that returns a featuremap of the same dimension and reduction cell that returns a feature mapwhere the height and width are reduced. Multi-Cell architecture choosesthe neural network path based on the resolution of input images. Trainingloss includes both Mean Square Error (MSE) loss and Cross Entropy (CE)loss. In testing phase, we can either use scores and probabilities to predictthe label, and our experiments on Kaggle show that scores based predictionusually achieves better performance.

where m denotes the number of input instances, 1{·} de-notes the indicator function, y(i) denotes the i-th label andProbj denotes the probabilities output by softmax activation.However, if some fundus is misclassified, it doesn’t considerthe difference between that different stages that image isclassified into. However, as we aforementioned, the price ofmisclassifying a proliferative DR (grade 4) as no DR (grade0) is much higher than that of misclassifying a mild non-proliferative DR as no DR. In other words, even if the imageis not classified, we want its predicted label so to as close

(a) Original image (b) Preprocessed image

Fig. 3. Image Preprocessing

to the ground truth as possible. To achieve this goal, wecan leverage the Mean Square Error (MSE) in the regressiontask, which is defined as follows:

L2 =1

m

m∑i=1

(y − y(i))2 (3)

where y is the output score. Although MSE considers thedistance between a false prediction and the true label, there isan issue that it is usually more difficult to optimize than thatof classification. For example, if the distance δ = |y − y(i)|is small the squared value δ2 will be smaller and it will betoo small to optimize.

In this paper, we propose to integrate the MSE loss withCE loss. Since MSE computes the distance between differentclasses, it complements the CE loss. The proposed lossfunction is defined as follows:

L = L1 + L2 + Lreg (4)

where Lreg denotes the regularization loss (weight decayterm) used to avoid overfitting.

C. Multi-Cell Architecture

Since the downsampling of the original retinal image withlarge resolution often leads to information loss especiallywhen the lesion is small, it is not optimal to down samplethe image into a very small size, e.g. 224 × 224 pixels,that is often used in general computer vision. On the otherside, if the input image is large and we pass it to BaseNetarchitectures, it will introduce more computational costs.Further, for late stage disease, the large local receptive field isneeded, which would cause the increase of kernel size/depth,which may lead to the gradient vanishing/exploding problemin optimization. To facilitate the training of CNN with largeresolution image, we propose Multi-Cell architecture bygradually increasing the resolution of the image and the depthof network.

TABLE ISPATIAL RESOLUTION OF INPUT IMAGE AND SOME FEATURE MAP

input image 224×224 256×256 448×448 720×720before switch 5×5 8×8 12×12 21×21

after multi-cell 5×5 8×8 5×5 4×4As shown in Fig.2, the multi-cell architecture module

chooses the convolutional neural network path (depth ofnetwork) based on the resolution of input image: i) Whenthe input image is small(resolution < 350 × 350 pixels),the output before switch goes directly to average pooling;ii) When the input image is medium size (350 × 350 pixels

TABLE IIRESULTS OF EACH MODULE

Train MSE CE Multi-Task M2CNNTest scores prob. scores prob. scores prob.

224×224 0.720 0.725 0.742 0.718 - -448×448 0.790 0.772 0.812 0.782 0.830 0.812720×720 0.835 0.751 0.841 0.826 0.844 0.842

is important for DR diagnosis. No matter using scores orprobabilities (prob.) test method, both performances of multi-task learning are better than that of single task. That isto say, when optimizing the multi-task loss function, MSEloss and CE loss help each other to achieve a better localoptimum in solution space. Similar results can also be foundin Table III by comparing BaseNet with BaseNet+MT ontesting set. Further, we find that scores based evaluationusually corresponds to better performance, so we use scoresbased evaluation.

Multi-Cell Architecture Module: To verify multi-cellcan accelerate the training speed, we conduct experimentsby training with large resolution images directly, and theresults are shown in Fig. 4. We can see that the resultof “448*@32k/4.0h” is better than “224@60k/4.2h” and“448@20k/4.2h”, while the result of “448*@40k/5.5h” isbetter than “448@26k/5.5h”. That is to say, using multi-cell architecture with fine-tune strategy, we can get a betterperformance. Similar results can also be found in Table IIIby comparing M2CNN with BaseNet+MT on the testing set.All of these results are based on multi-task learning.

Fig. 4. Comparison of different training method. “224@20k/1.4h” denotesthe input resolution is 224 × 224 pixels, the number of training steps is20,000 and it costs 1.4 hours to train. “448*@32k/4.0h” denotes it is fine-tuned on the model of “224@20k/1.4h” and train another 12,000 (32,000- 20,000) steps using 448 × 448 pixels input images. “448@20k/4.2h”denotes it is trained using 448 × 448 pixels input images without fine-tuning.

C. Performance Comparison

We also compare our M2CNN with the methods achievethe best performance on Kaggle challenge and the state-of-the-art method [11] for CNN based DR diagnosis. Theresults are shown in Table Table III. We can see that ourM2CNN algorithm achieves 3-rd rank (top 0.45%) in Kagglechallenge, which validates the effectiveness of our method.Further, our solution is related to the input and networkoptimization, so it can be readily integrated with other CNN

based DR diagnosis network and can be applied to thediagnosis of other diseases.

TABLE IIICOMPARISON WITH OTHER ALGORITHMS

Algorithm val set test setMin-pooling 0.860 0.849Zoom-in-Net [11] 0.857 0.849o O 0.854 0.844Reformed Gamblers 0.851 0.839M-Net+A-Net [11] 0.837 0.832BaseNet 0.835 0.828BaseNet+MT 0.841 0.838M2CNN 0.844 0.841

IV. CONCLUSION

In this paper, based on the characteristics of DR image, wedesign a novel Multi-Cell Multi-Task Convolutional NeuralNetworks (M2CNN), which can tackle the DR diagnosiswith high resolution images and improves the classification.Experimental results validate the effectiveness of our solutionfor DR image classification.

V. ACKNOWLEDGE

The project is supported by NSFC (NO. 61502304), Shanghai Subject Chief Scientist (A type) (No.15XD1502900) and grant under Y80002RA01,Y60001RA01, Y61102DL03, Y50709WR08 by ChineseAcademy of Sciences.

REFERENCES[1] Diabetic retinopathy detection. https://www.kaggle.com/c/

diabetic-retinopathy-detection/data.[2] M. D. Abràmoff, Y. Lou, A. Erginay, W. Clarida, R. Amelon, J. C.

Folk, and M. Niemeijer. Improved automated detection of diabeticretinopathy on a publicly available dataset through integration of deeplearning. Investigative ophthalmology & visual science, 57(13):5200–5206, 2016.

[3] M. J. Cree, J. A. Olson, K. C. McHardy, P. F. Sharp, and J. V.Forrester. A fully automated comparative microaneurysm digitaldetection system. Eye, 11(5):622–628, 1997.

[4] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet:A large-scale hierarchical image database. In Computer Vision andPattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages248–255. IEEE, 2009.

[5] A. D. Fleming, S. Philip, K. A. Goatman, J. A. Olson, and P. F. Sharp.Automated microaneurysm detection using local contrast normaliza-tion and local vessel detection. IEEE transactions on medical imaging,25(9):1223–1232, 2006.

[6] B. Graham. Kaggle diabetic retinopathy detection competition report.University of Warwick, 2015.

[7] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classificationwith deep convolutional neural networks. In Advances in neuralinformation processing systems, pages 1097–1105, 2012.

[8] T. Nadu. Classifying diabetic retinopathy using deep learning archi-tecture.

[9] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi. Inception-v4,inception-resnet and the impact of residual connections on learning.In AAAI, pages 4278–4284, 2017.

[10] R. Venkatesan, P. Chandakkar, B. Li, and H. K. Li. Classificationof diabetic retinopathy images using multi-class multiple-instancelearning based on color correlogram features. In Engineering inMedicine and Biology Society (EMBC), 2012 Annual InternationalConference of the IEEE, pages 1462–1465. IEEE, 2012.

[11] Z. Wang, Y. Yin, J. Shi, W. Fang, H. Li, and X. Wang. Zoom-in-net:Deep mining lesions for diabetic etinopathy detection. In MICCAI,2017.

https://www.kaggle.com/c/diabetic-retinopathy-detection/datahttps://www.kaggle.com/c/diabetic-retinopathy-detection/data

I INTRODUCTIONII Our MethodII-A PreprocessingII-B Multi-Task LearningII-C Multi-Cell Architecture

III EXPERIMENTSIII-A Experimental SetupIII-B Evaluation of Different ModulesIII-C Performance Comparison

IV CONCLUSIONV AcknowledgeReferences