6
The Challenges of Applying Deep Learning for Hemangioma Lesion Segmentation Pedro G. Alves INESC TEC and Universidade do Porto, Portugal Jaime S. Cardoso INESC TEC and Universidade do Porto, Portugal Maria do Bom-Sucesso Centro Hospitalar do São João and Universidade do Porto, Portugal Abstract—Infantile Hemangiomas (IH) make up the most common type of benign vascular tumors affecting children. They can grow for several months until beginning to involute. In present-day clinical practice there’s no objective monitoring protocol. For more objective measures, an automatic evaluation system (CAD system) is needed to aid clinicians in assessing the effectiveness of a given patient’s response to a treatment. One of the stages of these systems is the lesion segmentation. This work addresses the automatic segmentation of lesions in IH. Acknowledging that the methods in the literature for IH lesion segmentation lag behind the state-of-the-art in the image segmentation community, we conduct a comparison of various methodologies for the segmentation of the IH, including both shallow and deep methodologies. Acknowledging the lack of data in the field for a robust learning of deep models, we also evaluate transfer learning techniques to benefit from knowledge extracted in other skin lesions. The best results were obtained with the shortest path method and a multiscale convolutional neural network that merges two pipelines working at different scales. Although promising, the results put in evidence the need for better databases, collected under suitable acquisition protocols. Index Terms—Image Segmentation, Neural Networks, Medical Expert Systems, Infantile Hemangioma. I. I NTRODUCTION Infantile Hemangiomas (IH) are the most common vas- cular tumors of infancy, with an incidence of around 4% to 5%, appearing more frequently on females, Caucasians and premature children. They exhibit three main evolutive phases: a proliferative phase, where the tumor grows rapidly, a maturation/plateau phase, where it ceases to grow, and an involution phase, in which the color of the lesion is attenuated and its size diminishes [1]. While being benign, many IH have various related compli- cations which include ulceration, functional difficulties, airway obstruction, disfigurement and scarring, auditory canal occlu- sion and vision impairment. To prevent these complications, an adequate diagnosis must be delivered to determine if treatment is needed to stop further proliferation and induce regression, or to avoid scarring and disfigurement. Given that IH are so heterogeneous, deciding if treatment is needed and when it is needed requires detailed knowledge of the IH natural history This work is partially financed by the ERDF - European Regional Devel- opment Fund through the Operational Programme for Competitiveness and Internationalisation - COMPETE 2020 Programme, and by National Funds through the FCT - Fundação para a Ciência e a Tecnologia within project «POCI-01-0145-FEDER-006961». and must be made individually for each patient [2]. Also, for high risk IH, timing is one of the most important factors to decide whether to intervene. The main issue regarding the management of IH is the lack of a standard and universal diagnostic and monitoring protocol. Current clinical practice consists in the examination of the IH features, such as location, color, shape, size and subtype, to de- termine if there is any risk for proliferation or if it has already started involuting. The hemangioma’s height and width is manually measured and the affected area estimated, while the regression degree is visually estimated by the amount of fading (graying) of the normal saturated red color [2]. Naturally, this is an inaccurate and subjective approach since that the ratio affected area / area reduction is very difficult to measure. This leads to different evaluations between physicians, with the estimation of the degree of regression being based on the classification of a single individual, that may or may not have experience with such cases [3]. Therefore, an automatic method to quantitatively assess cutaneous hemangiomas is needed to provide practitioners with precise, accurate and faster measurements than those taken by manual procedures, that can help deciding whether treatment is or not suitable. Over the years, computer-aided diagnosis (CAD) systems have been a subject of great interest for the detection and analysis of skin lesions from digital images, as they can help physicians easily identifying malignant lesions, a process that is very time consuming and of subjective interpretation. Works related with CADs for IH have only begun in the last decade [4]. As such, they are not as mature as in other types of skin lesions, such as melanoma. Thus, the contributions of this paper are: the first study on deep models for IH segmentation, comparing with handcraft representations; the study on the benefit of transferring knowledge from other types of skin lesions to hemangioma images, overcoming the lack of large databases in the field. II. STATE- OF- THE-ART REVIEW The diagnosis and management of IH have received a strong attention from the pediatric community [5], albeit not in the CAD perspective. The “Haemangioma Activity Score” from 2011 [6], developed manually by experts considering 78 photographs, is a linear combination of symptoms, yielding a severity score. Before 2006, there was no reported work of CAD systems for IH lesions [4]. Nonetheless, skin lesions

The Challenges of Applying Deep Learning for Hemangioma

  • Upload
    others

  • View
    1

  • Download
    0

Embed Size (px)

Citation preview

Page 1: The Challenges of Applying Deep Learning for Hemangioma

The Challenges of Applying Deep Learning forHemangioma Lesion Segmentation

Pedro G. AlvesINESC TEC and

Universidade do Porto, Portugal

Jaime S. CardosoINESC TEC and

Universidade do Porto, Portugal

Maria do Bom-SucessoCentro Hospitalar do São João and

Universidade do Porto, Portugal

Abstract—Infantile Hemangiomas (IH) make up the mostcommon type of benign vascular tumors affecting children.They can grow for several months until beginning to involute.In present-day clinical practice there’s no objective monitoringprotocol. For more objective measures, an automatic evaluationsystem (CAD system) is needed to aid clinicians in assessing theeffectiveness of a given patient’s response to a treatment. One ofthe stages of these systems is the lesion segmentation.

This work addresses the automatic segmentation of lesions inIH. Acknowledging that the methods in the literature for IHlesion segmentation lag behind the state-of-the-art in the imagesegmentation community, we conduct a comparison of variousmethodologies for the segmentation of the IH, including bothshallow and deep methodologies. Acknowledging the lack of datain the field for a robust learning of deep models, we also evaluatetransfer learning techniques to benefit from knowledge extractedin other skin lesions.

The best results were obtained with the shortest path methodand a multiscale convolutional neural network that merges twopipelines working at different scales. Although promising, theresults put in evidence the need for better databases, collectedunder suitable acquisition protocols.

Index Terms—Image Segmentation, Neural Networks, MedicalExpert Systems, Infantile Hemangioma.

I. INTRODUCTION

Infantile Hemangiomas (IH) are the most common vas-cular tumors of infancy, with an incidence of around 4%to 5%, appearing more frequently on females, Caucasiansand premature children. They exhibit three main evolutivephases: a proliferative phase, where the tumor grows rapidly,a maturation/plateau phase, where it ceases to grow, and aninvolution phase, in which the color of the lesion is attenuatedand its size diminishes [1].

While being benign, many IH have various related compli-cations which include ulceration, functional difficulties, airwayobstruction, disfigurement and scarring, auditory canal occlu-sion and vision impairment. To prevent these complications, anadequate diagnosis must be delivered to determine if treatmentis needed to stop further proliferation and induce regression,or to avoid scarring and disfigurement. Given that IH are soheterogeneous, deciding if treatment is needed and when it isneeded requires detailed knowledge of the IH natural history

This work is partially financed by the ERDF - European Regional Devel-opment Fund through the Operational Programme for Competitiveness andInternationalisation - COMPETE 2020 Programme, and by National Fundsthrough the FCT - Fundação para a Ciência e a Tecnologia within project«POCI-01-0145-FEDER-006961».

and must be made individually for each patient [2]. Also, forhigh risk IH, timing is one of the most important factors todecide whether to intervene.

The main issue regarding the management of IH is the lackof a standard and universal diagnostic and monitoring protocol.Current clinical practice consists in the examination of the IHfeatures, such as location, color, shape, size and subtype, to de-termine if there is any risk for proliferation or if it has alreadystarted involuting. The hemangioma’s height and width ismanually measured and the affected area estimated, while theregression degree is visually estimated by the amount of fading(graying) of the normal saturated red color [2]. Naturally, thisis an inaccurate and subjective approach since that the ratioaffected area / area reduction is very difficult to measure.This leads to different evaluations between physicians, withthe estimation of the degree of regression being based onthe classification of a single individual, that may or may nothave experience with such cases [3]. Therefore, an automaticmethod to quantitatively assess cutaneous hemangiomas isneeded to provide practitioners with precise, accurate andfaster measurements than those taken by manual procedures,that can help deciding whether treatment is or not suitable.

Over the years, computer-aided diagnosis (CAD) systemshave been a subject of great interest for the detection andanalysis of skin lesions from digital images, as they canhelp physicians easily identifying malignant lesions, a processthat is very time consuming and of subjective interpretation.Works related with CADs for IH have only begun in the lastdecade [4]. As such, they are not as mature as in other types ofskin lesions, such as melanoma. Thus, the contributions of thispaper are: the first study on deep models for IH segmentation,comparing with handcraft representations; the study on thebenefit of transferring knowledge from other types of skinlesions to hemangioma images, overcoming the lack of largedatabases in the field.

II. STATE-OF-THE-ART REVIEW

The diagnosis and management of IH have received astrong attention from the pediatric community [5], albeit notin the CAD perspective. The “Haemangioma Activity Score”from 2011 [6], developed manually by experts considering 78photographs, is a linear combination of symptoms, yieldinga severity score. Before 2006, there was no reported workof CAD systems for IH lesions [4]. Nonetheless, skin lesions

Page 2: The Challenges of Applying Deep Learning for Hemangioma

like melanoma or psoriasis have been thoroughly researched.Oliveira et al. [7] present an excellent review of the state-of-the-art for the segmentation of pigmented skin lesions, butwhere IH is not included.

Zambanini et al. [4] were the firsts to develop a methodto automatically segment the hemangioma. They performedsegmentation based on color features using a single layerperceptron that classifies the pixels as hemangioma or non-hemangioma. Results were obtained on a set of 120 imagestaken with a digital camera, from which 30 were used astraining set and 90 were used as test set. They obtained amean accuracy of 93.2% and a border error of 32.1%.

More recently, Sultana et al. [8] performed segmentationusing a two-step thresholding in the a* channel of the L*a*b*color space. Experiments were performed on a set of 25 pho-tographs taken with a digital camera in which the acquisitionprocess was not rigorous concerning lighting, camera distanceand angle, or hemangioma positioning. These photos are 24bits per pixel and have a resolution of 4324×3240 pixels. Theauthors do not disclose any quantitative performance results. Inanother work [9], the authors use a color constancy approachto preprocess the images and apply the illumination correctionstep. Segmentation of the lesion area was performed like intheir previous work, on the same set of images, on which theyreport a detection rate of 85% and false positive rate of 16.4%.

Oprisescu et al. [10] concluded in their work that the bestcolor space to use for segmentation is the L*a*b* space.They propose a segmentation procedure based on MAP-Markov segmentation. They used a dataset of 40 RGB images,800× 600 pixels, acquired with a digital camera at the MarieCurie Children’s Hospital from Bucharest, reporting a bordererror of 19.49%.

Neghina et al. [11] employ three different methods: Otsu’smethod, Fuzzy C-Means clustering (FCM) and Region Grow-ing based on FCM. They used a set of 33 images, taken indifferent conditions of illumination, with different cameras,and with the child standing naturally in different positions.The best obtained result was with the Region Growing FCM-based method, with a performance of 90.27%. In anotherwork [12], the authors use a segmentation method based onSelf Organizing Maps followed by a morphological step toreduce the number of classes. The results are obtained for thesame dataset, with a reported performance increase of 1.06%relatively to the previous work using Region Growing FCM.

All the surveyed works rely on handcrafted features as inputto the segmentation method, with no work being reported ondeep learning. In here, we give a first step in that direction.

III. ADOPTED METHODOLOGY

We tackle the segmentation of lesions in IH by establishingbaseline results using the state-of-the-art methods and thenintroducing novel approaches in the scope of IH. Firstly, weuse handcrafted segmentation approaches that work with low-level, fixed image features, such as image statistics, pixelintensity or image gradients. Then, we study the possibility ofimproving results using deep methodologies, which are able

Fig. 1. Example of the considered ROI images extracted from the originalphotos. The red rectangle delimits the bounding box of the lesion area andthe blue rectangle corresponds to an expansion of 20%.

Fig. 2. The framework followed in this work, with the various methodologiesused in each stage.

to learn multiple levels of data representation that correspondto multiple levels of abstraction from a set of images.

We work with a region of interest (ROI) of the lesion,considering the bounding box that contains the lesion andenlarging it by 20%, see Fig. 1. This ensures that all ROI’sdimensions are proportional to the lesion’s dimensions and abalance between lesion and non-lesion pixels. Resizing to astandard size was considered, but due to the high variabilityin image size, resolution and structures, it was not applied asit would severely distort most images.

To ease and improve the segmentation stage, a preprocessingstage was devised to reduce variability and heterogeneity of thedataset (Fig. 2). First we perform illumination uniformizationby applying a histogram transformation to the L∗ channelof the L*a*b* color space of all images, such that thisnew histogram approximately matches the mean histogramcomputed using all images. To achieve color constancy acrossall images, the algorithm Shades of Gray [13] is used. Tosmooth intrinsic noise, but still preserve edges, a GuidedFilter [14] was chosen, using the input image as the guidanceimage. A sharpened version of the input was also considered,but it would transfer noisy structures to the filtered image.

A. Handcrafted Methodologies

To establish baseline segmentation results, we used fivedifferent approaches. These were chosen taking into accountthe reviewed state-of-the-art and the fact that each uses a dif-ferent method of segmentation: thresholding, clustering, edgedetection, active contour and shortest path calculation. The

Page 3: The Challenges of Applying Deep Learning for Hemangioma

methods chosen were thresholding with Otsu’s method, edgedetection using Canny’s Algorithm, fuzzy c-means clustering,boundary detection with gradient vector flow snakes [15] andsegmentation using a closed shortest path [16].

1) Thresholding using Otsu’s Method: After preprocessing,the images are converted to the L*a*b* color space and thea* channel is retained and used as input image. A thresholdt is computed for each image using Otsu’s method, which isthen used to binarize the image. To refine the segmentation,morphological operations and hole filling are applied and thefinal result is obtained by retaining only the largest object.

2) Edge Detection with Canny’s Algorithm: The detectionof borders using Canny’s algorithm was made using also thea* channel. The standard deviation of the Gaussian functionused to filter the image and the values of the double thresholdsT1 and T2 are dependent on each image.

It is difficult to obtain a closed contour simply by applyingthe method. Therefore, to obtain a well defined border andcomplete lesion segmentation, the postprocessing morpholog-ical operations of dilation, closing and pixel bridging wereused. The final step was filling the contour.

3) Fuzzy c-Means Clustering: Again, the input image usedin this method is the a* channel of the preprocessed image.The number of clusters is set to c = 2 and the membershipvalues of each pixel intensity is computed using the FCMalgorithm. Then, these values are inspected and each pixelis labeled as belonging to the class with the correspondinghighest membership, constructing the binary image. As theimage’s color and contrast have high variability, many pixelsare mislabeled. This is more pronounced in images exhibitingregression. Therefore, some postprocessing operations of open,erosion and hole filling are applied. The final segmented imageis obtained by preserving the largest object in the image.

4) Boundary Detection with Gradient Vector Flow Snakes:This method requires the input image - the a* channel -and the snake initialization. Given the input image, the edgemap is computed, from which the gradient vector flow fieldis obtained. The snake initialization was set to be an ellipsecentered at the image centre, and the number of iterations setto 25.

5) Segmentation with Closed Shortest Path: The last of the“shallow” methodologies tested was the closed shortest pathapproach. The input is still the a* channel, and the seed pointis located at the image’s “center of mass”. This center of massis simply the weighted average of the pixels along the rowsand the columns of the image. A kernel of size dependent onthe input image’s is defined to compute the gradient alongthe radial directions, and the weight of each pixel is thencalculated. This weight, along with the distance to the seedpoint are used to compute the cost of each edge and to findthe shortest path.

The number of neighbors that yielded the best resultswas 20, which produces smoother edges when compared toconsidering 8 neighbors.

Fig. 3. For a given image, a central pixel is defined and two possible patchesare drawn.

Fig. 4. Linear CNN architecture, 1R-CNN, with one input.

B. Deep Methodologies

Following the trend of the latest decade on deep method-ologies, we also train a convolution neural network (CNN) toIH segmentation.

1) Convolutional Neural Networks: Deep Learning thrivesfor being able to learn multiple levels of data representationthat correspond to multiple levels of abstraction, where high-level concepts are defined from low-level ones.

The approach chosen for this method was to use squaredpatches of the image, centered at a given pixel, as input.This central pixel is classified either as hemangioma or non-hemangioma. The network is therefore learning to identify towhich class a pixel belongs to, given its surroundings. Anequal number of patches are extracted randomly from bothclasses. These patches are created with dimensions dependingon the largest side of the image, to ensure that its resolution isproportional to the image’s resolution. The resolutions of thepatches were chosen as 1/10 and 1/5 of the image’s largestside as shown in Fig. 3.

Two different network architectures were considered in thiswork. One is a standard linear CNN and the other follows aDAG topology that combines two of these linear architectures.In the CNNs’ representations of Figures 4 and 5 are shownthe dimensions of the data as it goes through the network.The 2R-CNN architecture receives two patches with differentresolutions, while the 1R-CNN receives only one patch, bothnetworks outputting the probability of the central pixel of thepatch belonging to each class.

Page 4: The Challenges of Applying Deep Learning for Hemangioma

Fig. 5. DAG CNN architecture, 2R-CNN, showing the two different inputs.

2) Transfer Learning: Only recently has transfer learningbeen used to solve tasks related to skin lesions. Lopez etal. [17] used pre-trained, publicly available CNNs and fine-tuned them using new datasets, for the purpose of melanomascreening; Liao [18] goes further and tries to develop a uni-versal skin disease diagnosis system, fine-tuning pre-existingCNNs with images from various types of skin lesions; for thepurpose of segmentation, Qi et al. [19] address the problemof melanoma segmentation by fine-tuning a pre-existing CNNand then transforming it to a Fully Connected Network.

In this work, transfer learning will be done in a supervisedway. The goal is to train a CNN on a set of melanomaimages and then make use of that knowledge by fine-tuningthe CNN to be applied to hemangioma lesions. This meansthat, instead of randomly initializing the weights and biasof all the convolutional layers when training the CNN withIH images, those initializations come from the training donewith melanoma images. In summary, in this supervised transferlearning approach, we transfer source weights to initialize thetarget network.

IV. EXPERIMENTAL EVALUATION

A. Dataset and metrics

To evaluate the proposed framework, a database consistingof 200 photos from 29 different patients is used. These wereacquired by a pediatrician using a smartphone during thepatient’s consultation, without a controlled acquisition setup.The lesions covered are from various locations on the body(face, neck, arm, trunk, leg, hand). Each patient has onlyone hemangioma, from which at least 2 photos were taken.From the 29 patients, 13 have photos from more than oneappointment. A binary ground-truth image was constructedfor each image, differentiating the lesion from the rest. Forhemangiomas with multiple structures, the lesion area wasconsidered as the region encompassing all structures.

We adopt the following metrics to evaluate the segmentationresults obtained with the various methods, comparing theresults to the ground-truth image:• Accuracy measures the fraction of pixels that are cor-

rectly segmented, ACC = TP+TNTP+FP+TN+FN

• Sensitivity or True Positive Rate, measures the proportionof lesion pixels that are correctly identified as such, givenby SE = TP

TP+FN• Specificity or True Negative Rate, measures the propor-

tion of background pixels that are correctly identified assuch, given by: SP = TN

FP+TN• The Dice coefficient measures the similarity of two

segmentations, giving more weight to instances where thesegmentations agree, DSC = 2TP

2TP+FP+FN• The Jaccard index is a measure of similarity between

two segmentations. It relates the TP to the number ofpairs that are in the same cluster or belong to the sameclass, neglecting the TN: Jind = TP

TP+FP+FN• The exclusive disjunction (XOR) measure is the level

of disagreement between the segmented image and theground truth image: XOR =

|Igt⊕Is||Igt| , where ⊕ denotes

the exclusive disjunction (XOR) operation.• The Hausdorff distance measures the dissimilarity be-

tween borders. Given the set of points that constitute thesegmented border, S = s1, ..., sn, and the ground-truthborder, G = g1, ..., gm, the minimum distance betweeneach point of S and any point of G is computed, andthe maximum obtained value is retained, h(S,G) =maxs∈S ming∈G ‖s−g‖; the reciprocal is also computed,h(G,S). We adopt as Hausdorff distance the worst ofboth: HD(S,G) = max(h(S,G), h(G,S))

In all these definitions, TP stands for pixels correctly labeledas lesion; TN for pixels correctly labeled as non-lesion; FPfor pixels incorrectly labeled as lesion and FN for pixelsincorrectly labeled as non-lesion.

B. Segmentation Results

Fig. 6 shows a typical IH image, together with the seg-mentation mask obtained with each of the methods underevaluation. In Table I we present the mean results obtainedfor each shallow method.

1) Segmentation using Convolutional Neural Networks: Forboth approaches (1R and 2R), the weights and biases wererandomly initialized, the learning rate set to 10−4, and thenumber of epochs set to 1000. For the implementation of thenetworks the MatConvNet toolbox for MATLAB was used,and training was carried out using a NVIDIA GTX960M.The 200 images were randomly split into 4 subsets of 50images. Two of these sets were selected for training and onefor validation. From each training image, 750 patches fromeach class were extracted, with a total number of 150000patches for training and 75000 patches for validation. On botharchitectures the patches are all resized to 31×31×3, to whichthe mean patch values of the training data are subtracted.

After training the networks, the segmentation of the imagesis made in the following manner: for each pixel of the image a

Page 5: The Challenges of Applying Deep Learning for Hemangioma

TABLE IMEAN RESULTS OBTAINED FOR EACH SEGMENTATION TECHNIQUE.

Method ACC (%) SP (%) SE (%) HD (px) XOR (%) Dice (%) Jacc (%)Otsu’s Method 84.30 89.58 81.57 138.92 45.26 81.05 70.40

Canny’s Algorithm 85.65 68.17 95.35 144.25 40.62 70.69 62.66Fuzzy c-Means 84.44 89.38 81.88 138.42 44.86 81.08 70.45

GVF Snakes 91.83 84.87 95.89 97.90 23.52 87.44 78.56Shortest Path 93.37 91.79 94.28 50.13 19.14 90.35 83.04CNN 1R-10 91.14 92.71 90.53 118.20 25.63 88.30 79.99CNN 1R-5 90.16 81.57 95.14 119.53 27.86 84.50 75.01CNN 2R 92.99 91.67 93.92 105.56 20.10 90.20 82.95

1 stage transfer 87.84 88.57 87.85 128.23 35.26 83.47 73.142 stage transfer 87.94 87.46 88.64 127.38 34.91 83.30 73.05

Fig. 6. Example of a lesion and the results obtained with each segmentationtechnique.

patch is extracted, resized and evaluated through the network,producing a vector of probabilities with size 1×2, with a scorefor each class. The image’s corresponding pixel is labeled (0for non-hemangioma and 1 for hemangioma) to the class withhighest probability. The results for each tested network arealso presented in Table I.

2) Transfer Learning Results: Regarding the TransferLearning approach for segmentation, the same patch-wisetraining was performed. To this end, it was used the publiclyavailable SMARTSKINS melanoma dataset [20], with oneexample presented in Fig. 7. First, a set of 80 melanomaimages taken with a smartphone was randomly divided into4 subsets of 20 images each. The images were preprocessedin the same fashion as the hemangioma images. From eachimage, 1250 patches of each class were extracted, in a totalof 150000 patches used for training and 50000 patches usedfor validation. A network using the 1R-10 architecture wastrained using these patches, in the same fashion as explainedin the previous subsection. Then, fine-tuning of the last fullyconnected layer was performed using the hemangioma data,“freezing” the other layers parameters. The results of this fine-tuned network can be seen in Table I.

The goal of the previous approach was to transfer informa-

Fig. 7. Two examples of images of melanoma: SMARTSKINS dataset (left)and ISIC dataset (right).

tion from different lesions but acquired with a similar sensor(a smartphone). However, the amount of available images isscarce, which poses a problem when using CNNs, that requirelarge volumes of data for training robustly without overfitting.There are numerous publicly available datasets of melanomabut, in those, the images were acquired with a dermoscope.

Dermoscope images (Fig 7) are very different from thoseacquired with a smartphone: the lesion is looked at through amagnifier that has a non-polarizing light source and which mayneed a medium between the lens and the skin. Therefore, toavail the large datasets available, a two stage transfer learningframework was devised, as shown in Fig. 8. First the networkis trained using the dermoscope images and the final param-eters are transferred to become the next CNN initialization.Then, this network is fine-tuned using the melanoma imagesacquired with smartphone - cross-sensor transfer learning.This fine-tuned network is then fine-tuned again using thehemangioma images - domain transfer learning.

The melanoma dataset provided in the ISIC 2017 chal-lenge [21] was used to train the network for the first transferlearning stage. This dataset has 2000 dermoscope melanomaimages and their respective ground-truth segmentations. Theimages were preprocessed as explained above and, since theycan often contain hairs, a simple hair removal algorithm wasused. The image set was divided in 1500 images for trainingand 500 images for validation, in which 300 patches wereextracted from each image. Again, a network following the1R-10 architecture was chosen for training. Then, fine-tuningof the trained network is performed using the data from theSMARTSKINS dataset, and then again using the hemangioma

Page 6: The Challenges of Applying Deep Learning for Hemangioma

Fig. 8. The proposed framework for two stage transfer learning.

data. Performance results obtained with this framework arepresented in Table I.

C. Discussion

Among the shallow methodologies, the methodology basedon the shortest path attained the best results. Snakes areright behind in terms of performance. In the middle of theperformance ranking we have the clustering methods, withCanny’s algorithm achieving worst performance.

Regarding the deep methodologies, an interesting aspectis the improvement in performance with the use of twodifferent inputs, as seen by the results obtained with the 2R-CNN architecture. This implies that, in the cases that thenetwork had troubling deciding correctly with only one patch,information from the other patch, which has another “pointof view” helped improve the decision. The performance ofthe best deep methodology and best shallow methodologyare very close, with the computation of the shortest pathslightly surpassing the 2R-CNN. However, if we look at theHaussdorf distance obtained for both methods, the shortestpath calculation greatly outperforms the CNN, suggesting thatthe shortest path method is making less strong errors (butprobably erring more often).

Concerning the transfer learning approach, results suggestno benefit in the process of evolving from melanoma images,both for the single and 2-stage transference. The segmentationperformance surpasses most of the shallow methodologies, butfall short to the results obtained with the other CNNs.

V. CONCLUSION

The present work explored new paths that have not yetbeen considered in the state-of-the-art of IH segmentation,evaluating the potential of deep learning methodologies. Theconventional closed path approach, relying in simple features,obtained the best results. Note that this is the first use of thismethodology in IH segmentation. The deep CNNs obtainedcompetitive performance, but not surpassing the closed pathmethod. Note that the CNNs were learned from a very smalldataset, for current standards in computer vision. We aimnow at working towards a large, publicly available, database

that could become a reference for CAD research in IH. Overthis database, we will reassess the potential of the differencetechniques and extend the work for lesion localization andlesion regression over time.

REFERENCES

[1] C. Kilcline and I. J. Frieden, “Infantile hemangiomas: How commonare they? a systematic review of the medical literature,” PediatricDermatology, vol. 25, no. 2, pp. 168–173, 2008.

[2] K. B. Püttgen, “Diagnosis and management of infantile hemangiomas,”Pediatric Clinics of North America, vol. 61, no. 2, pp. 383 – 402, 2014.Pediatric Dermatology.

[3] L. C. Chang, A. N. Haggstrom, B. A. Drolet, E. Baselga, S. L. Chamlin,M. C. Garzon, K. A. Horii, A. W. Lucky, A. J. Mancini, D. W. Metry,et al., “Growth characteristics of infantile hemangiomas: implicationsfor management,” Pediatrics, vol. 122, no. 2, pp. 360–367, 2008.

[4] S. Zambanini, G. Langs, R. Sablatnig, P. Donath, and H. Maier,“Automatic surveying of cutaneous hemangiomas,” in 18th InternationalConference on Pattern Recognition, vol. 1, pp. 1022–1025, IEEE, 2006.

[5] D. H. Darrow, A. K. Greene, A. J. Mancini, and A. J. Nopper, “Diagnosisand management of infantile hemangioma,” Pediatrics, 2015.

[6] S. R. Janmohamed, F. de Waard-van der Spek, G. Madern, P. de Laat,W. Hop, and A. Oranje, “Scoring the proliferative activity of haeman-gioma of infancy: the haemangioma activity score (HAS),” Clinical andexperimental dermatology, vol. 36, no. 7, pp. 715–723, 2011.

[7] R. B. Oliveira, E. Mercedes Filho, Z. Ma, J. P. Papa, A. S. Pereira, andJ. M. R. Tavares, “Computational methods for the image segmentationof pigmented skin lesions: A review,” Computer methods and programsin biomedicine, vol. 131, pp. 127–141, 2016.

[8] A. Sultana, M. Zamfir, M. Ciuc, S. Oprisescu, and M. Popescu, “Auto-matic segmentation of infantile hemangiomas,” in Signals, Circuits andSystems (ISSCS), International Symposium on, pp. 1–4, IEEE, 2015.

[9] A. Sultana, S. Oprisescu, and M. Ciuc, “Automatic evaluation of he-mangiomas for follow-up monitoring,” in E-Health and BioengineeringConference (EHB), pp. 1–4, IEEE, 2015.

[10] S. Oprisescu, M. Ciuc, A. Sultana, and I. Vasile, “Automatic segmenta-tion of infantile hemangiomas within an optimally chosen color space,”in E-Health and Bioengineering Conference, pp. 1–4, IEEE, 2015.

[11] C. Neghina, M. Zamfir, A. Sultana, E. Ovreiu, and M. Ciuc, “Automaticdetection of hemangiomas using unsupervised segmentation of regionsof interest,” in Communications (COMM), 2016 International Confer-ence on, pp. 69–72, IEEE, 2016.

[12] C. Neghina, M. Zamfir, M. Ciuc, and A. Sultana, “Automatic detectionof hemangioma through a cascade of self-organizing map clustering andmorphological operators,” Procedia Computer Science, vol. 90, pp. 145– 150, 2016. MIUA 2016.

[13] G. D. Finlayson and E. Trezzi, “Shades of gray and colour constancy,”in Color and Imaging Conference, vol. 2004, pp. 37–41, Society forImaging Science and Technology, 2004.

[14] K. He, J. Sun, and X. Tang, “Guided image filtering,” IEEE transactionson pattern analysis and machine intelligence, vol. 35, no. 6, pp. 1397–1409, 2013.

[15] C. Xu and J. L. Prince, “Snakes, shapes, and gradient vector flow,” IEEETransactions on image processing, vol. 7, no. 3, pp. 359–369, 1998.

[16] J. S. Cardoso, I. Domingues, and H. P. Oliveira, “Closed shortestpath in the original coordinates with an application to breast cancer,”International Journal of Pattern Recognition and Artificial Intelligence,vol. 29, no. 01, p. 1555002, 2015.

[17] A. R. Lopez, X. Giro-i Nieto, J. Burdick, and O. Marques, “Skin lesionclassification from dermoscopic images using deep learning techniques,”in Biomedical Engineering (BioMed), 2017 13th IASTED InternationalConference on, pp. 49–54, IEEE, 2017.

[18] H. Liao, “A deep learning approach to universal skin disease classifica-tion,” University of Rochester Department of Computer Science, 2016.

[19] J. Qi, M. Le, C. Li, and P. Zhou, “Global and local informationbased deep network for skin lesion segmentation,” arXiv preprintarXiv:1703.05467, 2017.

[20] M. J. M. Vasconcelos, L. Rosado, and M. Ferreira, “Principal axes-basedasymmetry assessment methodology for skin lesion image analysis,” inInt. Symp. on Visual Computing, pp. 21–31, Springer, 2014.

[21] M. Berseth, “Isic 2017-skin lesion analysis towards melanoma detec-tion,” arXiv preprint arXiv:1703.00523, 2017.