John David Osborne and Song Gao

Machine Classification of Melanoma and Nevi from Skin

Lesions

OutlineMelanoma

Problem DescriptionBiologyClinical Features

Previous ResearchMethods

Sample Images in ExperimentsSteps in Proposed Algorithm

Experimental ResultsDiscussion

Limitations of Current WorkFuture Directions

Melanoma and NeviMelanoma is a form of skin cancer

Not the most common, but the most malignant75% of all skin cancer fatalities

Melanocytes (cells that produce the pigment melanin) become cancerousEnvironmental (sunlight exposure) and genetic

influences

Nevi or NaeviSingular term is “Nevus”Benign tumor of melanocytesOften confused with melanoma

Melanoma versus Nevus

Melanoma Nevi

Previous Classification ResearchExtensive research done with detect differences

from clinically captured digital images of suspicious moles

K-nearest-neighbor classifier was shown to have a mean sensitivity of 95% and 98%, mean specificity of 78% and of 79% on melanoma and nevi respectively.

Kernel Logistics PLS classifier was shown to have performance similar to dermatologists with a sensitivity of 95% and a specificity of 60%

Another separate recent publication has claimed accuracy over 95%

Problem RemainsSpecificity isn’t that goodConsequences are seriousGold standard is histological slide

Histology: Melanoma versus Nevi

Melanoma Nevus

Histological Criteria lesion asymmetry poor circumscription of the lesion with single cells

extending beyond the dermal component irregular and confluent nests variable in size and shape pagetoid spread (suprabasal melanocytes) confluent growth and the dermal epidermal junction (DEJ) absence of maturation (failure of melanocyte nuclei to

become smaller with further descent into the nucleus) involvement of the hair follicle cytological and nuclei atypia

nuclear enlargement melanocytic nuclei enlarged relative to keratinocytic ones

variability in size and shape of nucleushyperchromatism and prominent nucleoli)mitoses in the dermis and the presence of dermal necrotic

melanocytes [1, 2].

Previous WorkHistology Image Processing

MinimalMuch more complicated

NeuroblastomaIntracellular protein localization

Determination of malignancy based on staining pattern

Automated identification of abnormal metaphase chromosome cells for the detection of chronic myeloid leukemia using microscopic images

To the best of our knowledge, no other group is distinguishing between melanoma and nevus on the basis of histology slides

Sample Images in Experiments

40x

200x

100x

400x

Components in Case Image

Steps in Proposed Algorithm1. Remove irrelevant areas

Coarse filtering Fine filtering – more accurate! Recovering

2. Distinguish relevant areas3. SVM training for the prediction

Extract features SVM model training

Coarse FilteringAll images are converted into HSV color

space, which is more perceptual uniform compared with RGB color space.

Thresholdh=0.7, (252°)

Thresholds=0.05

0 0°

120°

240°

red tissue

nucleus

Slide area

Fine FilteringRed area occupies most remaining pixels of the image

after coarse filtering.A precise threshold is required to further remove the

red area.A histogram with 256 bins is built based on the h-value

of the remaining pixels.The interval of dominant bin represents the h-range of red

tissue.Pixels with h-value above this interval are all related with

red tissue.Polynomial curve fitting – p(x)

Better describe the distribution of the histogramThe higher the degree is, the better the curve fits.argmax(p(x)) is more accurate as a threshold than the

interval value of the highest bin.Pixels with h-value larger than argmax(p(x)) is removed

from the image.

6th degree10th degree

Recovering (1/2)The nuclear stain bleeds

over into the surrounding giving them a similar hue and saturation to the red tissue.

The surrounding pixels with the form of small segments are also removed by the filtering procedure.

Small segments are related with cytoplasm, which is relevant area.

Need recover the small segments

Mask after filteringWhite: irrelevant area

Recovering (2/2)What’s the area threshold?

A histogram is built based on the descending area of segments.

Bin width – area interval (e.g. 10 pixels)Bin height – # of segments within

corresponding area intervalThe area threshold is determined by the

area interval of the 1st bin which has a lower number of segments than an user input parameter. (e.g. 10 segments)

Demo of Image Segmentation(a) Original Image | # of pixels: 3133440 (b) After Coarse Filtering | # of pixels: 1740271

(c) After Color Filtering | # of pixels: 2002070(d) After Recovering | # of pixels: 1950855

(e)

Otsu’s method

Coarse filtering

Fine filtering

Recovering

Distinguish between blue area and white area

Relevant areas

Extracting FeaturesCriteria

1. The size of a nucleus becomes larger within melanocytes.

2. The shape of nucleus of a melanocyte tends to become more asymmetric.

Four features1. The ratio of the number of nuclei to the area of

cytoplasm2. The ratio of the area of nuclei to the area of

cytoplasm3. The ratio of the perimeter of a nucleus to its area4. The ratio of the major length of a nucleus to its

minor lengthMajor axis

Min

or a

xis

Major axis

The SVM Training for the PredictionA multi-class support vector classification

(SVC) is provided by LIBSVM*

Two kernel functions are used in the SVC, such as rbf (Radial basis function) function and linear function.

15 feature combinations with 4 different magnification of training dataset are trained on SVC.

*http://www.csie.ntu.edu.tw/~cjlin/libsvm/

Experiments (1/2)

Evaluation criteriaAccuracy: # of correctly predicated recordsMelanoma – positive; nevus – negative

Magnification

40x 100x 200x 400x total

# of images 31 33 32 30 126

# of melanoma 18 18 18 18 72

# of nevus 13 15 14 12 54

Statistical information of image dataset

PositivesFalseofNegativesTrueof

NegativesTrueofyspecificit

##

#

NegativesFalseofPositivesTrueof

PositivesTrueofysensitivit

##

#

Experiments (2/2)

SVC model with diff kernel and feature combinations

400x

Accuracy Specificity Sensitivity

nu-SVC rbf [3+4] 0.90 0.75 1.0

nu-SVC rbf [2+3+4] 0.87 0.67 1.0

nu-SVC rbf [1+2+3+4] 0.87 0.67 1.0

nu-SVC linear [3+4] 0.87 0.83 0.89

nu-SVC linear [2+3+4] 0.87 0.83 0.89

nu-SVC linear [1+2+3+4] 0.90 0.83 0.94

SVC comparative results

Limitations of Current FeaturesArea Features

The number of nuclei to the area of cytoplasm. Each segment in the blue area is a nucleus, and the area of the white area is the area of cytoplasm.

The area of nuclei to the area of cytoplasmAtypical Nucleus Features

The perimeter of a nucleus to the area of it.The major length of a nucleus to the minor

length of itOur feature selection has 2 major problems

Melanocyte ProblemLymphocyte Problem

The Melanocyte Problem-Relies on someone else to fill the image with melanocytes

-Not really automated

-Won’t work on all areas of the slide

-Nuclear atypia in non-melanocytes has nothing to do with melanonoma

22

The Lymphocyte ProblemInflammatory infiltrate is full of lymphocytes with staining nucleus

Lymphocytes are very small, and a dense distribution of nuclei

Indicates inflammation, present in various disease and infections including melanoma

Looking for nuclear superploidy, may be counting lymphocytes

23

Limitations of Current WorkUsing only a tiny fraction of available known

featuresOver a dozen features available

Lack of feedback for performance metrics for cell type identification and ground truthAre we looking at the right cells?Have melanocyte masks, could use them

Lack of regional informationIs this melanocyte in the right place

Reliance on human to set slideInability to classify the various types of

melanoma or nevi

Melanocyte TaxonomyNevus Variants

Becker’s Nevus

Junctional Nevus

Compound Nevus

Banal Nevus (has numerous variants)

Dysplastic Nevus

Balloon cell nevus

Meyerson’s nevus

Halo (Sutton’s) nevus

Recurrent nevus (psuedomelanoma)

Inverted type A (clonal) nevus

Cockarde nevus

Nevus spilus

Collision tumors

Eccrine-centered nevus

Acral nevus

Melanoma Variants

Nevoid Melanoma

Small cell melanoma

Desmoplastic melanoma

Melignant blue nevus

Pigment-synthesizing melanoma

Rhabdoid melanoma

Myxoid melanoma

Adenoid (psuedoglandular) melanoma

Angiotripic (pseudovascular) melanoma

Signet-ring cell melanoma

Balloon cell melanoma

Clear-cell melanoma

Metaplastic Melanoma

Spitzoid melanoma

Giant cell melanoma

Future DirectionsCurrent work

Used simple (banal) melanoma and some cases of melanoma in situ

Simple banal nevus and one dysplastic nevusSlides were manually processed to select

areas of interestObvious extensions

Handle more types of Melanoma and NeviAdditional diagnostic features

Get and use the entire slideAbility to recognize cell typesAbility to recognize regions and layers

Maturation of Nests

Additional FeaturesA - Asymmetry in an intradermal naevus (25x)B – Lymphocyte Infiltration (100x)C - Confluent nests in a junctional nevus (200x)D - Poor circumscription: the junctional melanocytic proliferation ends with single cells (200x)E - Predominance of single melanocytes and suprabasal melanocytes (200x)F - Involvement of the hair follicleG - Cytological atypia in a compound naevus (300x)

27

Lowest Hanging FruitAdditional Feature - Lesion Asymmetry

Highly diagnosticNeed entire slide

Cell type recognitionMelanocyteLymphocyte (small, darkly staining)

Can use additional feature of lymphocyte invasionKeratinocytesEpithelial cells

Region detection

Nevus Melanoma

Allows detection of nesting, maturation and pagetoid spread

Highest Hanging FruitMultiple types of nevus and melanoma

Need knowledge base of diagnostic criteria

Could make the algorithm more general purpose (allow detection of melanoma mimics)

30