InterpretableDiscoveryinLargeImageDataSets

KiriL.WagstaffandJakeLee

JetPropulsionLaboratory,CaliforniaInstituteofTechnology

December7,2017

NIPSInterpretableMachineLearningSymposium

©2017,California InstituteofTechnology.Governmentsponsorshipacknowledged.ThisworkwasperformedinpartattheJetPropulsionLaboratory,CaliforniaInstituteofTechnology,underacontractwithNASA.

DiscoveryinLargeDataSets• Scientificdiscoveriesoftencomefromoutliers

WagstaffandLee- NIPS2017 2

ByFlickruserKlaus

DiscoveryinLargeImage DataSets

• Challenges:Representation,Explanations

WagstaffandLee- NIPS2017 3

HiRISE– 1.4MimagesCredit:NASA/JPL-Caltech/Univ.ofArizona

HumanfacesCredit:Pixabay userGeralt

Surveillance PlanetaryScience

NoveltyDetectionMethods• Clustering• IsolationForest[Liuetal.,2008]• Density-based(e.g.,LocalOutlierFactor[Breunig etal.,2000])• SVD• DEMUD:SVD-based+explanations• Explanation:SVDresidual;informationthemodelcouldnotexplain[Wagstaffetal.,2013]

WagstaffandLee- NIPS2017 4

WhyDEMUD?

0 5 10 15 20 25 30 351

2

3

4

5

6

Number of selected examples

Num

ber o

f cla

sses

disc

over

ed

DEMUDCLOVERNNDMSEDERInterleaveStatic SVDRandom

Rareclassdiscovery– UCIglassdata

WagstaffandLee- NIPS2017 5

Explanations– UCIglassdata

• IncrementaldiscoveryusingSVDmodelofselections• Especiallygoodfordiscoveringrareclasses• Explanationsjustifyselections

WhyDEMUD?• IncrementaldiscoveryusingSVDmodelofselections• Especiallygoodfordiscoveringrareclasses• Explanationsjustifyselections• Explanationshelpusersclassifyitems

0 5 10 15 200

10

20

30

40

50

60

70

80

90

100

Selection number

Cum

ulat

ive

accu

racy

DEMUD explanationsNo explanationsRandom

ChemCam expertclassificationperformance

WagstaffandLee- NIPS2017 6

200 300 400 500 600 7000

2

4

6

8

10

12

14

16

18

20 x 10−4

Wavelength (nm)

Inte

nsity

+Mn 259.34−Fe 273.91−Fe 274.62−Fe 274.67−Fe 274.88−Fe 274.93−Fe 275.53−Fe 275.58+Mn 293.28+Mn 293.86+Mn 293.91+Mn 294.88−Mg 516.73−Mg 517.19−Mg 518.34

Explanations:ChemCam spectra

Rhodochrosite:MnCO3

DEMUDforImages• Representation• Rawpixels• SIFT[Lowe,2004],HOG[Dalal &Triggs,2005]• CNNfeatures[Razavian etal.,2014]

WagstaffandLee- NIPS2017 7

DEMUD+CNNRepresentations

Classprobs

WagstaffandLee- NIPS2017 8

Images

[Krizhevsky etal.,2012]

DEMUD+CNNRepresentations

Images

Classprobs

WagstaffandLee- NIPS2017 9

DEMUD

Features

[Krizhevsky etal.,2012]

DEMUDExplanationswithCNNFeatures

WagstaffandLee- NIPS2017 10

DEMUDFeatures

Selection

Explanation

?

Invertresidualstogetvisualexplanations

DEMUDExplanationswithCNNFeatures

WagstaffandLee- NIPS2017 11

DEMUDFeatures

Selection

Explanation

?

DeepGoggle:Generateinputthatyieldsfeaturevalues

(Mahendran&Vedaldi,2015)

CNNFeatureInversionMethods

Invertresidualstogetvisualexplanations

DEMUDExplanationswithCNNFeatures

WagstaffandLee- NIPS2017 12

DEMUDFeatures

Selection

Explanation

?

DeepGoggle:Generateinputthatyieldsfeaturevalues

(Mahendran&Vedaldi,2015)

Up-Conv:Predictoriginalimage

withsecondNN(Dosovitskiy &Brox,2016)

CNNFeatureInversionMethods

Invertresidualstogetvisualexplanations

Experiments– ImageNet

WagstaffandLee- NIPS2017 13

• 1000images• 10classes• Evenlydistributed

Experiments– ImageNet

WagstaffandLee- NIPS2017 14

DEMUD-CNNSVD-CNNDEMUD-pixelSVD-pixelRandom

WagstaffandLee- NIPS2017 15

Selection

Deep

Goggle

Up-Con

vExplanations– ImageNetBassoon Dial Foodpacket Dogsled Zucchini

Experiments– MSLRoverimages

WagstaffandLee- NIPS2017 16

• 6737images• 26classes• Unevendistribution

Experiments– MSLRoverimages

WagstaffandLee- NIPS2017 17

DEMUD-CNNSVD-CNNDEMUD-pixelSVD-pixelRandom

WagstaffandLee- NIPS2017 18

Explanations– MSLRoverimagesSelection

Deep

Goggle

Up-Con

v

Chemin inlet REMSUVsensor MAHLIcal target Turret Ground

Summary• DEMUD+CNNfeatures+CNNfeatureinversion• Fastdiscoveryofnovelimages

• Withvisualexplanations

• Whatwillyoufindinyourimagedataset?

WagstaffandLee- NIPS2017 19

Thankyou:NASAPlanetaryDataSystem(PDS)ImagingNode